What the research says about training effectiveness
The best documented finding in security awareness research is that the effect of training is real but does not last. Reinheimer and colleagues [4] ran a field study in a German public authority covering 409 employees and measured their ability to distinguish phishing messages from genuine ones immediately after training and in the months that followed. The improvement was statistically significant straight after training and held after four months, but after six months it no longer did. That is the strongest available argument that once-a-year training does not work: at least six months pass between one session and the next, during which the organisation is back where it started.
The same work answers the question of how to refresh. The authors tested four forms of reminder and found that video material and interactive message examples performed best - their effect held for a further six months, keeping phishing recognition elevated twelve months after the first training. The organisation studied consequently adopted a six-month reminder cycle. Purely textual forms performed worse in that comparison.
Bullée and colleagues [3] approached the problem from another direction: they decomposed 74 documented social engineering scenarios and examined which persuasion mechanisms they relied on. The result bears directly on training content. The dominant mechanism turned out to be authority, present in about 63 per cent of attacks and clearly more frequent than the others. The attacks were short - fewer than two interaction steps on average - and telephone contact predominated in the set studied. A programme focused solely on email therefore omits the channel this analysis found most frequent, and teaching people to recognise one persuasion mechanism covers more real attempts than drilling them on specific message patterns.
The third finding concerns context. Lallie and colleagues [5], analysing cyberattacks during the pandemic, showed how strongly attackers adapt their content to current events and how much a sudden change in working conditions helps them. The practical conclusion is that training material two years old describes scenarios that no longer exist, and that vulnerability peaks precisely when an organisation is going through change - a reorganisation, a new system, a move to remote work.
The way such programmes are conceived has changed too. NIST SP 800-50 Rev. 1, published in September 2024 [6], abandoned the awareness, training and education split familiar from the 2003 version, judging it hard to apply in practice, and shifted the emphasis to building a learning programme aimed at behaviour change and tied to the organisation's real risk.
One clarification is worth making about what those 68 per cent from DBIR 2024 actually measure. The figure covers two different things: falling for social engineering, and plain error - a misconfiguration, data sent to the wrong address, a lost device. Splitting it matters, because only the first is amenable to training in the classical sense; the second responds far more to process change and technical controls. The conclusion for the programme is stable, though: phishing deserves first place, but a programme built solely around phishing misses half the problem.
What specifically to train on
Topic selection should not rest on intuition. The starting point is the current threat picture - the annual threat landscape published by ENISA [2], for instance - set against what actually reaches the reporting channel in that organisation. The second part matters more and is skipped more often, because it is what distinguishes a tailored programme from one bought off the shelf.
The core of a programme is usually eight areas. The first and largest is social engineering: recognising manipulation attempts, verifying the sender's identity independently and - in light of the findings of Bullée and colleagues - treating an appeal to authority as a warning signal, on the telephone as much as in email. The second is authentication: a password manager, multi-factor authentication as standard, no credential sharing, and awareness that a password once used on another service is a public password. The third is information handling: what is confidential, how to label it and what must never leave by an external channel. The fourth is data protection to the extent an employee needs it: what personal data is, what to do about a breach and what rights data subjects have (see the article on GDPR audits).
The remaining four areas are more practical. The fifth covers workplace hygiene - locking the screen, a clear desk, documents left in the printer. The sixth concerns mobile and remote work: public networks, personal devices used for work and keeping the two roles apart. The seventh, in practice the most important, is what to do during an incident: how to recognise one, whom to tell, what not to delete and within what time. The eighth is role specifics - finance faces different scenarios from HR, and system administrators different ones again.
Standing apart is the module for the management body, required outright by article 20(2) of the NIS2 directive [7]. Its content differs from general training: it is not about recognising messages but about the ability to assess risk and risk management practices, knowledge of sector-typical threats and awareness of the personal liability the rules create.
Format: what works and what does not
Retention research yields a simple conclusion about form: what counts is the frequency of contact, not the length of a single session. Short modules repeated every few weeks hold recognition better than one long meeting, and reminders based on video and interactive examples performed clearly better in the study by Reinheimer and colleagues [4] than purely textual forms. Scenarios requiring a decision with immediate feedback work well, as does discussing real incidents once anonymised - the latter because they move the problem out of the abstract and into a familiar context.
What does not work is equally well documented. An annual presentation stops having a measurable effect long before the next one. Generic content that does not relate to the realities of the sector requires the audience to translate it into their own work, which they usually do not. Purely technical material explaining what phishing is, without explaining why this particular employee is a target, does not change behaviour.
The most serious design mistake is punishing a click. The result is not greater caution but fewer reports - an employee who fears the consequences stays silent during a real incident, and it is time to report that determines the size of the damage. An approach built on a reporting culture rather than sanction is also the one NIST SP 800-50 Rev. 1 adopts [6].
Phishing simulations
Simulations are the most used tool in these programmes and simultaneously the easiest to get wrong. A sensible frequency is at least quarterly, and every four to six weeks in more mature organisations, with short modules in between. Difficulty should rise: the first campaigns can use patterns that are easy to spot, later ones can reference the organisation's real processes, and the most advanced can model situations where a specific person or role is the target.
After a campaign, what matters is what happens in the following hour. Somebody who clicked should land on a short page explaining how the message could have been recognised - not on a notice about their lapse. Individual results should not be published; team-level aggregates are enough to manage the programme and do not destroy the trust that reporting depends on. Scenarios that play on strong emotions, such as fictitious redundancies or messages about personal misfortune, should also be avoided - they generate resistance to the whole programme and their instructional value is negligible.
Much of the data on simulation effectiveness comes from vendors of training platforms, which calls for caution in reading it. The industry review published by KnowBe4 [8] reports that the proportion of people clicking simulated messages falls over a year or so of a programme from around a third to a few per cent. That is, however, material from a company selling such programmes, based on its own customers' data rather than a controlled study - the figures are worth treating as an indication of direction, not as a measured causal effect. The peer-reviewed retention research described above is a much stronger basis in that role.
Programme metrics
Without measurement there is no way to tell a programme that works from one that is merely ticked off. Methodological guidance for building such measures comes from ISO/IEC 27004:2016 [9]; the set below is a practical minimum.
The basic measure is coverage - the proportion of staff who completed the planned modules on time. It is easy to collect and useless in isolation, because it measures participation rather than effect. Two measures that describe effect are the click rate on simulated messages and the report rate for flagging them as suspicious. Of the two the second matters more, because clicking tells you about a single mistake while reporting tells you whether the organisation learns about attack attempts at all. The ratio between them is a better maturity indicator than either alone.
The third measure is time to report, counted from delivery of the message to it being reported. It translates directly into the consequences of a real incident and into regulatory deadlines, including the 24-hour early warning under NIS2 [7]. The fourth is the number of real events reported by staff, and a rise in the first months of a programme is a good sign rather than a bad one: what grows is not the number of attacks but the number noticed. That is exactly why reports, clicks on real messages and actual compromises have to be counted separately - the first of those numbers should rise, the other two fall.
All these measures should be reported regularly and in one place, because individually each can be improved without security improving. A click rate near zero is not a success but a signal to review the programme: most often it means simulations that are too easy, recognisable by sender or campaign pattern, or a culture in which nobody risks interacting with a suspicious message or reporting it.
Legislation that requires training
The training obligation today follows from several independent bases, each covering a slightly different group. The NIS2 directive [7] names, in article 21(2)(g), "basic cyber hygiene practices and cybersecurity training" among the risk management measures, and separately, in article 20(2), obliges members of the management body to undergo regular training enabling them to identify risks and assess risk management practices. Those two provisions are distinct, and satisfying one does not discharge the other. In Poland they apply through the national cybersecurity system act [10].
In the public sector an independent basis is the KRI regulation [11]. Its § 19(2)(6) requires training for people involved in processing information and, alone among the whole list, states the subject matter directly: information security threats, the consequences of breaching the rules including legal liability, and the use of measures minimising human error. The provision sets no frequency, which is sometimes mistaken for the absence of an obligation.
The GDPR [12] approaches this differently: article 39(1)(b) assigns awareness-raising and staff training to the data protection officer as one of their tasks, which makes it a duty of the role rather than of the organisation as such. ISO/IEC 27001:2022 [13] in turn provides control A.6.3, covering awareness, education and training for everyone within the scope of the information security management system.
Two bases are newer and cover narrower groups. DORA [14] requires financial entities to include security and operational resilience training in their mandatory training programmes, covering senior management as well. Following the 2026 amendment, article 4 of the AI Act [15] requires providers and deployers to take measures that support the development of AI literacy among staff and other people operating or using AI systems on their behalf. It expressly does not require them to guarantee a specified level of literacy for each individual. This duty is distinct from information security, although in practice it is often addressed in a shared module.
Where this is heading
The clearest change is on the attacker's side. Language models have lowered the barrier to attacks tailored to a specific person: content free of language errors and referencing publicly available information about the recipient has stopped being a sign of effort invested. Add to that voice synthesis techniques, which move the problem onto the telephone channel - the same one that turned out to be most frequent in the analysis by Bullée and colleagues [3]. The practical answer is not to teach recognition of ever subtler signals, because that road has run out, but to build independent verification into the process for every action with significant financial or access consequences.
The second change concerns the scope of programmes, which today have to cover the use of tools built on artificial intelligence: the limitations of such models, the risk of feeding them data that should not be disclosed, and their susceptibility to manipulation through input content. Article 4 of the AI Act [15] gives this a legal basis, but the need predates the rule.
The third change follows directly from the work of Lallie and colleagues [5]: since time pressure and unusual situations clearly increase susceptibility, it makes sense to teach people to recognise their own state and to introduce a simple rule of pausing before acting in urgent situations. It is the one element of a programme that works regardless of what the specific attack scenario looks like.
How to recognise a mature programme
A well-built programme can be described by a few easily checked characteristics. It has a regular rhythm - at least quarterly points of contact, not one meeting a year - and differentiated tracks for the management body, administrators, HR, finance and everyone else. It includes phishing simulations with immediate feedback and no sanction for clicking, and a reporting mechanism available to everyone that flags a suspicious message in one action, with reports reaching a team able to handle them (see the article on SOC).
Organisationally, a mature programme includes induction training that a new employee completes before gaining access to systems rather than several weeks later, and documented training for the management body, as article 20(2) of NIS2 requires. Materials - policies, instructions, contact details - are available on the intranet rather than circulated once and forgotten.
Finally, a mature programme is measured and reviewed. A monthly summary of coverage, click rate, report rate and time to report makes regression visible before an incident reveals it, and an annual review allows scenarios to be replaced with ones matching the current threat picture [2]. Holding it all together is a culture in which reporting is valued - including reporting one's own mistake. Without it, the other elements measure something other than what they claim.
Frequently asked questions
- What does an awareness programme cost for an organisation of 50 people?
With an external platform (KnowBe4, Hoxhunt, Cofense): PLN 30 to 80 per user per year, so PLN 1,500 to 4,000 a year for 50 people. With bespoke content (4crypto on site plus e-learning): PLN 5,000 to 15,000 for an annual programme of four sessions, simulations and evaluation.
- On site or online?
A hybrid. The annual introduction and the quarterly briefing work better on site (contact, discussion). Microlearning and simulations belong online (scale, automation).
- Is a training platform mandatory?
No, but it helps enormously. Without one you have to buy or produce content, plan phishing campaigns, analyse results and report - 20 to 40 hours a month. A platform at PLN 1,500 to 4,000 a year for a mid-sized organisation is far cheaper than the staff time.
- What if an employee clicks phishing links repeatedly?
Do not punish - that discourages reporting. Instead: a short one-to-one session of 15 to 30 minutes, a change of simulation pattern starting from the simplest, and monitoring in the SOC (see SOC 24/7). If the problem persists after six to twelve months of intensive coaching, it becomes an HR conversation.
- How do you train a board that "has no time"?
A board format: two to three hours a year, delivered by an external expert (gravitas). The content focuses on personal responsibility (NIS2 article 20) and on real incidents in the sector. For larger organisations: quarterly one-to-one briefings for the chief executive and the finance and technology leads (15 to 30 minutes: the top three vectors in the sector, and how ready the organisation is).
- Does the AI Act change security awareness?
Yes - fundamentally. Since 2 February 2025 there is an AI literacy requirement for staff using AI systems (AI Act article 4) [15]. The content: what AI is, its limitations, hallucinations, prompt injection, data protection in prompts, and limits on the use of consumer language models (see GDPR audit).
- Does training have to be in Polish?
For staff working in Poland - yes. The requirement follows from general management principles: the content has to be intelligible. In international organisations: bilingual material, or the working language with a Polish version for Polish-speaking staff.
- A zero per cent click rate in a phishing simulation - good or bad?
Bad - an alarming signal. Classic security theatre. The three commonest causes:
- The simulations are too easy - a generic "your Office 365 account is expiring" with typos. Staff spot them, the statistics look excellent, and a well-crafted business email compromise still works.
- Staff know it is a simulation - a leak from IT, an obvious sender (typically
@simulated-phishing.com), or campaign patterns they have learned. - Staff are afraid to raise legitimate doubts - a penalty for clicking makes them ignore the message and not report it either.
There is no single credible threshold below which a click rate proves programme maturity - the published figures come mainly from training platform vendors and describe their own customers. What is certain is that a zero result cannot be interpreted: it does not distinguish a resilient organisation from one testing with patterns that are too easy. A better basis for judgement is the report rate and the time to report, and, in research terms, the persistence of improvement over time measured by Reinheimer and colleagues [4].
- Report rate or click rate - which matters more?
Report rate. Click rate is "who got caught" - a negative indicator. Report rate (the percentage of staff who flagged something suspicious through the report phishing button in their mail client) is an indicator of security culture. The logic here is simple and needs no research: an organisation learns about a phishing campaign when somebody reports it, so the report rate sets the upper bound on what can be detected outside the technical filter. Specific thresholds quoted in training vendors' materials [8] describe their own customers and are not the result of a controlled study - follow your own trend rather than comparing against somebody else's benchmark.
The consequence: report both monthly, but celebrating a rising report rate matters more culturally than policing the click rate.
- An employee reported phishing by ordinary email rather than the button - what now?
Appreciate it and make the next step easier. Phishing reported by any means (email, phone, chat) has value. In practice:
- Reply within an hour with thanks and feedback on whether it really was phishing.
- Offer to install the button in Outlook or Gmail - a one-off configuration, after which reports reach the SOC with metadata.
- Enable the phishing reporting button in Microsoft 365 (Defender for Office 365), or Cofense Reporter / KnowBe4 PAB - free or low cost.
- Educate - a short intranet tutorial, "how to report phishing in one click".
This approach matches NIST SP 800-50r1 [6], which directs attention entirely to behaviour change and building a security culture rather than to sanctions for clicking.
- Real incidents went up after the training - good or bad?
Usually good - it normally means more reporting, not more attacks. The mechanism is simple: before the programme some attempts were reported to nobody, so they did not exist in the statistics. Once it starts, the number of reports rises, then stabilises at a higher level, and may even fall over time as staff filter harmless messages more accurately. The pace of that change depends on the starting point and cannot honestly be quoted as a universal figure - you have to measure your own baseline before the programme starts, or later numbers have nothing to be compared with.
Distinguish the key metrics: the number of reported attempts, the number of clicks on real phishing, and the number of actual compromises (credentials, malware). The first rises - that is good. The other two should fall - that is what confirms the programme is working.
- Is AI literacy under the AI Act the same as security awareness?
Related, but different. Following the 2026 amendment, AI Act article 4 [15] requires measures that support the development of AI literacy. It does not require the organisation to guarantee a specified level for every individual. A combined module with security awareness is permitted and often makes economic sense. The AI literacy content:
- What AI is - a language model, a classifier, a decision system.
- Limitations - hallucinations, bias, no reasoning over facts.
- Security - prompt injection, data exfiltration via output, jailbreaking.
- Data protection - what may go into prompts (see GDPR audit).
- Ethics and accountability - who answers for a wrong AI decision.
For the board: an extended module on AI Act risk classification and responsibility for high-risk systems.
Need consulting in this area?
A free 30-60 minute consultation. No obligations. We discuss needs, scale and a high-level timeline.
Related content
Other competence areas
- IT security audit
- Vulnerability scanning
- Penetration testing
- Device and system hardening
- Email security audit
- KRI compliance audit
- KSC and NIS2 audit
- GDPR compliance audit
- Information security policy
- ISMS - information security management system
- SOC 24/7 - monitoring and response
Compliance and regulation
Bibliography and sources
All cited sources are publicly available. ISO/IEC standards, IETF RFCs, EU directives and national legal acts link to the original documents.
- [1]reportVerizon Business (2024). 2024 Data Breach Investigations Report (DBIR). Verizon, 17th edition · https://www.verizon.com/business/resources/reports/dbir/
- [2]reportEuropean Union Agency for Cybersecurity (ENISA) (2025, wersja 1.2 z 9 stycznia 2026 r.). ENISA Threat Landscape 2025. ENISA · https://www.enisa.europa.eu/publications/enisa-threat-landscape-2025
- [3]peer-reviewedBullée, J.-W., Montoya, L., Junger, M., Hartel, P. (2018). On the anatomy of social engineering attacks-A literature-based dissection of successful attacks. Journal of Investigative Psychology and Offender Profiling, vol. 15, no. 1, pp. 20-45 · DOI: 10.1002/jip.1482
- [4]peer-reviewedReinheimer, B., Aldag, L., Mayer, P., Mossano, M., Duezguen, R., Lofthouse, B., von Landesberger, T., Volkamer, M. (2020). An investigation of phishing awareness and education over time: When and how to best remind users. Sixteenth Symposium on Usable Privacy and Security (SOUPS 2020), USENIX Association, pp. 259-284 · https://www.usenix.org/conference/soups2020/presentation/reinheimer
- [5]peer-reviewedLallie, H. S., Shepherd, L. A., Nurse, J. R. C., Erola, A., Epiphaniou, G., Maple, C., Bellekens, X. (2021). Cyber security in the age of COVID-19: A timeline and analysis of cyber-crime and cyber-attacks during the pandemic. Computers & Security, vol. 105, art. 102248 · DOI: 10.1016/j.cose.2021.102248
- [6]standardMerritt, M., Hansche, S., Ellis, B., Sanchez-Cherry, K., Nethery Snyder, J., Walden, D. (2024). NIST SP 800-50r1: Building a Cybersecurity and Privacy Learning Program. National Institute of Standards and Technology, wrzesień 2024 r. Zastąpiła SP 800-50 (2003) i SP 800-16 (1998) · DOI: 10.6028/NIST.SP.800-50r1
- [7]regulationParlament Europejski, Rada UE (2022). Dyrektywa (UE) 2022/2555 (NIS2) w sprawie środków na rzecz wysokiego wspólnego poziomu cyberbezpieczeństwa. Dziennik Urzędowy UE, L 333, 27.12.2022 · https://eur-lex.europa.eu/legal-content/PL/TXT/?uri=CELEX:32022L2555
- [8]reportKnowBe4 (2024). 2024 Phishing by Industry Benchmarking Report. KnowBe4 Research · https://www.knowbe4.com/resources/reports/phishing-by-industry-benchmarking-report
- [9]standardInternational Organization for Standardization (2016). ISO/IEC 27004:2016 - Monitoring, measurement, analysis and evaluation. ISO/IEC · https://www.iso.org/standard/64120.html
- [10]regulationSejm RP (2018). Ustawa z dnia 5 lipca 2018 r. o krajowym systemie cyberbezpieczeństwa (KSC). Dz.U. 2018 poz. 1560 (z późn. zm.) · https://isap.sejm.gov.pl/isap.nsf/DocDetails.xsp?id=WDU20180001560
- [11]regulationRada Ministrów RP (2024). Rozporządzenie Rady Ministrów z 21 maja 2024 r. w sprawie Krajowych Ram Interoperacyjności (KRI). Dz.U. 2024 poz. 773 · https://isap.sejm.gov.pl/isap.nsf/DocDetails.xsp?id=WDU20240000773
- [12]regulationParlament Europejski, Rada UE (2016). Rozporządzenie (UE) 2016/679 (RODO) w sprawie ochrony osób fizycznych w związku z przetwarzaniem danych osobowych. Dziennik Urzędowy UE, L 119, 4.5.2016 · https://eur-lex.europa.eu/legal-content/PL/TXT/?uri=CELEX:32016R0679
- [13]standardInternational Organization for Standardization (2022). ISO/IEC 27001:2022 - Information security, cybersecurity and privacy protection - Information security management systems - Requirements. ISO/IEC · https://www.iso.org/standard/27001
- [14]regulationParlament Europejski, Rada UE (2022). Rozporządzenie (UE) 2022/2554 (DORA) w sprawie operacyjnej odporności cyfrowej sektora finansowego. Dziennik Urzędowy UE, L 333, 27.12.2022 · https://eur-lex.europa.eu/legal-content/PL/TXT/?uri=CELEX:32022R2554
- [15]regulationParlament Europejski, Rada UE (2024). Rozporządzenie (UE) 2024/1689 (AI Act) ustanawiające zharmonizowane przepisy dotyczące sztucznej inteligencji. Dziennik Urzędowy UE, L seria, 12.7.2024 · https://eur-lex.europa.eu/legal-content/PL/TXT/?uri=CELEX:32024R1689
- [16]reportVerizon Business (2026). 2026 Data Breach Investigations Report (DBIR). Verizon · https://www.verizon.com/business/resources/reports/dbir/