How to run a phishing simulation in Microsoft 365 without teaching your staff the wrong lesson

A phishing simulation is the one security exercise that can leave an organisation measurably worse off than doing nothing at all. Run well, it tells you how quickly your people notice an attack and how fast they tell somebody. Run badly, it teaches them that reporting things gets you named in a spreadsheet, and the reporting stops.
The reason this matters more than it used to is that phishing is now almost the whole of the UK threat picture for ordinary businesses. The government’s most recent breaches survey found phishing was the most prevalent attack type by a distance, and that among businesses that were breached, the share hit by phishing and nothing else has risen again. Meanwhile fewer than one in five businesses run any staff training at all.
Most organisations that decide to do something about it reach for Microsoft, on the reasonable assumption that a Microsoft 365 subscription includes the tool. For a large number of them it does not, and the licence that carries it is not the one they are on.
Answer the licensing question properly and choose the right thing to measure, and the exercise gives you something genuinely valuable: a reliable figure for how quickly your organisation notices an attack and tells somebody. Get either of those wrong and you will have spent money teaching your staff to stay quiet.
Quick summary
The decisions that determine whether this works are made before anyone sends an email.
Read together, those points push in one direction: the tool is cheap and capable, and the value of the exercise is decided almost entirely by what you choose to measure and how you treat the people who fail.
What you actually need to license
This is the question that stops most projects, and it is worth getting the answer exactly right before anybody builds a business case.
Attack simulation training lives in Microsoft Defender for Office 365 Plan 2. The Microsoft Defender for Office 365 features service description puts it in a single sentence: “This service is available to organizations that have either Microsoft 365 E5/G5, Office 365 E5/G5, or Microsoft Defender for Office 365 Plan 2. Attack simulation training is a Plan 2 capability.”
The practical consequence catches out a lot of UK small and medium businesses. Microsoft 365 Business Premium, which is the sensible default for most organisations under a couple of hundred staff, includes Defender for Office 365 Plan 1. The feature availability table in the service description gives attack simulation training as “No” for Plan 1 and “Yes” for Plan 2. There is no partial version, no cut-down edition, and nothing to switch on in the admin centre.
Two further points that changed recently and are easy to miss. Effective 1 July 2026, Office 365 E3 and Microsoft 365 E3 also include Plan 1 rather than Plan 2, so an E3 tenant is in exactly the same position as a Business Premium one. And there is a narrow E3 trial offering which Microsoft describes as “a subset of capabilities”: a Credential Harvest payload and a choice of two training experiences, with no other capabilities included.
The options, with all figures taken from Microsoft’s UK pricing pages and all excluding VAT, look like this:
- Defender for Office 365 Plan 2 as an add-on: £3.80 per user per month, paid yearly, on an auto-renewing annual subscription.
- Defender for Office 365 Plan 1, for comparison: £1.54 per user per month on the same terms.
- Microsoft 365 Business Premium: £16.90 per user per month paid yearly, or £20.28 on a monthly subscription.
- Microsoft 365 E5: £51.60 per user per month, annual commitment.
For a forty person business already on Business Premium, the add-on works out at £152 a month, or £1,824 a year before VAT. Moving the same forty people from Business Premium to E5 to get the same feature would cost roughly nine times that in additional licence spend. The add-on is the obviously correct route if simulation is the only thing you are missing, though it is worth confirming with your licensing partner that the add-on can be attached to your specific base subscription, because Microsoft’s published pricing page does not state which base SKUs it can sit on top of.
If you simply want to find out whether this is worth doing at all, the 90 day Defender for Office 365 trial is the honest first step, and it will tell you more than any amount of internal debate.
The seven techniques, and which one to start with
Microsoft offers more attack types than most organisations need, and choosing the wrong one first is how you get a number you cannot act on.
Six of the seven are curated from the MITRE ATT&CK framework, with the seventh being a teaching tool rather than a test.
- Credential Harvest. The message contains a link to a page that asks for a username and password. Compromise is recorded when the user enters credentials, which Microsoft does not store.
- Malware Attachment. The message carries an attachment. Compromise is recorded when the user opens it and selects Enable Editing in Protected View.
- Link in Attachment. A credential harvest hidden inside an attachment rather than in the message body.
- Link to Malware. A link to a file on a well known sharing service such as SharePoint or Dropbox.
- Drive-by URL. A link to a site that runs code in the background, sometimes called a watering hole attack. Here the click itself is the compromise.
- OAuth Consent Grant. A malicious application asks the user to approve access to their data. Compromise is recorded when they accept the prompt.
- How-to Guide. Not a test at all. A short teaching message that shows people how to report a suspicious email, delivered straight to the inbox.
Start with Credential Harvest. It is the technique that matches what actually happens to UK businesses, it distinguishes cleanly between clicking a link and handing over a password, and it is the only one where every training assignment trigger behaves sensibly. Microsoft’s own guidance is explicit that Malware Attachment has no click event to measure, and that on Drive-by URL a click and a compromise are the same thing, which makes both of them poor choices for a baseline.
Setting up a first campaign
A handful of settings do most of the work, and the defaults are mostly sensible.
- Duration. The default is two days, which is also the minimum, and the maximum is thirty. Interaction stops being captured at the end date you set.
- Targeting. You can select individuals, import a CSV with one address per line, or use groups. Microsoft 365 Groups work static or dynamic; distribution groups and mail-enabled security groups are static only, and hidden membership groups are not supported at all.
- Training. Leave it on “Assign training for me”, which picks modules based on previous simulation and training results. The due date defaults to 30 days after the simulation ends, with 15 and 7 day options.
- Notifications. Positive reinforcement, training assignment and training reminder notifications can each be left on Microsoft’s defaults, customised, or turned off. Reminders default to weekly.
- Repeat offenders. The threshold defaults to two consecutive compromised simulations and can be changed in settings.
Three prerequisites are worth checking before you launch rather than after. Populate the Department field in Microsoft Entra ID, because without it the reporting cannot break results down by team, which is the only cut of the data that leads to a useful conversation. Check that your simulation URLs resolve in the browsers your staff actually use, since Microsoft warns that Google Safe Browsing can block them. And be aware that the service uses Safe Links to track clicks even when the “Track user clicks” setting in your Safe Links policies is turned off, which is worth knowing if you have deliberately disabled it.
Two limitations to plan around. Attack simulation training supports on-premises mailboxes “but with reduced reporting functionality”, and there are no PowerShell cmdlets for any of it, so everything happens in the portal. If you run a hybrid estate or you automate everything, both of those will shape what you can realistically do.
Reading the report without fooling yourself
The portal gives you a click rate, and the click rate is the number most organisations take to their board. It is also the number the NCSC specifically warns against leaning on.
The guidance is direct about why. Metrics, it says, “express an organisation’s values, and if you appear to value the absence of reports of problems, you incentivise people to keep quiet about issues”. The recommendation that follows is the single most useful sentence in the whole document: “as well as measuring how many people clicked on a phishing email, focus on how many people reported it.”
So the honest scorecard has four numbers on it, not one.
- Reporting rate. How many recipients told somebody. This is the only number that scales into a real defensive capability.
- Time to first report. How long between delivery and the first person flagging it. This is what determines whether your response team gets ahead of an incident.
- Compromise rate, not click rate. Clicking a link and submitting a password are different failures. Defender records them separately, so report them separately.
- Trend across campaigns. A single result is a snapshot with no meaning. The direction of travel over four or five campaigns is the finding.
Two reporting caveats will bite if you do not know about them. Simulation messages reported through non-Microsoft tools are not captured in the reports at all, so if you use a third party reporting button your reporting rate will read as near zero regardless of what people actually did. And message read signals are unreliable in a couple of common Outlook configurations, so treat “opened” as approximate.
Our post on security awareness training makes the wider argument for why the trend matters more than the snapshot, and it is worth reading alongside this.
What the evidence actually says
This is the part vendor material leaves out, and it is the part that should shape your programme.
The most rigorous study of the question ran for 15 months across 14,733 employees of a large company, and was published at the IEEE Symposium on Security and Privacy in 2022. Its findings are uncomfortable for anyone selling simulation as a training mechanism. Over the 15 months, 4,729 participants, or 32.10%, clicked at least one simulated phish, and 3,747, or 25.43%, went on to do something genuinely dangerous such as submitting credentials or enabling macros. The authors’ conclusion was that embedded training “does not make employees more resilient to phishing, but instead it can have unexpected side effects” that can make them more susceptible.
One caveat matters and the researchers state it themselves: the training in that deployment was voluntary, with no enforcement that employees read it or completed the quizzes. That is not an argument for dismissing the finding, because voluntary is how most real programmes run, but it does mean the study tested a particular delivery model rather than training in general.
The same study found something considerably more positive, and it points at what a simulation programme is genuinely good for. Of those 14,733 people, 4,260 reported at least one suspicious email, and in total participants reported 14,401 emails. The authors concluded that using employees as a collective detection mechanism is practical at scale, and that new campaigns can be detected within minutes of launch.
That is the case for running simulations, and it is not the case the market usually makes. Simulations are a poor teaching tool and a good way to build and measure a reporting habit.
The NCSC’s published case study of a real Dridex campaign at an unnamed UK financial services firm of around 4,000 staff shows the same shape. Of 1,800 malicious emails, 1,750 were stopped by email filtering. Fifty reached inboxes. Twenty-five were reported in total, some of them by people who had already clicked, and those reports were the first indication the attack had got through. Fourteen were clicked, thirteen of the resulting malware instances failed to launch because the devices were up to date, and the single installation was detected when it called home and cleaned within a few hours.
Read honestly, filtering and patching did the defending. Reporting did the detecting. Nobody in that story spotted a phish and saved the day through vigilance, and the NCSC does not claim they did.
Where simulations go wrong
The NCSC’s guidance on phishing contains a section titled, without hedging, “The problems with phishing simulations”. Anyone commissioning one should read it first.
- They cannot achieve what they are sold as achieving. “No training package, including phishing simulations, can teach users to spot every phishing attempt.” Expecting staff to scrutinise every message is described as “an unrealistic and counter-productive goal because responding to emails and clicking links is an integral part of work”.
- There is genuine legal exposure. The guidance states that “phishing simulations can also create legal risk” and that “punishing people for clicking on emails you’ve sent starts to resemble entrapment”. It advises that “you should always check with your HR department before undertaking any phishing simulations”. That is not boilerplate, and it is the step most organisations skip.
- They can damage the thing you are trying to build. “Phishing simulations also erode trust between employees and security. Employees who are afraid for their jobs will not report mistakes.”
- They are usually over-weighted. “Training your users, particularly in the form of phishing simulations, is the layer that is often over-emphasised in phishing defences.”
None of that means do not run one. The NCSC is equally clear elsewhere that it is “still worth training users to spot suspicious links”, and that user reports are a valuable source of intelligence. What it means is that a simulation is a measurement instrument for your reporting culture, not a substitute for the technical controls that do the actual work: filtering, multi-factor authentication that is enforced rather than merely configured, and current patching. Our write-up of the Microsoft 365 password spray campaign is a good illustration of an attack that no amount of user vigilance would have stopped.
Common mistakes and pro tips
The failures here are cultural far more often than they are technical.
- Mistake: running one before you have somewhere for people to report to. If there is no reporting button and no known route, a simulation measures nothing and trains nobody. Fix the reporting path first, ideally with the How-to Guide technique.
- Mistake: naming individuals. The moment results become personal, reporting drops and your instrument breaks. Report by department, never by person.
- Mistake: not involving HR before the first campaign. The NCSC’s advice on this is explicit and it protects you as much as it protects your staff.
- Mistake: treating the click rate as the score. A falling click rate with a falling reporting rate is a worse position than the one you started in, and it looks like success on a dashboard.
- Mistake: assuming Business Premium includes it. It does not, and discovering that halfway through planning wastes a fortnight.
- Mistake: using a third party reporting button and Microsoft simulations together. Reports through non-Microsoft tools do not appear in Defender’s reports at all.
- Pro tip: start with a How-to Guide, not a test. Teach people how to report, then measure whether they do. A baseline taken before anyone knows how to report is measuring the wrong thing.
- Pro tip: populate the Department field in Entra ID first. Without it you cannot segment results, and unsegmented results produce no actions.
- Pro tip: run the 90 day trial before you buy anything. Three months is enough for two campaigns and a trend, which is enough to decide.
- Pro tip: tell people the programme exists. Announcing that simulations happen does not spoil the measurement, and it removes most of the entrapment objection.
Do those and the exercise produces a number you can act on, rather than a number you have to explain.
Not sure whether your licences even cover this?
Our free Microsoft 365 Risk and Cost Health Check tells you exactly which Defender plan your tenant is on, what security features you are paying for and not using, and where your real phishing exposure sits. You get a written summary either way, with no obligation to buy anything from us.
The bottom line on phishing simulations
The licensing question decides whether this is a project at all, and it has a better answer than it used to. Attack simulation training is a Defender for Office 365 Plan 2 capability, and Business Premium and E3 both carry Plan 1. But the Microsoft Defender Suite for Microsoft 365 Business Premium add-on, at £7.70 per user per month excluding VAT, brings Plan 2 to a Business Premium tenant for less than half the per-seat cost of moving everybody to E5.
The measurement question decides whether it is worth doing. A phishing simulation is not a good way to teach people to spot phishing, and the best available evidence says so fairly bluntly. It is a good way to find out whether your people report things, how fast, and whether that is improving. Those are genuinely useful facts about an organisation, and almost nobody has them.
The cultural question decides whether it does harm. Speak to HR before the first campaign, report by department and never by name, announce that the programme exists, and treat a rising reporting rate as the win rather than a falling click rate. Get that wrong and you will suppress the reporting you depend on to detect the real thing.
If you want the technical layers doing the heavy lifting before you start testing people, our Microsoft 365 email security post covers the controls that stop most of this before anybody sees it, and our managed IT support team configures and runs this work for businesses across the West Midlands every week.
STRAIGHT ANSWERS
Frequently asked questions
Does Microsoft 365 Business Premium include phishing simulation?
Not on its own. Business Premium includes Microsoft Defender for Office 365 Plan 1, and attack simulation training is a Plan 2 capability. Microsoft's service description lists it as "No" for Plan 1 and "Yes" for Plan 2, and as of 1 July 2026 the same applies to Office 365 E3 and Microsoft 365 E3. The fix does not require an E5 upgrade, though. Microsoft sells a Defender Suite for Microsoft 365 Business Premium add-on at £7.70 per user per month excluding VAT, which includes Plan 2 and therefore attack simulation training, and which Microsoft states requires a Business Premium subscription.
What is Attack Simulator, and is it the same thing?
Attack Simulator is the older name for the same capability, and Microsoft's current documentation calls it attack simulation training. Traces of the old name survive: the portal address is still security.microsoft.com/attacksimulator, and one tip in Microsoft's own documentation still refers to "Attack Simulator". If you are following an older guide that uses the old name, it is describing the same feature.
How much does a phishing simulation cost in the UK?
If you already hold Microsoft 365 E5 or Office 365 E5, nothing extra. On Business Premium, the Microsoft Defender Suite add-on is published at £7.70 per user per month, paid yearly and excluding VAT, which for a forty person business is £308 a month or £3,696 a year before VAT. That is less than half the per-seat cost of moving the same people to E5, and it brings identity and endpoint protection with it rather than simulation alone. There is also a 90 day Defender for Office 365 trial if you want to test the approach before committing to anything.
Is a phishing simulation required for Cyber Essentials?
No. Cyber Essentials covers five technical controls: firewalls, secure configuration, security update management, user access control and malware protection. None of them is about user education, and neither Cyber Essentials nor Cyber Essentials Plus requires a phishing simulation or any awareness training. If a supplier tells you otherwise, they are describing their own package rather than the scheme.
How long should a phishing simulation run?
Microsoft's default is two days, which is also the shortest you can set, and the longest is thirty. Two days is usually right for a first campaign: long enough to catch people who were out of the office, short enough that the exercise does not dominate a fortnight. Interaction stops being recorded at the end date you specify.
What is a good click rate?
There is no defensible benchmark, and any supplier quoting an industry average is selling you a comparison rather than a measurement. Your own trend across several campaigns is the only meaningful figure, and even then the reporting rate is the more useful of the two. A falling click rate alongside a falling reporting rate usually means people have learned to ignore email rather than to flag it.
Should we discipline people who fail?
No, and the NCSC is unusually direct about this. Its guidance warns that "punishing people for clicking on emails you've sent starts to resemble entrapment", that simulations "erode trust between employees and security", and that "employees who are afraid for their jobs will not report mistakes". It also advises checking with HR before running any simulation at all. Punishment breaks the reporting culture that makes the exercise worth running.
Does it work on on-premises mailboxes?
Partly. Microsoft states that attack simulation training supports on-premises mailboxes "but with reduced reporting functionality". If you run a hybrid Exchange estate, expect gaps in the results for those users, and factor that in before you draw conclusions from a tenant-wide figure.
What is the 300 user limit on the Defender Suite add-on?
The Microsoft Defender Suite for Microsoft 365 Business Premium is capped at 300 users, which is the same ceiling Microsoft applies to Business Premium itself. If you are above 300 staff you are outside the small and medium business licensing track altogether, and the route to attack simulation training is Microsoft 365 E5, Office 365 E5, or a standalone Defender for Office 365 Plan 2 arrangement your licensing partner can price.
