Attack Simulation Training & Security Awareness

About This Lab

Attack Simulation Training is a built-in feature of Microsoft Defender for Office 365 Plan 2 that lets security teams run realistic phishing and social engineering simulations against their own users. It supports five social engineering techniques: Credential Harvest (fake sign-in pages), Malware Attachment, Link in Attachment, Link to Malware, and Drive-by URL.

When users fail a simulation (click a phishing link or enter credentials), they are automatically assigned training modules from Microsoft’s content library covering topics like phishing awareness, password hygiene, and safe browsing. Simulation Automations allow you to schedule recurring campaigns with payload rotation so users are tested continuously. not just once per quarter. This lab walks you through configuring simulations end-to-end: selecting techniques, customizing payloads, targeting user groups, launching campaigns, analysing results, and building a long-term awareness programme.

🏢 Enterprise Use Case

Contoso Financial, a 3,000-employee financial services firm, must satisfy NIST 800-53 SA-11 and ISO 27001 Annex A.7.2.2 requirements for ongoing security awareness training. The company’s current phishing click rate is 28% based on a recent third-party assessment. well above the industry benchmark of 15%. After a real-world BEC attack cost the company $240,000, the CISO has mandated monthly phishing simulations and targeted training for repeat offenders.

Success criteria: reduce click rate below 10% within 6 months, 95% training completion, automated monthly campaigns running without manual intervention, executive-level reporting dashboards.

🎯 What You Will Learn

1. Navigate the Attack Simulation Training portal and understand the simulation lifecycle
2. Select social engineering techniques appropriate for different user populations
3. Customize phishing payloads including email templates, sender details, and landing pages
4. Configure the Advanced Delivery policy so simulations bypass MDO filters
5. Target users by department, role, or previous simulation performance
6. Launch simulations, monitor delivery, and troubleshoot non-delivery
7. Interpret simulation analytics: click rate, compromise rate, report rate
8. Assign and track training modules for users who failed simulations
9. Build Simulation Automations for recurring monthly campaigns with payload rotation
10. Create executive-level reports and KPI dashboards for security awareness programmes

🔑 Why This Matters

Over 90% of successful cyber attacks begin with a phishing email. technology alone cannot eliminate this risk. Regulations including PCI-DSS, HIPAA, SOX, NIST 800-53, and ISO 27001 require periodic security awareness training and testing. Organizations running monthly simulations consistently reduce phishing click rates by 60-80% within 12 months.

The average cost of a BEC attack is $125,000+. Attack Simulation Training is included with the E5 licence at no additional cost. Simulations combined with immediate training create a feedback loop that changes user behaviour far more effectively than annual awareness presentations.

• Completed Lab 01. Safe Links & Safe Attachments configured
• Microsoft Defender for Office 365 Plan 2 (included in Microsoft 365 E5)
• Attack Simulation Administrator or Global Administrator role
• Exchange Online mailboxes for target users (cloud-hosted, not on-premises)
• A test security group with 5-10 pilot users for initial simulation
• Advanced Delivery policy configured for the simulation sending domain

Open the Microsoft Defender portal at security.microsoft.com and navigate to Email & collaboration → Attack simulation training. The dashboard presents three main tabs:

• Overview: Aggregated metrics showing simulation completion rates, compromise rates, and training status across all campaigns
• Simulations: Where you create and manage individual simulation campaigns or view completed ones
• Simulation automations: Recurring campaign schedules with automatic payload rotation
• Content library: Payloads (email templates), login pages, training modules, and end-user notifications

Review the Recommended actions panel on the Overview tab. Microsoft provides guidance based on your simulation history and user vulnerability data. If this is your first time, the panel will recommend creating your first simulation.

# Connect to Exchange Online to verify Attack Simulation Training prerequisites
Connect-ExchangeOnline

# Verify that the Attack Simulation Administrator role is assigned
Get-ManagementRoleAssignment -RoleAssignee (Get-Mailbox -Identity $env:USERNAME).PrimarySmtpAddress |
    Where-Object { $_.Role -like "*Attack*" } |
    Format-Table Role, RoleAssignee -AutoSize

# Check that target mailboxes are cloud-hosted
Get-Mailbox -RecipientTypeDetails UserMailbox -ResultSize 10 |
    Select-Object DisplayName, PrimarySmtpAddress, RecipientTypeDetails |
    Format-Table -AutoSize

Before launching any simulation, configure Advanced Delivery so that phishing simulation emails bypass MDO filters and reach user inboxes. Without this, your own security stack will quarantine the simulation emails.

• Navigate to Email & collaboration → Policies & rules → Threat policies → Advanced delivery
• Select the Phishing simulation tab and click Edit
• Add the sending domain and sending IP addresses used by Attack Simulation Training
• For Microsoft-native simulations, the domains are automatically registered; for third-party tools, add their sending infrastructure manually
• Add any simulation URLs to the allowed list so Safe Links does not wrap or block them

# WHAT: View current Advanced Delivery configuration for phishing simulation bypass
# WHY:  Attack Simulation Training emails must bypass MDO filters (Safe Links, Safe Attachments,
#       anti-spam, anti-phishing) to reach user inboxes. Without Advanced Delivery, your own
#       security stack will quarantine the simulation emails, making the campaign fail.
# OUTPUT:
#   - PhishSimOverridePolicy: The policy object (should exist and be enabled)
#   - PhishSimOverrideRule: Lists allowed sending domains and IP ranges for simulations
#     If empty, simulations will be blocked by MDO filters.

# View current Advanced Delivery configuration
Get-PhishSimOverridePolicy | Format-List

# View phishing simulation override rules (domains and IPs allowed to bypass filters)
Get-PhishSimOverrideRule | Format-List

# If no override policy exists, create one (required for third-party simulation tools)
# Microsoft-native simulations are automatically registered; third-party tools need manual config
New-PhishSimOverridePolicy -Name "Phishing Simulation Override"

# Add sending domains and IPs for third-party simulation tools
# Domains: The From address domain used by the simulation platform
# SenderIpRanges: The mail server IPs of the simulation platform (CIDR notation)
New-PhishSimOverrideRule -Name "Simulation Domains" `
    -Policy "Phishing Simulation Override" `
    -Domains "simulation.contoso.com","phishtest.contoso.com" `
    -SenderIpRanges "203.0.113.0/24"

Before creating a simulation, plan your campaign strategy. Consider these dimensions:

• Technique selection: Start with Credential Harvest. it is the most common real-world attack and provides the clearest metric (did the user enter credentials?)
• Target audience: Begin with a pilot group of 10-20 users across departments to establish a baseline before rolling out organization-wide
• Timing: Launch simulations on weekday mornings (9-11 AM local time) when users are actively checking email and most susceptible
• Payload theme: Choose themes relevant to your industry. shipping notifications for logistics, invoice requests for finance, benefits announcements for HR
• Training assignment: Pre-select training modules that will be automatically assigned to users who fail the simulation
• Notification strategy: Decide whether to show a “you were phished” landing page immediately or let users discover the training assignment in their email

Document your campaign plan with target click-rate goals. For a first simulation, expect 20-30% click rates. Target reducing this to below 10% within 6 months through repeated simulations with targeted training.

Navigate to Simulations and click Launch a simulation. Walk through the wizard:

1. Select technique: Choose Credential Harvest. This creates a fake sign-in page that records whether users enter their credentials
2. Name your simulation: Use a descriptive name like “Q1-2026 Credential Harvest · IT Password Reset” for easy tracking
3. Select payload: Browse the built-in payload library or create a custom payload. Filter by Predicted compromise rate to select an appropriately challenging payload
4. Select login page: Choose a login page template that matches the payload theme (e.g., Microsoft 365 sign-in page for an IT-themed payload)
5. Target users: Click Add users and select your pilot security group. You can also target based on department, country, or previous simulation results
6. Assign training: Select training modules from the content library. Recommended: “How to recognize phishing” and “Protecting credentials”
7. Landing page: Configure the page users see after clicking. use the built-in “You were phished” page or create a custom branded page
8. Schedule: Choose “Launch immediately” for your first test or schedule for a specific date/time

Review the summary and click Submit. The simulation enters the Queued state and begins sending within minutes.

For more realistic simulations, create custom payloads tailored to your organization. Navigate to Content library → Payloads and click Create a payload:

• Sender details: Use a display name that employees would recognize (e.g., “IT Help Desk”, “HR Benefits Team”) with a sender address from an external look-alike domain
• Email body: Write a convincing email using social engineering principles: urgency (“Your password expires in 24 hours”), authority (“From the IT Director”), and fear (“Account will be locked”)
• Phishing link: Insert the #{phishingUrl}# dynamic tag where you want the phishing link to appear in your email template
• Landing page: Create a custom login page using the landing page editor. match your organization’s branding for maximum realism

Also create custom end-user notifications under Content library → End user notifications. These are emails sent to users after they complete (or fail to complete) assigned training.

<!-- Sample payload email body for IT Password Reset theme -->
<p>Dear #{userName}#,</p>
<p>Your Microsoft 365 password will expire in <strong>24 hours</strong>.
To avoid losing access to your email, Teams, and OneDrive,
please update your password immediately.</p>
<p><a href="#{phishingUrl}#" style="background:#0078D4;color:#fff;
padding:10px 24px;text-decoration:none;border-radius:4px">
Update Password Now</a></p>
<p>If you do not update within 24 hours, your account will be
temporarily locked and you will need to contact the IT Help Desk.</p>
<p>Thank you,<br>IT Help Desk</p>

After submitting the simulation, monitor its progress from the Simulations tab. The simulation transitions through these states:

1. Queued: Simulation is prepared and waiting to begin sending
2. In progress: Emails are being delivered to target users
3. Completed: All emails delivered and data collection window has closed

If emails are not being delivered, check these common issues:

• Advanced Delivery policy is not configured or misconfigured
• Third-party email security gateway is blocking simulation emails
• Target mailboxes have inbox rules or transport rules redirecting emails
• Users are in a group excluded from mail flow policies

# Verify simulation email delivery using message trace
Connect-ExchangeOnline

# Search for simulation emails sent in the last 24 hours
$simSender = "phishsim@simulation.contoso.com"
Get-MessageTrace -SenderAddress $simSender -StartDate (Get-Date).AddDays(-1) -EndDate (Get-Date) |
    Select-Object Received, SenderAddress, RecipientAddress, Subject, Status |
    Format-Table -AutoSize

# Check if emails were quarantined (indicates Advanced Delivery issue)
Get-QuarantineMessage -SenderAddress $simSender -StartDate (Get-Date).AddDays(-1) |
    Select-Object ReceivedTime, SenderAddress, RecipientAddress, Subject, QuarantineTypes |
    Format-Table -AutoSize

Click on the completed simulation to view its detailed results. The report provides these key metrics:

• Total users targeted: Number of users who received the simulation email
• Clicked link: Users who clicked the phishing link (click rate)
• Entered credentials: Users who submitted data on the fake sign-in page (compromise rate)
• Reported the email: Users who used the Report Message button (report rate. the metric you want to increase)
• No action taken: Users who did not interact with the email

Click on the Users tab to see individual user results. Sort by “Compromised” to identify users who need training. Export the data to CSV for record-keeping and compliance documentation.

Key performance benchmarks:

• Click rate < 10%: Good. your users can identify most phishing attempts
• Click rate 10-20%: Average. continued simulation training recommended
• Click rate > 20%: High risk. immediate and intensive training required
• Report rate > 70%: Excellent. users actively report suspicious emails

Training modules are automatically assigned to users who failed the simulation (if configured in Step 4). You can also manually assign training from the Content library → Training modules tab.

• Recommended modules for credential harvest failures: “Phishing attacks”, “How to spot a phishing email”, “Protecting your credentials”
• Recommended modules for link-based failures: “Safe browsing habits”, “Recognizing malicious URLs”
• Recommended modules for attachment failures: “Email attachment safety”, “Malware awareness”
• Training due date: Set a 7-14 day deadline for training completion. Users receive reminder notifications
• Escalation: Configure manager notifications for users who do not complete training by the due date

Monitor training progress from the Training tab. Track completion rates by department to identify groups that may need additional support or in-person training sessions.

# Export simulation results using Microsoft Graph API
# Requires Microsoft.Graph PowerShell module
Connect-MgGraph -Scopes "AttackSimulation.Read.All"

# Get all simulations
$simulations = Get-MgSecurityAttackSimulation
$simulations | Select-Object DisplayName, Status, LaunchDateTime,
    @{N='CompromisedCount';E={$_.Report.SimulationUsersCount.CompromisedCount}},
    @{N='ClickedCount';E={$_.Report.SimulationUsersCount.ClickedLinkCount}},
    @{N='ReportedCount';E={$_.Report.SimulationUsersCount.ReportedPhishCount}} |
    Format-Table -AutoSize

# Get detailed user results for a specific simulation
$simId = $simulations[0].Id
$users = Get-MgSecurityAttackSimulationSimulationUser -AttackSimulationId $simId
$users | Select-Object SimulationUser, IsCompromised, AssignedTrainingsCount,
    CompletedTrainingsCount | Export-Csv "SimulationResults.csv" -NoTypeInformation

Write-Host "Results exported to SimulationResults.csv" -ForegroundColor Green

Simulation Automations let you schedule recurring phishing campaigns that run automatically. Navigate to Simulation automations and click Create automation:

1. Name: “Monthly Phishing Awareness Campaign · All Users”
2. Technique: Select Credential Harvest and Link in Attachment to cover multiple attack vectors
3. Payloads: Select 5-10 payloads with varying difficulty levels. The automation will rotate through them each month
4. Target users: Select “All users” or a dynamic group that automatically includes new employees
5. Training: Enable automatic training assignment with a 14-day completion window
6. Schedule: Set to monthly recurrence. Randomize the send date within a 7-day window to prevent users from anticipating the simulation
7. Duration: Set the simulation to run indefinitely or for a specific period (e.g., 12 months)

Users who fail multiple simulations represent the highest risk. Create a targeted escalation programme:

• First failure: Automatic online training assignment (standard modules)
• Second failure: Additional in-depth training modules plus manager notification
• Third failure: One-on-one security awareness session with the security team plus additional account controls (e.g., step-up MFA for email access)
• Chronic repeat offenders: Restrict access to email on unmanaged devices, implement stricter Conditional Access policies, require manager approval for external email forwarding

Use the Users analytics tab to filter for users who have been compromised in two or more simulations. Create a dynamic security group for repeat offenders to target them with additional simulations and training.

# Identify repeat offenders across all simulations
Connect-MgGraph -Scopes "AttackSimulation.Read.All"

$allSimulations = Get-MgSecurityAttackSimulation -Filter "status eq 'completed'"
$compromisedUsers = @{}

foreach ($sim in $allSimulations) {
    $users = Get-MgSecurityAttackSimulationSimulationUser -AttackSimulationId $sim.Id
    foreach ($user in $users | Where-Object { $_.IsCompromised -eq $true }) {
        $email = $user.SimulationUser.Email
        if ($compromisedUsers.ContainsKey($email)) {
            $compromisedUsers[$email]++
        } else {
            $compromisedUsers[$email] = 1
        }
    }
}

# Display repeat offenders (compromised 2+ times)
$repeatOffenders = $compromisedUsers.GetEnumerator() |
    Where-Object { $_.Value -ge 2 } |
    Sort-Object Value -Descending |
    Select-Object @{N='UserEmail';E={$_.Key}}, @{N='TimesCompromised';E={$_.Value}}

Write-Host "`nRepeat Offenders (Compromised 2+ times):" -ForegroundColor Yellow
$repeatOffenders | Format-Table -AutoSize
Write-Host "Total repeat offenders: $($repeatOffenders.Count)" -ForegroundColor Cyan

Navigate to the Overview tab of Attack Simulation Training to view trend data. The dashboard shows how your organisation’s key metrics have changed over time:

• Compromise rate trend: Track the percentage of users entering credentials across all simulations. This should trend downward over months
• Report rate trend: Track the percentage of users reporting simulation emails. This should trend upward
• Training completion rate: Monitor the percentage of assigned training that is completed on time
• Simulation coverage: Percentage of organisation users who have been included in at least one simulation in the last 90 days

Compare your metrics against Microsoft’s industry benchmarks. The “Recommended actions” panel provides specific suggestions based on your data, such as increasing simulation frequency or targeting high-risk departments.

Export monthly reports capturing:

• Organisation-wide click rate and compromise rate
• Department-level breakdown
• Repeat offender count and trend
• Training completion rates by module
• Comparison to previous month and to 6-month baseline

Transform your simulation data into a formal Security Awareness Programme with executive-level KPIs and documented procedures:

• Define KPIs: Organisation-wide click rate < 10%, report rate > 70%, training completion > 95%, repeat offender rate < 3%
• Establish cadence: Monthly simulations for all users, weekly for high-risk groups (finance, executives)
• Create an executive dashboard: Monthly one-page report showing KPI trends, risk reduction, and ROI (cost of simulations vs. estimated cost of avoided incidents)
• Integrate with HR: Link repeat offender data with HR policy for escalation and accountability
• Annual programme review: Compare year-over-year click rates to demonstrate programme effectiveness for audit and compliance reporting

# Generate an executive summary report
Connect-MgGraph -Scopes "AttackSimulation.Read.All"

$simulations = Get-MgSecurityAttackSimulation -Filter "status eq 'completed'" |
    Sort-Object LaunchDateTime -Descending

Write-Host "===== SECURITY AWARENESS EXECUTIVE REPORT =====" -ForegroundColor Cyan
Write-Host "Report Date: $(Get-Date -Format 'yyyy-MM-dd')"
Write-Host "Total Simulations Run: $($simulations.Count)"
Write-Host ""

$totalTargeted = 0; $totalCompromised = 0; $totalReported = 0
foreach ($sim in $simulations) {
    $targeted = $sim.Report.SimulationUsersCount.TotalCount
    $compromised = $sim.Report.SimulationUsersCount.CompromisedCount
    $reported = $sim.Report.SimulationUsersCount.ReportedPhishCount
    $totalTargeted += $targeted
    $totalCompromised += $compromised
    $totalReported += $reported
}

$avgClickRate = if ($totalTargeted -gt 0) { [math]::Round(($totalCompromised / $totalTargeted) * 100, 1) } else { 0 }
$avgReportRate = if ($totalTargeted -gt 0) { [math]::Round(($totalReported / $totalTargeted) * 100, 1) } else { 0 }

Write-Host "KEY PERFORMANCE INDICATORS:" -ForegroundColor Yellow
Write-Host "  Average Compromise Rate : $avgClickRate%" -ForegroundColor $(if ($avgClickRate -lt 10) { 'Green' } elseif ($avgClickRate -lt 20) { 'Yellow' } else { 'Red' })
Write-Host "  Average Report Rate     : $avgReportRate%" -ForegroundColor $(if ($avgReportRate -gt 70) { 'Green' } elseif ($avgReportRate -gt 40) { 'Yellow' } else { 'Red' })
Write-Host "  Total Users Tested      : $totalTargeted"
Write-Host "  Total Compromised       : $totalCompromised"
Write-Host ""

# Recent simulation trend (last 6)
Write-Host "RECENT SIMULATION TREND:" -ForegroundColor Yellow
$simulations | Select-Object -First 6 | ForEach-Object {
    $rate = if ($_.Report.SimulationUsersCount.TotalCount -gt 0) {
        [math]::Round(($_.Report.SimulationUsersCount.CompromisedCount / $_.Report.SimulationUsersCount.TotalCount) * 100, 1)
    } else { 0 }
    Write-Host "  $($_.LaunchDateTime.ToString('yyyy-MM-dd')) | $($_.DisplayName.Substring(0,[Math]::Min(40,$_.DisplayName.Length))) | Click Rate: $rate%"
}

What You Accomplished

• Navigated the Attack Simulation Training portal and configured prerequisites
• Set up Advanced Delivery policy to ensure simulation emails bypass MDO filters
• Created and launched a Credential Harvest simulation with custom payloads
• Analysed simulation results including click rate, compromise rate, and report rate
• Assigned targeted training modules to users who failed the simulation
• Created automated monthly simulation campaigns with payload rotation
• Built a repeat offender escalation programme
• Established executive reporting and KPI tracking for your security awareness programme

Next Steps

• Next Lab: Configure Anti-Phishing & Zero-Hour Auto Purge
• Run simulations using all five social engineering techniques to cover the full threat landscape
• Integrate simulation data with Microsoft Sentinel for correlation with real phishing incidents
• Expand the programme to cover vishing (voice phishing) and smishing (SMS phishing) awareness