- 8 million emails over a 13 month span
- 75% of organizations are training more than 1,000 employees
- Representing organizations from US (86%) and Europe (14%)
- Representing 23 industries
Tackling a mountain of unmined data in search of answers can be a daunting task. Starting from scratch, we understood that we would likely face challenges to our pre-conceived notions of what works well and were prepared to accept what the data would tell us, however challenging it might be. Our goals were simply to understand what and how much data was available for analysis. We began with basic questions; how many scenarios are clients running? What type of scenarios are they and what do they contain? Are there any trends based on time, content, type or context?
Once the data began pointing in certain directions, we employed multiple methods of analysis including deep dives with our Cambridge and LSE partners into the statistical likelihood of repeated failures in recognition. This portion of analysis focused on a consistent set of recipients experiencing repeated phishing scenarios across 2 quarters. This tracking of the same recipient base yielded interesting findings regarding expected repeat offenses along with a baseline analysis of how long users would need to engage with education to learn from it.
As our partners dug into statistical likelihoods within the above, we internally continued to compile a vast set of average response rates based on type, industry and emotional content. This lead us to the conclusions in the report regarding which scenarios appear to be more effective and began to point to why that might be.
Finally, we tackled the larger question; does phishing training work and can we show it in our metrics? This work entailed sampling client data sets and results. From this analysis we were able to develop clearer views of reduction in scenario response rates, decreases in repeat offense and increases in user reporting of suspicious email.
By the end, we reached a better understanding of how phishing programs could be constructed for maximum measurable effectiveness. Along the way we encountered several, seemingly obvious, a-ha moments like; When a scenario is sent is not nearly as important as when it is received, or that the complexity of scenario type wasn’t as impactful as the content or context of the phish itself.
All of this and more is contained in the data analysis report. We are hopeful that it will act as a guide for your anti-phishing programs and point you in directions you may not have considered within your specific data sets. With appropriate program construction, any organization can achieve the success measures we have outlined, reduce the actual footprint of phishing threats in their environment and take positive, mitigating actions in the reduction of risk.
John ‘Lex’ Robinson
Principal Client Engagement Manager
Leslie ‘Eve’ Corbo
Senior Client Engagement Manager