Notes: MITRE ATT&CK Round 6 Results - SentinelOne Still Leads, Palo Alto Still Underappreciated, Microsoft Okay, & Consolidation Continues
Summary
- The MITRE ATT&CK Round 6 results highlight the performance of endpoint security vendors, revealing strengths and weaknesses in real-time detection and protection capabilities.
- PANW and S lead in autonomous, AI-driven threat detection, while MSFT lags behind, relying heavily on configuration changes and generating excessive alerts.
- CRWD's unexpected dropout raises questions about its product's scalability and automation, suggesting deeper architectural issues and reliance on manual intervention.
- The endpoint security market is consolidating, with fewer vendors expected in the future, driven by the shift from traditional AV to EDR and XDR solutions.
The MITRE ATT&CK Round 6 results were recently released, providing an important benchmark for evaluating the performance of various endpoint security players against their marketing claims. This round represents the most challenging test to date, as MITRE incorporated real-world attacks from some of the most advanced and the most successful hacking groups, including DPRK, CL0P, and LockBit. Protection tests were also included, and, as expected, some vendors dropped out due to their lack of real-time protection capabilities, thus participating in only detection and visibility testing.
With the rise in difficulty, a significant number of vendors initially joined the test, only to drop out before the final results were published. As a result, Round 6 serves as another valuable data point to expose which players were unable to stand up to the challenge.
Here’s a quick summary of the results:
- PANW has seen impressive growth, with its focus on autonomous agents and data engineering paying off significantly, outpacing competitors like MSFT, CRWD, and others.
- S remains the top performer across nearly all fronts, though it did suffer a 100% false positive rate in the LockBit test.
- MSFT still offers a solid product but doesn’t stand out, performing on par with the average.
- CRWD dropped out unexpectedly at the last stage, citing a plausible reason for its exit.
- Weaker players like ESTC, IBM, FTNT, RPD, and others also dropped out after the first stage of testing, highlighting their vulnerability.
- Overall, the vendor count decreased from 29 in Round 5 to 19 in Round 6, with two of the exits driven by M&A — Carbon Black (now owned by AVGO) and Secureworks (acquired by Sophos).
Vendor List
Although MITRE ATT&CK is an independent and objective test, its primary goal is to help elevate the performance of all vendors, rather than ranking or evaluating them in terms of superiority. As a result, extracting actionable insights from MITRE's test results can be challenging. Additionally, its recent GUI visualization is poorly designed, making it difficult to compare more than three vendors on a single screen. Users are required to repeatedly select vendors when switching between tests and scenarios, which adds unnecessary friction to the analysis process. Furthermore, MITRE doesn't provide all individual vendor results in a JSON or CSV file. It only provides the entire cohort's overall data in JSON/CSV.
Hence, we will briefly visualize things we find important here.
Source: Raw data from MITRE, adapted by Convequity
It’s interesting to observe several vendors dropped out of the MITRE ATT&CK test, and this clearly reveals which companies are simply playing marketing games versus those aiming for true best-of-breed (BoB) solutions.
ESTC (Elastic) once positioned itself as a next-gen SIEM poised to replace incumbents like Splunk. It also hoped to rival EDR players with better contextualization. However, ESTC’s approach comes from a different direction. Lacking agent knowhow and relying solely on log collection, protection is a significant weak point for the company. The test results and its eventual dropout underscore a critical point: security is a deep, specialized field, and a new entrant can’t simply tick the feature boxes and claim to have a robust EDR/XDR product.
Similarly, RPD’s (Rapid7) attempt to expand into EDR is driven by its maturing vulnerability management (VM) market. As a weaker player in the VM space, RPD dropped out of both the detection and protection testing stages, while its VM rival, QLYS (Qualys), stayed in for both. While QLYS stays in the game, it generates the highest number of alerts, which essentially makes the product unusable. Users have shared similar frustrations, citing that while the solution works, it’s not very user-friendly. However, unlike ESTC and RPD, QLYS is at least transparent, releasing its test results. This suggests a willingness to face the issues head-on and improve, rather than hiding behind poor performance.
Cylance, once owned by BB (Blackberry), was acquired by Arctic Wolf, an MSSP focused on automating human labor in security operations. While Arctic Wolf had strong momentum between 2020 and 2022, its growth has since slowed, and its IPO has been delayed multiple times. This could be attributed to its lack of a fully tech-driven, standardized software focus. In contrast, PANW (Palo Alto Networks) is working on 100% software-enabled automation with a small fraction of IT services. Arctic Wolf, however, appears to be a hybrid of IT services and 50% software-enabled automation. Cylance’s drop out in Round 6 reflects its ongoing struggles, we’ve anticipated for a long while, and Artic Wolf's shortcomings appear to have exacerbated them.
Some startups also dropped out, including Uptycs and Deep Instinct. Deep Instinct has bet heavily on reinforcement learning (RL) and deep learning (DL) to deliver superior endpoint security. Unlike OpenAI, however, Deep Instinct is still in the early, tedious R&D phase. We were impressed by its ability to hire Lane Bess, the former PANW CEO and ZS COO, who played a pivotal role in scaling these cybersecurity leaders. Initially focused entirely on protection, Deep Instinct received early test feedback that showed good protection rates, albeit with high false positives. In recent quarters, it seems the company is pushing for faster GTM success, realizing it can’t wait for years to perfect its AI-driven endpoint protection. EDR/XDR has become a necessity, but like many newcomers, Deep Instinct is struggling to rapidly build a BoB EDR solution.
