Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

In the rapidly evolving landscape of artificial intelligence, particularly with the advent of sophisticated Large Language Models (LLMs) and autonomous AI agents, the security domain stands at a critical juncture. We’re witnessing an explosion of capabilities, from automated vulnerability discovery to real-time threat detection. Yet, the prevailing narrative around evaluating these powerful systems often misses a critical dimension: cost. A recent paper, “Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents,” published by Paul Kassianik, Blaine Nelson, and Yaron Singer, fundamentally challenges this oversight, proposing a more pragmatic and economically relevant framework for assessment.

Executive Summary: The Economic Imperative for AI Security

The excitement around AI agents solving complex security challenges — be it red-team exploitation or blue-team incident response — is palpable. Traditional benchmarks, however, have primarily focused on peak performance under idealized conditions, often with generous computational budgets. This approach, while useful for showcasing theoretical capability, is increasingly insufficient for operational security environments. In the real world, every LLM inference call, every tool execution, every telemetry query, and every data enrichment request incurs a tangible cost, whether in compute resources, API fees, or time.

This paper argues that for AI agents to be truly impactful in security operations, we must move beyond success rate alone. We need to evaluate them through a cost-success lens. This isn’t just an academic exercise; it’s an economic imperative. Organizations deploying AI agents for security tasks need to understand not just if a system can succeed, but how much it costs to achieve that success, and whether that cost makes it practically viable. This research provides a crucial framework for answering these questions, delivering insights that redefine the conversation around practical AI security agent deployment.

Technical Deep Dive: Deconstructing Performance by Spend

The core methodology of “Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents” revolves around a granular decomposition of agent performance. Instead of merely reporting the highest achieved success rate, the authors meticulously analyze performance at fixed cost levels, breaking down the spend into two primary categories:

1. Inference Spend: The computational budget consumed by the LLM’s reasoning processes.
2. Tool Spend: The cost associated with utilizing external tools, APIs, databases, or telemetry systems.

The research applies this cost-aware lens to two distinct security challenges:

• Offensive Security: Cybench challenges, simulating Capture The Flag (CTF) scenarios focused on vulnerability discovery and exploit development.
• Defensive Security: Splunk BOTS v1 investigation challenges, mimicking Security Operations Center (SOC) incident response tasks.

The findings reveal a fascinating and critical divergence in scaling regimes for red-team versus blue-team tasks:

• Offensive (Red Team) Scaling: Performance in CTF-like offensive tasks shows a direct correlation with increased test-time compute. This means that with more reasoning budget, offensive agents, particularly those powered by advanced LLM architectures, tend to improve significantly. Intriguingly, the study demonstrates that scaled open-weight models can achieve performance comparable to frontier proprietary systems while maintaining a strong cost-competitiveness. This suggests a powerful future for democratized, high-impact offensive security tooling built on open-source Machine Learning models.

• Defensive (Blue Team) Scaling: In stark contrast, defensive SOC investigation tasks do not scale in the same linear fashion with raw reasoning budget. Success in these scenarios hinges far more heavily on disciplined tool use, efficient telemetry navigation, and selective enrichment. Throwing more compute at the problem without intelligent interaction with the environment proves inefficient. This indicates that for defensive AI agents, architectural design focusing on robust tool integration, dynamic decision-making for external queries, and contextual awareness trumps sheer model size or inference budget. It’s about working smarter, not just harder.

The paper provides an interactive website (https://evals.frontier.security) where these results can be explored, offering transparency and deeper insight into the cost-success frontiers of various models.

Real-World Applications: Optimizing Security Posture and Budget

The implications of this research are profound for anyone involved in building, buying, or deploying AI agents for security:

• For CISOs and Security Leaders: This framework offers a pragmatic way to evaluate vendor claims and make informed purchasing decisions. It shifts the focus from aspirational success rates to tangible ROI, allowing for more accurate budgeting and resource allocation for AI-powered security tools. Understanding the distinct scaling behaviors can guide investment strategies for offensive versus defensive capabilities.
• For AI/ML Engineers and Researchers: The findings provide clear guidance for future model development. For offensive capabilities, the push for more capable, efficient LLMs continues to be valuable. For defensive agents, the emphasis should shift towards robust agentic architectures that excel at tool orchestration, contextual reasoning, and efficient data retrieval, rather than simply scaling up model parameters.
• For Security Architects and Practitioners: This research provides a lens to design more effective and economically viable automated security workflows. For example, knowing that defensive agents benefit more from precise tool use, efforts can be directed towards integrating richer, more actionable telemetry and building sophisticated tool-calling interfaces.
• For Cybersecurity Product Development: Vendors can differentiate their offerings not just by raw performance but by demonstrating superior economic efficiency, especially in complex, high-volume defensive tasks.

Future Outlook: Towards Economically Rational AI Security

Looking ahead 2-3 years, the principles laid out in “Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents” are poised to become a cornerstone of AI agents development in security. We can anticipate:

• Integrated Cost-Optimization in Agent Design: Future agent architectures will likely incorporate cost-awareness directly into their planning and execution modules. Agents will learn to choose the most cost-effective path to a solution, dynamically balancing inference costs against tool utilization.
• Specialized Agent Ecosystems: A clearer bifurcation will emerge, with highly optimized “reasoning engines” for complex offensive challenges and “tool-master” agents excelling at navigating dense security environments for defensive tasks. Hybrid architectures that dynamically combine these strengths based on real-time cost signals will also gain prominence.
• Standardized Cost-Aware Benchmarking: The industry will likely adopt more sophisticated benchmarks that go beyond simple task completion, incorporating latency, reliability, and granular cost metrics as essential components of evaluation. This will drive innovation towards truly practical and deployable Machine Learning solutions.
• Adaptive Budgeting for Autonomous Security: Organizations will be able to dynamically allocate compute budgets to their LLM-powered security agents based on threat levels, operational needs, and the demonstrated cost-efficiency of specific agent behaviors.

Key Takeaways

• Operational security demands a shift from peak performance to cost-aware evaluation for AI agents.
• The paper “Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents” introduces a critical framework for measuring economic efficiency alongside task success.
• Offensive (red team) agents scale well with increased compute, with open-weight models showing strong cost-competitiveness.
• Defensive (blue team) agents rely more on disciplined tool use, telemetry navigation, and selective data enrichment than on raw reasoning budget alone.
• This research offers a pragmatic roadmap for developing, evaluating, and deploying economically viable LLM and AI agents in cybersecurity.
• Explore the interactive results at https://evals.frontier.security to delve deeper into these crucial insights.

The future of intelligent security systems isn’t just about what they can do, but what they can do efficiently and affordably. This work is a vital step toward making that future a reality.