How to Choose a Model for Your Agentic SOC
Back to top
November 16, 2026
How to Choose a Model for Your Agentic SOC
When organizations evaluate AI models for security operations, the first question is often which model performs best. Benchmark rankings can help answer that question, but they offer only a partial view of how a model will perform in a live security operations center (SOC).
A model that excels in controlled tests may prove too slow, expensive or unreliable when applied to real-world security incidents. Performance can also shift as new model versions emerge, changing the relative strengths of competing systems.
The Model as the Reasoning Engine Supporting AI
At the center of an AI system is the model, which interprets security alerts, evaluates supporting evidence and determines the next steps in an investigation. When given access to relevant data and security tools, the model can develop an investigative plan and decide what information to examine next.
But reasoning alone is not enough to conduct reliable security investigations. The model also depends on context, including relevant data, historical activity and other information needed to assess an alert. A harness — the software layer that coordinates the model’s work, manages tool interactions and enforces policies — determines how that reasoning is applied within the organization’s environment.
The effectiveness of an AI system in a security operations center (SOC) therefore depends not only on the model’s reasoning capabilities, but also on the quality of its context and the controls governing its execution.
In-Depth: What the Model Does
A model’s primary responsibility is to reason through a problem and to determine the next appropriate step. In a SOC, a model might need to interpret an alert, query a data source, assess results, and decide whether the evidence warrants further investigation, escalation to an analyst, or closure.
A model’s decision process and decisions themselves unfold across multiple rounds of interactions, as opposed to a single exchange. In ExtraHop’s internal testing, a typical investigative question required three to seven rounds of interaction, with a median of five, while a full investigation has a median of 39 rounds.
As an investigation progresses, the model must repeatedly process the questions, results and prior steps that make up the conversation. Each additional round can increase the amount of information the model must review, adding to processing time and cost. A model that performs only marginally slower in a single test can therefore fall significantly behind over the course of a full investigation.
For security leaders, evaluating AI models requires looking beyond the accuracy of individual answers to determine whether a model can complete an investigation within the time and cost constraints of a working SOC.
Time Is a Performance Metric
In a security operations center (SOC), the time an AI model takes to complete an investigation can be as consequential as the accuracy of its findings. An answer that arrives after an attacker has moved laterally through a network or the opportunity to contain an intrusion has passed may be accurate, but it is of limited operational value.
Over the past year, ExtraHop has tested models on live ExtraHop appliances through the company’s internal evaluation platform. Of 929 full-investigation runs, 140 (15%) failed to produce a completed answer—57 (6%) due to the model itself, and 83 due to server or provider failures.
ExtraHop measures performance by tracking wall-clock time from the initial question to the final answer. This differs from mean time to respond (MTTR), which also accounts for the work required of people after they receive a result. Across the configurations tested, the slowest took more than 20 times as long as the fastest to complete the same task.
The distinction matters for security teams evaluating AI for operational use. A model must not only reach the right conclusion, but complete the investigation within a timeframe that makes the result useful.
The Surprising Case for Slower Models
Most modern AI models offer a “thinking” mode that gives them additional time to work through a problem before responding. Across eight models that ExtraHop measured with thinking mode both on and off, enabling it roughly doubled response times (a median of 2.3 times, ranging from 1.1 to 3.5 times). Scores on the 1,600-point Agentic SOC Aptitude Test (ASAT)— over 100 tests measuring how well models investigate security incidents using network data — shifted from a 23-point drop (MiniMax M3.1 Flash, 1,510 to 1,487) to a 412-point gain, with a median of +170 and a mean of about +200.
The gains can be substantial. Across the test’s 400-to-1,600 scale, simply enabling the setting — without changing the model itself or upgrading the hardware — has lifted model scores from roughly 840 to 1,240, 1,110 to 1,280, and 1,410 to 1,535.
The biggest gains came in precision and discipline: query efficiency (+177 subscore points), accuracy (+150) and time reasoning (+135). Response planning benefited least, gaining just 32 points. Thinking mode made the models more exact in their queries and answers, but it barely changed how they planned a response.
A triage agent must investigate an alert thoroughly enough to avoid dismissing a genuine threat, while staying within predictable time and resource limits. The stakes are high: because an alert closed during triage may never receive further review, a missed threat can halt an investigation before a human analyst has a chance to examine the evidence.
In a triage situation, spending five minutes on triage rather than two may be the better choice, provided the agent remains within its assigned scope. The additional time can help uncover threats that a faster assessment might overlook, potentially preventing an active breach from going undetected.
Ultimately, the value of extended reasoning depends on whether it improves the quality of a decision enough to justify its cost, including slower response times and the operational risks of delaying threat detection and containment.
The Best Model on Paper May Not Be the Best for the Job
Benchmark rankings can obscure important differences in how models perform in production, including the effects of prompting, compression, model version and cost. In one test comparison, a top-ranked model scored 90 with a general prompt and 68 with a specific one. The specific prompt finished in 43 seconds and 24 tool calls, against 223 seconds and 78 for the general prompt. The higher score, in other words, did not translate into a more efficient investigation.
Compression introduces another trade-off between efficiency and performance. By reducing a model’s numerical precision, organizations can lower its memory and computing requirements, allowing it to run on less capable hardware. But the savings can come at the expense of accuracy, potentially undermining the very capabilities that make a model useful for security operations. In one ExtraHop test, an open-weight model scored 1,570 at 8-bit precision but fell to 1,490 at 2-bit — a decline that pushed it from near the top of the leaderboard to the middle of the pack.
The trade-off was measurable across the tested precision levels:
In one family of Google models that ExtraHop tested, versions 3.6 and 3.7 scored below 3.5 on SOC investigation tasks, and only 3.8 regained the lead. A newer release is not an automatic upgrade; it needs to be tested against the organization’s own workload.
Cost complicates the picture. Gemini 3.5 Flash scored 1,550 against 1,555 for the more expensive per-token Gemini 3.1 Pro, a gap within the test’s margin of error. Claude Opus 5.5 outscored Sonnet 5.5 by 30 points, but at 2.2 times the cost and 1.75 times the wait.
Model selection is therefore less about finding a universal winner than about knowing which trade-offs matter for a given job. A small gain in benchmark score may not justify a large increase in cost or latency and a modest configuration change can jostle a model’s standing considerably.
Why One Model Can’t Run an Entire SOC
Security operations include tasks with different requirements, and one model alone can’t serve all of them well. An agent screening alerts must be fast and consistent, while an agent reconstructing a breach must correlate evidence across several data sources over a long investigation.
Robyn Fisher, principal product marketing manager at ExtraHop, said a pricier frontier model isn’t necessarily more accurate on a simple task. Using a frontier model for every job, she said, adds cost without guaranteeing better results.
The practical answer is to match each model to a specific responsibility, then judge it on reasoning, context needs, tool efficiency, and cost against results.
What Has to Be True to Swap a Model
A replacement model earns its place by how it handles the skills the job depends on: retrieving data, reasoning through evidence, working quickly within strict limits. A model that analyzes well but can’t retrieve the data it needs is a poor substitute for the model already in production.
Routing is how organizations match models to jobs at scale. A routing system sends each question to the model best suited to answer it, preventing a single model from managing every task. Most routing is still configured by hand, but a new category of router models is starting to automate the choice.
A Model is Only as Good as Its Evidence
A security model cannot produce a reliable assessment without sufficient evidence. When an alert lacks supporting data, the model may generate a confident explanation that goes beyond what the facts establish.
In an ExtraHop test, an agent given only an alert couldn’t draw meaningful conclusions. Adding network data enabled the agent to describe the malware’s activity, while endpoint data allowed the agent to trace the source to a USB drive and produce a complete incident report.
Insufficient evidence can also lead an agent to close an alert without establishing whether a genuine threat exists. Because the decision may appear routine, an unresolved incident can be mistaken for a completed investigation.
Security leaders should evaluate how their systems collect and supply evidence and what happens when critical information is missing. If the available data cannot support a conclusion, the agent should flag the issue and defer closure until the investigation can substantiate its findings.
The Agentic Loop Around the Model Shapes the Outcome
When an agent identifies missing evidence and keeps an alert open, it is the harness — not the model alone — that determines the next course of action. Such decisions are governed by the agentic loop, the control process the harness executes around the model for each alert.
The harness sends the model the investigation context gathered to-date, then acts on the model’s recommendations.
A round consists of one complete exchange between the harness and the model. The model receives the alert and evidence collected, then requests an action, such as querying network records. The harness executes the request, evaluates the output, and returns the results to the model for further analysis.
Following each exchange, the harness determines the next investigative step: gather additional evidence, close the alert when findings are supported by underlying records, or escalate the case to a human analyst.
Two organizations can use the same model against the same data and produce different outcomes. The difference lies in how each organization configures its harness and chooses to set rules governing its agentic loop.
Discover more

Vice President, Technical Marketing
Paul Giorgi, Vice President of Technical Marketing at ExtraHop, with nearly three decades in cybersecurity sales engineering and solution architecture.
Share
Key Takeaways
- Benchmark rankings are only a starting point, since prompts, compression, and new versions can all change a model’s results.
- Speed matters as much as accuracy. A late assessment may arrive after an intrusion has already spread.
- Different SOC tasks need different models; from initial triage to incident reconstruction.
- Models reread the full investigation at every step, meaning that reading volume drives time and cost.
- Models can only identify threats from the evidence they receive. Missing evidence limits detection performance.



