Key Highlights:
- Coinbase’s historical replay evaluations showed consistent performance regressions in newer LLM versions—including comparisons across Opus, Sonnet, and GPT releases—which recorded lower fraud recall and dollar-weighted recall.
- A post-trained, specialized Qwen3.5-9B model outperformed Opus 4.5 across four key fraud metrics, boosting dollar-weighted recall by 35.4 points while slashing median latency by 55%.
- Coinbase advises payment providers to thoroughly evaluate potential model upgrades under actual decision configurations before adjusting prompts, thresholds, or system architectures.
Evaluation of Frontier LLM Upgrades in Fraud Detection
In a detailed evaluation using a fixed historical replay, cryptocurrency platform Coinbase benchmarked iterations of leading frontier large language models (LLMs) to assess their effectiveness in catching payment fraud. The company compared Opus 4.5 against Opus 5, Sonnet 4.6 against Sonnet 5, and GPT-5.4 against GPT-5.6 (sol). Across these evaluations, every newer model version demonstrated regressions in key performance areas: each recorded lower recall, a reduced F1 score (the harmonic mean of precision and recall), and diminished dollar-weighted recall.
In financial risk management, recall indicates the overall share of fraudulent events an automated system successfully catches, whereas dollar-weighted recall tracks the proportion of the gross financial fraud value flagged. During the test, Sonnet’s recall dropped by 22.2 percentage points, alongside a 22.9-point drop in dollar-weighted recall. Opus’s recall registered a smaller dip of 0.8 points. Furthermore, both newer Claude-series models experienced lower precision, flagging transactions where a smaller proportion turned out to be authentic instances of fraud.
The Risk of Misleading Precision Metrics
The comparative data from the GPT series underscored how isolated metrics can mask underlying functional blind spots. While GPT’s precision increased by 11.5 percentage points, its overall recall declined by 20.7 points, and its dollar-weighted recall slipped by 21.8 points. Consequently, even though the model’s fraud flags were individually more accurate, a significantly higher volume of fraud cases and total stolen monetary value escaped detection entirely in the replay analysis.
Coinbase clarified that the historical replay does not establish customer losses resulting from deploying these newer iterations, and the company noted it could identify these performance regressions without determining their precise underlying cause. These findings follow an earlier online experiment conducted by Coinbase, which measured the impact of selective LLM review against existing fraud engines and heuristic rules alone. That agent-enabled workflow yielded a 30% reduction in fraudulent transactions and a 22% drop in total fraud value, though it did not evaluate newer model versions.
| Model Comparison | Recall Impact | Dollar-Weighted Recall Impact | Precision Impact |
|---|---|---|---|
| Sonnet 4.6 vs. Sonnet 5 | -22.2 percentage points | -22.9 percentage points | Lower |
| Opus 4.5 vs. Opus 5 | -0.8 percentage points | Lower | Lower |
| GPT-5.4 vs. GPT-5.6 (sol) | -20.7 percentage points | -21.8 percentage points | +11.5 percentage points |
Custom Open Models and Latency Advantages
In an Oct. 8 disclosure, Coinbase outlined a compelling alternative to off-the-shelf generalist models: specialized, post-trained architectures. The company evaluated a custom Qwen3.5-9B model, which ultimately outperformed Opus 4.5 across four key fraud metrics. The tuned model improved the F1 score by 9.6 percentage points and increased dollar-weighted recall by 35.4 points. Coinbase achieved this by specializing the model using historical fraud outcomes and deterministic rewards designed to balance fraudulent and legitimate examples.
In addition to standard benchmark improvements, operational latency showed significant gains. Production measurements documented that the specialized Qwen model achieved a median end-to-end LLM-request latency of 0.683 seconds, compared to 1.515 seconds for Opus 4.5—representing a 55% relative reduction in response time. However, Coinbase highlighted that the faster inference speeds and benchmark improvements were calculated across different evaluation processes. Meanwhile, researchers behind the SR-Fraud study noted that their proprietary dataset cannot be released, limiting external replication and wider generalization. Their related payment-fraud research first appeared on Sept. 23 and was revised on Sept. 30, preceding the October disclosures.
Why This Matters
For fintech firms and payment processing providers, the findings indicate that deploying cutting-edge or updated general-purpose frontier models is not guaranteed to improve downstream domain-specific performance. Instead, default upgrades can inadvertently suppress detection rates, allowing substantially more fraudulent transaction value to pass through automated systems. Coinbase advises enterprise fraud and risk teams to avoid assuming newer equals better; companies should rigorously benchmark candidate models against historical datasets within their actual production setups, while evaluating system prompts, operational costs, latency constraints, and threshold adjustments independently.
Frequently Asked Questions
Why did newer foundation models miss more fraud during Coinbase’s tests?
While the exact causes were not definitively established in Coinbase’s tests, the replay showed that newer frontier versions exhibited regressions in recall and dollar-weighted recall. In some cases, like GPT-5.6 (sol), an increase in precision meant flags were more reliable, but a broader volume and higher dollar value of fraud slipped past detection.
How did Coinbase’s specialized Qwen3.5-9B model compare to commercial proprietary LLMs?
Coinbase reported that the post-trained Qwen3.5-9B surpassed Opus 4.5 by improving the F1 score by 9.6 percentage points and boosting dollar-weighted recall by 35.4 points. It also operated with a 55% reduction in median request latency, recording 0.683 seconds versus 1.515 seconds in production measurements.
What does Coinbase recommend for payment platforms considering LLM upgrades?
Coinbase recommends that payment providers directly evaluate whether an upgrade improves coverage inside their actual decision-making setup first. Risk teams should test changed prompts or decision thresholds separately, balancing detection accuracy alongside system latency, reliability, and hosting costs.
