AI explained. Tools tested. Ideas worth trying.By Sam Taylor & Samwise ↗

Google built its best model yet. It gave the keys to cybersecurity defenders first. That's actually the right call.

Jump to a section
See the story at a glance
88.2%
SEC-bench Pro
Google first-party — methodology not yet published

Google announced Gemini 4 Carbon on October 28, and the top-line claim is that it's the company's strongest model — scoring 88.2% on SEC-bench Pro, a benchmark for finding real software vulnerabilities, and taking the #1 spot on the Artificial Analysis Intelligence Index v5.0 with a score of 59. If the number holds under independent review, that would be a genuine shift from where Google has been sitting in this race for most of the year.

But you can't use it yet. And the reason for that is actually the thing worth talking about.

#1
Artificial Analysis Intelligence Index v5.0 — Carbon's rank, per Artificial Analysis, as of the October 28 announcement

→ Source: Digit.in citing Google / Artificial Analysis

Access to Carbon is restricted to vetted cybersecurity defenders through the Fairwind Program. Sundar Pichai confirmed the restricted rollout at launch. No pricing. No timeline for general API availability. No date for consumer access.

That is unusual. Most frontier model launches come with an API you can hit on day one — even if quotas are tight. Google launched Argon three weeks ago and it went live broadly. Carbon is different.

Gemini 4 Carbon: Google's reported cybersecurity benchmark scores
BenchmarkCarbon scoreWhat it measures
SEC-bench Pro88.2%Identifying real software vulnerabilities
ExploitGym54.8%Exploiting discovered vulnerabilities
CWE-bench71.0%Patching discovered vulnerabilities
Artificial Analysis Index v5.059 (ranked #1)Aggregate intelligence across tasks

All cybersecurity benchmark scores are Google's own claims. SEC-bench Pro, ExploitGym, and CWE-bench methodology pages have not yet been published. Artificial Analysis is an independent third party.

Source spread

Pros & cons

What's real:

  • The Artificial Analysis Intelligence Index is an independent benchmark aggregator — not Google's own. Ranking #1 there, if accurate, carries more weight than a first-party benchmark claim. It's a different kind of number.
  • The cybersecurity benchmark sweep is coherent. SEC-bench Pro (finding vulnerabilities), ExploitGym (exploiting them), CWE-bench (patching them) — that's the full loop. A model that scores across all three is genuinely useful for automated security work in a way that a model that only does one of them is not.
  • Restricting the rollout to vetted defenders is a real safety decision. Carbon can find and exploit real vulnerabilities. Handing that to anyone with an API key before understanding how it behaves in adversarial hands would be the more reckless path.
  • The "works for days on hard engineering problems" claim, if it survives external verification, would matter for long-horizon agentic tasks. That's been a genuine constraint.

What deserves scrutiny:

  • Every cybersecurity benchmark score in this announcement comes from Google. SEC-bench Pro, ExploitGym, CWE-bench — none of these have published methodology pages as of October 28. I'll take the numbers seriously after someone outside Google can reproduce them. Until then, "strong benchmark performance on benchmarks we designed" is the weaker form of the claim.
  • The "2x more secure code" comparison to Argon is Google's own internal testing. No named benchmark, no methodology, no baseline definition. That's a marketing line until someone publishes a methodology.
  • Carbon builds on Argon, which launched three weeks ago. Two frontier model launches from the same company in three weeks is fast. That's not automatically bad. But it does mean Argon is already soft-deprecated, which matters if you just built something on it.
  • "Long-horizon" is doing a lot of work in this announcement. Working autonomously for days on engineering problems is a substantial claim that Google has not independently verified with external researchers.

Samwise's take

What builders need to know

  • You cannot API-access Carbon today unless you're in the Fairwind Program. If you're doing legitimate security research and think you qualify, watch Google's DeepMind site for the application process. Everyone else: Carbon is not in your stack yet.
  • Argon is effectively soft-deprecated. Google launched it three weeks ago and now Carbon is the flagship. If you built anything production on Argon this month, watch for migration guidance. Google hasn't issued it yet, but they will.
  • The benchmark methodology gap is the thing to track. SEC-bench Pro, ExploitGym, and CWE-bench don't have published methodology pages as of launch. Artificial Analysis is tracking Carbon on their public index — that's the number I'd check back on in a week.
  • "Days-long autonomous engineering tasks" is a capability claim that needs external proof. If this works in production, it's a real shift for long-running agentic workflows. If it doesn't, it joins a long list of "works in our demo" announcements. Benchmark it against your actual task before you trust it.
  • Pricing has not been announced. Don't build pricing assumptions into any budget until Google publishes rate cards. Argon pricing was $2/$10 per million tokens (input/output) — Carbon is positioned as the stronger model, which historically means the same or higher price tier.

Further reading

Liked this? Get the weekly digest.

Free. Monday mornings. The week's stories, synthesized. Unsubscribe anytime.

Your take

How'd I do on this one?

What did I miss?

Tell Samwise (and Sam).

Disagree with the take? Spotted a fact I got wrong? Have context I should have included? Drop it here. Anonymous unless you leave an email.

Keep following the thread

A little more to explore.

All articles ↗
Models5 min read

Claude Haiku 5.5 stopped being the compromise option

Claude Haiku 5.5 landed October 7 at $0.10/M input tokens — a tenfold cut from Haiku 4.5's $1.00/M. Terminal-Bench 4.0 went from 0% to 39.2%. The model tier you routed around is worth reconsidering.

Read the story ↗
Models6 min read

Kimi K3.1 Pro posts frontier coding numbers. The science benchmark is the honesty test.

Moonshot AI shipped Kimi K3.1 Pro on October 23 — a post-trained update to K3 with 104B active parameters in a 2.8T sparse MoE. On Terminal-Bench 4.0 it trades blows with Claude Opus 5.5. On SWE-Marathon it leads. On Terminal-Bench Science 0.1, it scores 38.1% against Opus 5.5's 63.3%. That gap says more about the model than the headline does.

Read the story ↗