Vol. 1 · Edition 033Free · No paywall

Everyone Needs a Samwise

AI news · Synthesized · Opinionated · 🌿

Muse Spark 1.2 (Meta's eval)

82.9%

Claude Opus 5 (public leaderboard)

86.7%
Tools & Infra
By Sam Taylor with Samwise

On what Meta's internal Terminal-Bench score actually means, why the $0.10/M contributor tier changes the cost calculation for open-source maintainers, and where Muse Spark 1.2 actually lands against Opus 5

Meta's coding agent entered the market second. The contributor pricing might not care.

Source lean on this story
▲ avg

Anti-AI

00

Skeptic

01

Neutral

00

Pro (practical)

02

Pro (hyped)

00

← Anti-AI · Pro-AI →

If you're choosing a terminal coding agent today, the top of your shortlist probably has two names. Claude Code, which Anthropic ships directly and benchmarks aggressively. OpenAI Codex, which has been embedded in GitHub Copilot since 2021 and went standalone earlier this year. As of Tuesday, there's a third option — and the reason to look at it isn't the benchmark number that leads the press release.

Meta Superintelligence Labs launched Muse Code on August 5, a terminal-native coding agent for macOS and Linux powered by a new model called Muse Spark 1.2. The announcement came with benchmarks, a pricing structure, and one number that's generating the most comparisons: 82.9% on Terminal-Bench 2.1.

That number needs a footnote before it means anything.

82.9%
Muse Spark 1.2 Terminal-Bench 2.1 score — Meta's own internal evaluation; not submitted to the public leaderboard

→ Source: Meta Superintelligence Labs blog, August 5, 2026

The benchmark question

Terminal-Bench 2.1 is a coding agent benchmark that tests agentic task completion in a real shell environment — file editing, test running, codebase navigation, multi-step refactoring. The public leaderboard is maintained independently and requires submitting your model for evaluation under a standardized test harness run by the leaderboard maintainers, not the model developer.

Muse Spark 1.2's 82.9% is not from that process. Meta ran the evaluation internally, using the Terminal-Bench 2.1 task set, and published the number in the announcement post. That's a meaningful difference. What version of the task set, whether they used the full suite or a curated subset, and whether their harness matched the reference implementation — none of that is published.

Claude Opus 5 sits at 86.7% on the public leaderboard, evaluated by the leaderboard maintainers under the same conditions as every other submitted model. Whether the gap between 82.9% and 86.7% reflects a real capability difference, an evaluation methodology difference, or some mix of both is genuinely unclear from what Meta published.

The DeepSWE v1.1 result is a different story. Meta submitted to the public benchmark and got 59.3%, up from 53% on the prior Muse model version. That's independently verifiable. The directional improvement is real.

Coding agent benchmarks — public leaderboard vs. self-reported
ModelTerminal-Bench 2.1Eval sourceDeepSWE v1.1
Claude Opus 586.7%Public leaderboard
Muse Spark 1.2 (Meta)82.9%Meta internal eval59.3%
OpenAI Codex (latest)Not published

The pricing argument

Here's where Muse Code gets more interesting than the benchmark framing suggests.

Standard pricing for Muse Spark 1.2 is $1.25 per million input tokens, $4.25 per million output tokens. Competitive with Claude Sonnet 3.7, cheaper than Opus 5 — notable, but not a category-level difference.

The contributor tier is a different order of magnitude. Open-source contributors get access at $0.10 per million input tokens and $0.20 per million output tokens. That's roughly a 12× reduction on input, 21× on output, compared to standard pricing.

Why does this matter strategically? Coding agent adoption is sticky. Developers who use Muse Code for their open-source work — because it's dramatically cheaper — are learning its behaviors, its strengths, and its failure modes. When they're evaluating what to recommend or use at a job, that familiarity carries weight that a marketing comparison table doesn't.

The eligibility criteria aren't fully published yet. "Active contributor to qualifying open-source projects" is the announced framing, with specifics to come. The value of the contributor tier depends entirely on whether that definition is generous or narrow.

Source spread

What's real / What deserves a side-eye

What's real:

  • The contributor pricing is a structural market move, not a promotional gimmick. $0.10/$0.20 per million tokens is cheap enough to shift adoption decisions, particularly for developers whose open-source work isn't covered by an employer's tool budget.
  • The DeepSWE v1.1 improvement is independently verifiable. Going from 53% to 59.3% on a public benchmark is forward movement, full stop — the absolute number is secondary to the trajectory.
  • Meta has the engineering depth to iterate quickly. Muse Code v1 from a lab that has shipped Llama 3, Llama 4, and multiple production inference systems is a different starting point than v1 from a team without that track record.

What deserves a side-eye:

  • The 82.9% Terminal-Bench number has not been submitted to the independent leaderboard. The gap with Opus 5's 86.7% could be real, smaller than reported, or larger — you cannot tell from a self-reported evaluation without the methodology details.
  • "Active contributor to qualifying open-source projects" is doing a lot of work in one sentence. If eligibility is narrow — think "primary maintainer of a project with 1,000+ GitHub stars" rather than "person who merged a PR last month" — the pricing story becomes much less compelling for most developers who contribute occasionally.
  • On every benchmark where a public comparison exists, Muse Spark 1.2 trails Claude Opus 5. That's the actual competitive picture entering a market where the incumbent is both better-benchmarked and established.
For builders

What builders need to know

  • Treat the Terminal-Bench 82.9% as directional, not definitive. It's a self-reported number from an internal evaluation. Until Meta submits to the independent leaderboard, it's not directly comparable to Claude Opus 5's 86.7%. The DeepSWE v1.1 score (59.3%) is the benchmark to use for cross-model comparisons right now — it's independently verified.
  • Check contributor tier eligibility before building pricing models around it. The $0.10/$0.20 rate is a major cost difference, but eligibility criteria aren't fully published. Confirm whether your workflow qualifies before committing to an integration that depends on that price point.
  • Run your own codebase evaluation. Muse Code is macOS/Linux terminal-native — same architecture pattern as Claude Code. The questions that matter: Does it navigate your project structure correctly? Does it make the right changes on your specific stack? What's the failure rate on multi-step tasks? Benchmark scores answer none of these.
  • Standard pricing is competitive, not decisive. At $1.25/$4.25/M, Muse Spark 1.2 sits between Sonnet and Opus 5 pricing territory. If you're not contributor-eligible, the pricing argument is real but not dramatic. Quality evaluation on your actual work becomes the deciding factor.
  • Watch the leaderboard submission. Whether and when Meta submits Muse Spark 1.2 to the public Terminal-Bench leaderboard is the most important signal about competitive positioning confidence. A quick submission is bullish. A delay or a quiet non-submission is informative in the other direction.

Further reading

🌿

Liked this? Get the weekly digest.

Free. Monday mornings. The week's stories, synthesized. Unsubscribe anytime.

Your take

How'd I do on this one?

What did I miss?

Tell Samwise (and Sam).

Disagree with the take? Spotted a fact I got wrong? Have context I should have included? Drop it here. Anonymous unless you leave an email.