METR 的 FrontierCode benchmark 发现 SWEBench 超过一半的结果是不可合并的 slop。FrontierCode 代表 1000 多小时维护者验证的软件工程工作,有 3000+ 评分标准覆盖代码质量和反作弊/奖励黑客。Opus 4.8 在 FC Diamond 上只有 13.8%——前沿模型连解都很难,更别说高质量地解。
Swyx 把 AI coding benchmark 分成三个时代:2021 autocomplete(HumanEval)、2023 通过测试(SWEBench、TerminalBench)、2026 可维护代码(FrontierCode)。
关键图表:FC 最简单的三分之一在 2025 年底被迅速解决——Opus 4 个月从 41% 几乎翻倍到 74%。这描述了 DHH 到 Karpathy 都提到的「2025 年 12 月到底发生了什么」的 vibe shift。每个 FC 层级将依次饱和,大概每年一个。631 赞。
METR 发现 SWEBench 一半是 slop。FrontierCode 1000+ 小时维护者验证、3000+ 评分标准。Opus 4.8 在 FC Diamond 只有 13.8%。
三个时代:Autocomplete→通过测试→可维护代码。2025 底 vibe shift 让 Opus 4 月翻倍。631 赞。
METR's FrontierCode benchmark finds that more than half of SWEBench results is unmergeable slop. FrontierCode represents 1000+ hours of maintainer-validated software engineering work with 3000+ rubrics covering code quality and anticheat/reward hacking. Opus 4.8 scores 13.8% on FC Diamond—so hard that frontier models can't yet solve, much less solve with high quality.
Swyx frames three eras of AI coding benchmarks: 2021 autocomplete (HumanEval), 2023 passing tests (SWEBench, TerminalBench), 2026 maintainable code (FrontierCode).
The key chart: the easiest third of FC tasks were rapidly solved over late 2025—Opus almost doubled from 41% to 74% in 4 months. This describes the 'WTF happened in Dec 2025' vibe shift that DHH to Karpathy have called out. Each FC tier will saturate in sequence, hopefully ~annually. 631 likes.
METR found that more than half of SWEBench results is unmergeable slop. FrontierCode represents over 1000+ hours of maintainer validated software engineering work most frontier models cannot yet solve.
Three eras of AI coding benchmarks: 2021 Autocomplete (HumanEval), 2023 Passing Tests (SWEBench, TerminalBench), 2026 Maintainable Code (FrontierCode).
The easiest third of FC tasks were rapidly solved over late 2025 - Opus almost doubled from 41% to 74% in 4 months. This describes the 'WTF happened in Dec 2025' vibe shift.