Swyx 这条把 Noam Brown 的论点和开源 vs 闭源的竞争联系了起来:既然推理预算才是关键,那开源模型——每美元能跑更多 token——就应该用美元成本而非 token 数量来报告性能。
对开源模型提供商来说这是一个战略洞察:按美元成本报告会让它们的模型在对比中看起来有竞争力得多,因为每美元能换来更多思考。
这同时也给闭源实验室施加了压力——要求它们公开报告的 benchmark 数字到底用了多少 test-time compute。
Swyx 说:一个有趣的方式来践行 Noam 关于保持恒定推理预算做评估的说法——开源模型每美元的 token 里程远高于闭源 API。所以今天发布开源模型的人,显然应该按美元推理成本而非 token 数量来报告思维水平。
Swyx connects Noam Brown's argument to the open vs closed model debate: if inference budget is what matters, then open models — which have much better dollar-per-token mileage — should report performance by dollar cost, not token count.
This is a strategic insight for open model providers: reporting by dollar cost on popular inference providers would make their models look much more competitive against frontier API models, because you get more thinking per dollar.
It also pressures closed model labs to be more transparent about how much test-time compute their reported benchmark numbers actually used.
An interesting way to take Noam at his word in regards to always keeping a constant inference budget for any eval reporting — is that open models have a lot more dollar per token mileage than closed model APIs. So anyone launching an open model today or situationally incentivized toward open models should obviously report thinking levels measured by dollar inference on popular inference providers, instead of by number of tokens on the x axis