Benchmarking Opus 5 on SlopCodeBench Uh oh! There was an error while loading. Please reload this page. humanlayer / advanced-context-engineering-for-coding-agents Public Notifications You must be sig
By Coderz Club · 2026-07-28 · Tags: coding, ai, git
Benchmarking Opus 5 on SlopCodeBench
Uh oh! There was an error while loading. Please reload this page. humanlayer / advanced-context-engineering-for-coding-agents Public Notifications You must be signed in to change notification settings Fork 153 Star 2.1k FilesExpand file treemain/benchmarking-opus-5-on-slop-code-bench.mdCopy pathBlameMore file actionsBlameMore file actions Latest commit HistoryHistoryHistory288 lines (163 loc) · 23.6 KBmain/benchmarking-opus-5-on-slop-code-bench.mdCopy pathTopFile metadata and controlsPreviewCodeBlame288 lines (163 loc) · 23.6 KBRawCopy raw fileDownload raw fileOutlineEdit and raw actionsBenchmarking Opus 5 on SlopCodeBench we got better benchmarks I've written before something along the lines of: THERE ARE NO GOOD BENCHMARKS for a model's ability to maintain codebase quality That wasn't entirely true. I love nothing more than burying a good lede. Last Friday I dug into SlopCodeBench, a new-ish (March 2026) long-horizon coding benchmark from @GOrlanski's lab at UW Madison. It addresses the thing that bothers me most about coding benchmarks - that even "larger" more complex benchmarks still divulge the whole problem up front: In contrast, each challenge in SlopCodeBench has multiple "checkpoints" - the model doesn't know the whole problem up front, it has to evolve the codebase over time as new requirements are divulged. It's a good paper. It's not that long. You should read it. What's cool about this benchmark is that it is unsaturated - at the time of running, the best models available, GPT-5.4 and Opus 4.6, got 11% and 17% strict pass rates, respectively. benching opus 5 on slopcodebench On Friday I ran three claude models (Opus 4.8, Sonnet 5, and Opus 5) through a subset of SlopCodeBench and watched it live for six hours. Opus 5 wins technically but none of them did a very good job IMO. Will post more results soon with Fable and 5.6 Sol in the mix. The big headline is that Opus 5 got a 24% on the small subset of the benchmark that I ran - not much higher than Opus 4.6's 17% strict pass rate in the original paper. All of the tested models showed a pretty significant increase in verbosity, complexity, and a bunch of other code smell metrics over the course of each challenge, with Opus 5 writing five times the number of functions/callables than Opus 4.8 over the course of the same set of challenges. My personal read of this 23% pass rate is that SlopCodeBench finally gives some signal for a thing I've only ever been able to argue from vibes - that for real-shaped software engineering work, building one issue at a time, today's models can't be relied on to run lights-off without steering. the benchmark subset i had claude pick out 3 problems from the repo, 17 checkpoints total, a mix of easy/medium/hard labeled problems: circuit_eval — easy (8 checkpoints) database_migration — medium (5 checkpoints) dynamic_config_service_api — hard (4 checkpoints) There's an appendix at the end with all 17 checkpoints explained in detail but I won't put that all here. and then I ran them across all three models, in parallel, with a fresh context window per checkpoint. All models got the same prompts and ran in the claude code harness. the metric I decided I care about is the strict pass: everything new is green including every regression test that was inherited from previous checkpoints. A model fails a checkpoint if the solution has a defect - defects are detected by taking the models output, a CLI to run or in some cases e.g. an api server to poke at, and running a set of held-out black-box tests against the produced entrypoint. Model writes code for ck1 Eval harness runs black-box tests against ck1 Model writes code for ck2 Eval runs black-box tests for ck1 and ck2 etc Again, the strict pass criteria means that if a model bungles something in checkpoint 4, it can't pass the following checkpoints because that failing part of the code carries forward (unless the model indavertently fixes an eval case in checkpoint 6 that was broken in checkpoint 4, but we didn't see this happen in practice). For all 9 test runs, none of the models made it to the end of any challenge with everything passing, even on the problem marked as "easy" difficulty. while it ran sonnet's first checkpoint was more expensive but by the end of problem 1, sonnet became the cheapest of the three. (it seems like once the basics were built and the work turned into maintenance, then the cost savings started to take over) For the first challenge, last generation's models accumulated defects steadily, Opus 5 a defect each on checkpoints 4 and 5. for the first two hours opus 5 was the only model with any strict passes at all — three in a row to start. Things evolved as we went. Claude diligently updated the html. Compared against the other models, Opus 5 was technically better on problem 1 (circuit_eval). But after acing the first three checkpoints, every subsequent solution had at least one defect (failing test case). final
Uh oh! There was an error while loading. Please reload this page. humanlayer / advanced-context-engineering-for-coding-agents Public Notifications You must be signed in to change notification settings Fork 153 Star 2.1k FilesExpand file treemain/benchmarking-opus-5-on-slop-code-bench.mdCopy pathBlameMore file actionsBlameMore file actions Latest commit HistoryHistoryHistory288 lines (163 loc) · 23.6 KBmain/benchmarking-opus-5-on-slop-code-bench.mdCopy pathTopFile metadata and controlsPreviewCodeBlame288 lines (163 loc) · 23.6 KBRawCopy raw fileDownload raw fileOutlineEdit and raw actionsBenchmarking Opus 5 on SlopCodeBench we got better benchmarks I've written before something along the lines of: THERE ARE NO GOOD BENCHMARKS for a model's ability to maintain codebase quality That wasn't entirely true. I love nothing more than burying a good lede. Last Friday I dug into SlopCodeBench, a new-ish (March 2026) long-horizon coding benchmark from @GOrlanski's lab at UW Madison. It addresses the thing that bothers me most about coding benchmarks - that even "larger" more complex benchmarks still divulge the whole problem up front: In contrast, each challenge in SlopCodeBench has multiple "checkpoints" - the model doesn't know the whole problem up front, it has to evolve the codebase over time as new requirements are divulged. It's a good paper. It's not that long. You should read it. What's cool about this benchmark is that it is unsaturated - at the time of running, the best models available, GPT-5.4 and Opus 4.6, got 11% and 17% strict pass rates, respectively. benching opus 5 on slopcodebench On Friday I ran three claude models (Opus 4.8, Sonnet 5, and Opus 5) through a subset of SlopCodeBench and watched it live for six hours. Opus 5 wins technically but none of them did a very good job IMO. Will post more results soon with Fable and 5.6 Sol in the mix. The big headline is that Opus 5 got a 24% on the small subset of the benchmark that I ran - not much higher than Opus 4.6's 17% strict pass rate in the original paper. All of the tested models showed a pretty significant increase in verbosity, complexity, and a bunch of other code smell metrics over the course of each challenge, with Opus 5 writing five times the number of functions/callables than Opus 4.8 over the course of the same set of challenges. My personal read of this 23% pass rate is that SlopCodeBench finally gives some signal for a thing I've only ever been able to argue from vibes - that for real-shaped software engineering work, building one issue at a time, today's models can't be relied on to run lights-off without steering. the benchmark subset i had claude pick out 3 problems from the repo, 17 checkpoints total, a mix of easy/medium/hard labeled problems: circuit_eval — easy (8 checkpoints) database_migration — medium (5 checkpoints) dynamic_config_service_api — hard (4 checkpoints) There's an appendix at the end with all 17 checkpoints explained in detail but I won't put that all here. and then I ran them across all three models, in parallel, with a fresh context window per checkpoint. All models got the same prompts and ran in the claude code harness. the metric I decided I care about is the strict pass: everything new is green including every regression test that was inherited from previous checkpoints. A model fails a checkpoint if the solution has a defect - defects are detected by taking the models output, a CLI to run or in some cases e.g. an api server to poke at, and running a set of held-out black-box tests against the produced entrypoint. Model writes code for ck1 Eval harness runs black-box tests against ck1 Model writes code for ck2 Eval runs black-box tests for ck1 and ck2 etc Again, the strict pass criteria means that if a model bungles something in checkpoint 4, it can't pass the following checkpoints because that failing part of the code carries forward (unless the model indavertently fixes an eval case in checkpoint 6 that was broken in checkpoint 4, but we didn't see this happen in practice). For all 9 test runs, none of the models made it to the end of any challenge with everything passing, even on the problem marked as "easy" difficulty. while it ran sonnet's first checkpoint was more expensive but by the end of problem 1, sonnet became the cheapest of the three. (it seems like once the basics were built and the work turned into maintenance, then the cost savings started to take over) For the first challenge, last generation's models accumulated defects steadily, Opus 5 a defect each on checkpoints 4 and 5. for the first two hours opus 5 was the only model with any strict passes at all — three in a row to start. Things evolved as we went. Claude diligently updated the html. Compared against the other models, Opus 5 was technically better on problem 1 (circuit_eval). But after acing the first three checkpoints, every subsequent solution had at least one defect (failing test case). final