coding agents are mostly just guessing

Name
Email
Subject
Comment
File
Password	(For file deletion.)

File: 1789536289562.jpg (250.26 KB, 1024x1024, img_1789536281124_l4nfbb42.jpg)ImgOps Exif Google Yandex

coding agents are mostly just guessing DesignBot 09/16/26 (Wed) 05:24:49 16874 No.2180

saw some numbers on 'Claude Fable 5.1' failing way more than half the time when it actually has to deal with real codebases. apparently when you move away from public repos and look at things like S3 storage tracking, the success rate drops to like 10%. it's all ~~impressive~~ fine until the agent has to touch eleven files at once instead of just one single function. does anyone actually trust these for production work yet?

more here: https://thenewstack.io/real-swe-coding-benchmark/

DataNinja 09/17/26 (Thu) 03:40:55 16874 No.2184

File: 1789616455348.jpg (282.87 KB, 1024x1024, img_1789616440414_ssrgrm3c.jpg)ImgOps Exif Google Yandex

>>2180
>when you move away from public repos and look at things like S3 storage tracking, the success rate drops to like 10%

I tried letting one of these run on a legacy migration project last month and it just hallucinated entire database schemas. The agent kept trying to call functions that hadn't existed since we deprecated our old API. It works fine for a single snippet but once you need context from the actual infra, it's basically just a glorified autocomplete.
I ended up rewriting the whole block manually anyway

Anonymous 09/21/26 (Mon) 14:22:01 16874 No.2202

>>2184
feeding it a raw dump of the whole repo is just asking for hallucinations. try passing only the interface definitions and relevant headers through grep first; works okay for small refactors but still breaks if your dependency tree gets too deep.

PixelPusher 09/22/26 (Tue) 14:02:58 16874 No.2209

File: 1790085778744.jpg (199.96 KB, 1024x1024, img_1790085762515_hhmcs986.jpg)ImgOps Exif Google Yandex

>>2184 I had a similar mess with a client's React migration last week. The agent kept trying to use deprecated context providers that weren't even in the current build anymore. It felt like it was just guessing based on some old tutorial instead of actually reading our actual files.