prompt leakage from test fixtures

Name
Email
Subject
Comment
File
Password	(For file deletion.)

File: 1789529290369.jpg (118.99 KB, 1024x1024, img_1789529281598_avvpvtp6.jpg)ImgOps Exif Google Yandex

prompt leakage from test fixtures CodeMonkey 09/16/26 (Wed) 03:28:10 b4b2b No.2245

im still catching myself pasting entire pytest files into chatgpt like some kind of idiot. saw this thing where someone built a python 3.11 scrubber just to redact names like alex chen before the model sees them. it basically caught the model hallucinating because it was just memorizing the assert values instead of actually parsing the logic. wonder if i could use something that simple for my own env without it breaking everything when there are actual edge cases in the strings.

https://dev.to/magickong/learn-fixture-leakage-by-building-a-tiny-prompt-scrubber-3di6

Anonymous 09/18/26 (Fri) 09:46:02 b4b2b No.2257

>>2245
>it basically caught the model hallucinating because it was just memorizing the assert values instead of actually parsing the logic

I did something similar with a client's API docs and the model just started making up endpoints that didn't exist. It was literally just repeating my dummy variables back to me as if they were real production routes. I'm worried about using a scrubber for my own work too because it feels like one wrong regex would just wreck the whole context.

ScriptKid 09/18/26 (Fri) 19:31:33 b4b2b No.2261

>>2257
same thing happened when i was messing with some sql scripts for a freelance gig. the model just started hallucinating entire table schemas based on my one dummy column name lmao. if you use that scrubber it's gonna be a nightmare for debugging because the logs won't even match what the ai is seeing.

Anonymous 09/21/26 (Mon) 10:02:29 b4b2b No.2269

File: 1789984949555.jpg (145.62 KB, 1024x1024, img_1789984934135_i5zkkq3c.jpg)ImgOps Exif Google Yandex

>>2261
the logs won't even match because of the scrubber is a bit of an exaggeration. if you're just swapping names or dummy values, the logic flow stays identical so it shouldn't mess with your ability to trace back errors. you'll definitely notice the discrepancy if you're looking for specific strings but the actual code execution path stays the same.