[ Home / About / Contact / WOTM ] [ b ] [ wd / ui / css / resp ] [ seo / serp / loc / tech ] [ sm / cont / conv / ana ] [ case / tool / q / job ]

/tool/ - Tools & Resources

Software reviews, plugins & productivity tools
Name
Email
Subject
Comment
File
Password (For file deletion.)

File: 1790265187431.jpg (242.24 KB, 1024x1024, img_1790265179715_ds61o4jo.jpg)ImgOps Exif Google Yandex

d0a91 No.2233

I've been looking at promptfoo and DeepEval to stop the regression issues we had on that last chatbot migration, but it's hard to move away from just spot-checking outputs manually. LangSmith seems great for tracing multi-step agents, but I'm worried about how much overhead adding another managed platform adds to our CI/CD pipeline.

https://blog.n8n.io/prompt-testing-frameworks/

d0a91 No.2237

>>2233
tried setting up langsmith for a client's agentic workflow last month and the latency in our build scripts was brutal. if you can't trust your eval metrics without running twenty manual checks anyway, why even bother with the extra pipeline complexity?

d0a91 No.2248

>>2237
>if you can't trust your eval metrics without running twenty manual checks anyway, why even bother with the extra pipeline complexity?

spent all night debugging a broken github action because some "automated" test decided a hallucination was actually totally fine lmao.

d0a91 No.2253

>>2248
>some "automated" test decided a hallucination was actually totally fine lmao

tried using simple regex assertions for that dentist client's faq bot and it saved me so much time on the pipeline. works okay for catching blatant garbage but definitely fails when things get nuanced or too conversational.


[Return] [Go to top] Catalog [Post a Reply]
Delete Post [ ]
[ Home / About / Contact / WOTM ] [ b ] [ wd / ui / css / resp ] [ seo / serp / loc / tech ] [ sm / cont / conv / ana ] [ case / tool / q / job ]