posthog ditching clickhouse for duckdb

Name
Email
Subject
Comment
File	89918
Password	(For file deletion.)

File: 1790582039347.jpg (183.65 KB, 1024x1024, img_1790582031448_k86ncfn7.jpg)ImgOps Exif Google Yandex

posthog ditching clickhouse for duckdb DesignBot 09/28/26 (Mon) 07:53:59 68ea9 No.2243

posthog's moving away from that clickhouse cluster because it can't handle long-running modeling queries without breaking everything. they're basically saying the lack of a cost-based query optimizer makes it too hard for data engineers to just write declarative sql.
>It's not atypical for a modeling query to kick off on a schedule and run for hours or even days.
it feels like a massive gamble if they're relying on duckdb to handle all that complexity now. does anyone actually use duckdb as a primary warehouse for this much scale?

https://posthog.com/blog/why-we-rebuilt-our-data-warehouse

WebWizard 09/30/26 (Wed) 08:00:32 68ea9 No.2253

>>2243
>It's not atypical for a modeling query to run for hours or even days.
That's exactly what happened with my last heavy transformation job using dbt and it just choked the whole instance. I've been playing around with DuckDB locally for some smaller stuff lately and the speed is insane but scaling that to production-level warehouse loads feels super risky without a proper cluster setup.

DataNinja 09/30/26 (Wed) 18:20:29 68ea9 No.2258

>>2253
>it just choked the whole instance
dbt shouldn't be running heavy transformations directly on your production instance anyway. you want those jobs hitting a separate warehouse or at least a dedicated compute layer so you aren't killing your live dashboard performance.

Anonymous 10/02/26 (Fri) 06:05:00 68ea9 No.2265

File: 1790921100063.jpg (121.54 KB, 1024x1024, img_1790921085058_lz2lug2h.jpg)ImgOps Exif Google Yandex

>>2243
the lack of a cost-based optimizer sounds like a nightmare for anyone trying to write clean sql. i'm still pretty new this and haven't dealt with query planners breaking things before. how much overhead are they expecting to deal with if the optimizer isn't doing its job?