ClickHouse Will Let You Be Wrong at Scale

So I've posted before how I think ClickHouse is great. Its columnar storage is fast, it has amazing compression. I won't bore you with those details, you can see them all here: https://matduggan.com/clickhouse-is-winning-the-observability-wars/However all technology is great until you try to do something atypical with it. What I wanted was a realistic torture test: network firewall telemetry, on the order of 2 billion rows a day of mixed-vendor syslog, parsed at ingest into wide typed tables with a 14-day TTL. The dashboards needed to do two things at once in Grafana: fast lookup of individual log lines, and fast visualizations where you can stack ad-hoc filters on top of each other across every panel simultaneously.I learned a lot of things about ClickHouse. Also some things about myself, not all flattering. TL;DRThis ended up being significantly harder than I thought it would be, and part of that is that I made a ton of mistakes. Don't email me mean comments. I'm writing this up as…