ClickHouse for Real-Time Observability: Querying 100M+ Events/Day in Seconds
Speaker: Jason Anderson, Skool
In this video, Jason shares how Skool, a thriving community platform, leverages ClickHouse to power their data needs. Initially, Jason was searching for a solution capable of ingesting 100M+ rows daily while delivering lightning-fast queries. After switching to ClickHouse, query times dropped from minutes in Postgres to mere seconds! Now, Skool uses ClickHouse for multiple use cases, including observability, experimental features, and behavioral analytics. And here’s a fun fact—Jason Derulo is one of the celebrity members on their platform! Tune in to hear more about their journey with ClickHouse.
https://www.meetup.com/clickhouse-los...
ClickHouse provides a high-performance analytics database for real-time observability, reducing query times on telemetry data from minutes to seconds compared to traditional Postgres-based systems. The platform serves as a real-time monitoring solution capable of ingesting over 100 million telemetry records per day from both instrumented front-end and back-end systems. This high-volume data directly powers Grafana dashboards, creating a single pane of glass for monitoring system performance and load issues. ClickHouse also enables advanced behavioral analytics and A/B testing by directly querying raw user event data. It supports funnel analysis queries to identify user drop-off points in processes like sign-up flows and seamlessly integrates with open-source experimentation tools like GrowthBook. GrowthBook runs its stats engine queries directly against ClickHouse to measure experiment outcomes, leveraging the existing telemetry data without requiring data duplication or movement to a separate platform. For optimizing query performance with observability data in ClickHouse, a critical method is to manage index cardinality. Using a function like `toStartOfHour` on a high-cardinality `DateTime64` timestamp column when it is part of the primary key significantly improves index effectiveness. Effective schema design for observability workloads requires defining the primary key and partitioning strategy upfront based on expected query patterns, as these cannot be easily changed after table creation. For cloud deployments, migrating a ClickHouse Cloud instance from a `ReplicatedMergeTree` to a `SharedMergeTree` architecture is a recommended upgrade path that can yield significant performance increases for observability workloads.