Time Series Data and Stream Processing Performance Tuning
Talk 1: Time series is all hype: don’t do it
Time series data is a hot topic. In this talk we’ll showcase a workload poorly suited to time series data (experimentation) that LaunchDarkly managed to implement as time series data. We’ll dive into the performance problems that caused and take you on the journey we took to discover a more performant schema for our workload. We (still) use postgresql for this workload, so this talk will discuss postgres indexing strategies and query execution.
Talk 2: Performance tuning applications within stream processing frameworks (Flink)
When processing vast quantities of data, special attention needs to be paid to both the organization of applications in a stream processing framework and the mechanics of the processing. Key elements toward this end include hot shard mitigation, data aggregation, and event time skew handling. Profiling tools can inform whether experimental changes are yielding the desired improvements. This talk will cover some learning on each of these topics in the context of experiences with Apache Flink.