Promscale: Using Prometheus + Promscale + PostgreSQL to go from Observation to Understanding DoK #51
https://go.dok.community/slack
https://dok.community/
With:
Matvey Arye - Promscale team lead, TimescaleDB
Bart Farrell - Presenter
ABSTRACT OF THE TALK
Often when I talk about putting observability data into PostgreSQL people ask me: are you crazy? And yet this somewhat heretical view has the potential to unlock a lot of the power and promise of observability. Thanks to TimescaleDB (an extension to PostgreSQL), storing time series, metric data inside of a relational database is now efficient, fast, and scalable. This is thanks to its unique partitioning, compression, and horizontal-scalability features. But even if this is possible to do, why would you? The answer to that lies in the power of a flexible data model, joins, and SQL (which Promscale supports in addition to PromQL). A flexible data model allows you to combine metric data with various other data - from machine information such as the number of cores, memory, etc. to location information using GPS coordinates. This allows you to enrich your metrics with supplemental information using joins and performing much more sophisticated analysis using SQL for capacity analysis, BI, and more. A flexible data model brings us to our second heretical idea: combining multiple modalities of observation in a single database. Combining metrics, logs, traces, event data, etc. in one DB has two major advantages: the first being a similar analytical advantage to what is described above: the ability to join and cross-correlate various types of signals together. The second major advantage is operational simplicity. As we all know, databases are the hardest things in our infrastructure to maintain and operationalize because of that pesky thing called state. So why maintain multiple different types of database systems if you could maintain just one? While these ideas about observability data on Kubernetes may seem unusual and counter-intuitive, I hope they will generate interest and start a good conversation.
BIO
Mat has been working on data infrastructure in both academia and industry for the past decade. Currently, he is leading the Promscale team, to make it easy for people to store and analyze their Prometheus data in both PromQL and SQL. Previously, he completed his Ph.D. at Princeton and then worked as one of TimescaleDB's core architects where he concentrated on performance, scalability, and query power.
About Data on Kubernetes Community (DoKC)
DoKC is an independent tech community where practitioners go to run data workloads on Kubernetes. It is managed by Constantia.io. We produce a range of resources for end users looking to run DoK including:
DoK Landscape
An interactive landscape of products and services for running data workloads on Kubernetes. (https://dok.community/landscape/)
Videos and Blogs
Hundreds of videos and blogs including how tos, tutorials, use cases, case studies, and more. (https://dok.community/landscape/) (https://dok.community/blog/)
DoK Reports
DoKC works with an independent research firm to annually produce the industry’s only DoK-focused report. (https://dok.community/dok-reports/)
Livestreams and Local Meetups
Weekly livestreams on DoK-related topics including demos, tutorials, visionary talks, panels, and more. (https://www.meetup.com/data-on-kubern...)
DoK Days @ KubeCon
We host hybrid events twice a year for people to get together and learn in-person or online. Details are published on our main website. (https://dok.community/)
Connect With Us:
https://dok.community/
/ @dokcommunity
/ dokcommunity
/ data-on-kubernetes-community
https://github.com/dokc