Visualising and tracing requests through a cluster: Intergrating OpenTracing into OpenStack Swift
(Matthew Oliver) Making life easier for SRE's and Ops is important, so is visualising component interactions inside a cluster to see where improvements can be made to help drive development focus, and let's face it seeing graphs and visual traces is fun :)
Swift can log very verbosely but on production, especially with very large clusters you don't want to turn up your logging too much. Especially if a customer is having an issue, sometimes all you can do it come up with a hypothesis from the logs and then test in staging or dev environments. But what if you could start tagging a request through the cluster. Better, what if that trace was integrated into the software itself so we can breakdown not only the inter node requests but delve into whats happening on the node itself?
Well that's exactly what I've been playing with. What started out as middleware bench-marking, and sharing initial results with our SREs has snowballed into request tracing... and to be honest, it's pretty fun. Now we can see:
where a request spend it's time.
Start getting a visual understanding of what different requests look like in the cluster
Use the information to better tune the configuration and topology of the cluster
Find areas where we need to put more developer time to optimise different code paths.
https://lca2022.linux.org.au/schedule...
Videos licensed as CC BY-NC-SA 4.0
linux.conf.au is a conference about the Linux operating system, and all aspects of the thriving ecosystem of Free and Open Source Software that has grown up around it. Run since 1999, in a different Australian or New Zealand city each year, by a team of local volunteers, LCA invites more than 500 people to learn from the people who shape the future of Open Source. For more information on the conference see https://linux.conf.au/
Produced by Next Day Video Australia: https://nextdayvideo.com.au
#linux.conf.au #linux #foss #opensource
Sat Jan 15 14:25:00 2022 at Kia Ora Theatre