Software Best Practices for Data Pipelines: Form Energy's Multi-Day Storage R&D Pipeline 🌳
Software Best Practices for Data Pipelines: Form Energy's Multi-Day Storage R&D Pipeline 🌳 by Mark Fickett @ Form Energy
Software best practices from apps and services are familiar: scalability, flexibility, observability, etc. Mark will describe what these mean in terms of Form Energy's data pipeline for experiment data from its multi-day storage batteries, and how Form has met these software needs with specific practices and frameworks/services implemented in Python including Dagster, AWS, and OpenTelemetry/Honeycomb.
Fossil-fuel energy is the 2nd largest source of greenhouse gasses in the US [EPA]. As we transition to a renewable grid and increasing electrification, the variability of renewable production requires multi-day storage batteries. Form is developing 100-hour iron-air batteries, a drastically cheaper alternative to 8-hour lithium batteries. Internally, our data pipeline must supply research data so we can evaluate thousands of experiments and develop these novel batteries quickly -- and the data always has to be available and correct.
This talk is part of the conference’s climate tech 🌳 series, where speakers will discuss how technology can be used to mitigate climate change and make the world a better place.
The 5th annual Python Web Conf (#PWC2023) inspired attendees to use technology for good. The conference attracted 315 attendees from over 30 countries across 15 time zones. The highly engaging format featured 70 speakers, 5 tracks (including 64 talks and 5 tutorials) and took place virtually on March 13-17, 2023 on LoudSwarm by Six Feet Up.
More information about the conference can be found at: https://2023.pythonwebconf.com