Spinning Your Drones with Cadence Workflows and Apache Kafka - Paul Brebner
The rapid rise in Big Data use cases over the last decade has been accelerated by popular massively scalable open-source technologies such as Apache Cassandra® for storage, Apache Kafka® for streaming, and OpenSearch® for search. Now there’s a new member of the peloton, Uber's Cadence for code-based scalable fault-tolerant workflow orchestration. To illustrate the most important Cadence concepts (and more) we’ll build a realistic drone delivery service demonstration application, and discover how fault-tolerant workflows scale using event-sourcing and the Cassandra database (supplemented with Kafka and OpenSearch for enhanced visibility). We’ll also explore what happens when orchestration meets choreography, and use the drone application to illustrate different ways to integrate Cadence with Apache Kafka, including starting workflows from Kafka, coordinating workflows with Kafka, and reusing Kafka microservices. But how scalable is Cadence in practice? To find out, we’ll fill the sky with drones – how many drones can we get flying at once?
Paul Brebner
Open Source Technology Evangelist, Instaclustr by NetApp
Paul is the Open Source Technology Evangelist at Instaclustr (by Spot by NetApp). For the past five years, he has been learning new scalable Big Data technologies, solving realistic problems, building applications, and blogging and talking about a growing list of open source technologies including Apache Cassandra, Apache Spark, Apache Kafka, Redis, Elasticsearch, PostgreSQL, Cadence, and many more.
Since learning to program on a VAX 11/780, Paul has extensive R&D, teaching, and consulting experience in distributed systems, technology innovation, software architecture and engineering, software performance and scalability, grid and cloud computing, and data analytics and machine learning.
Paul has also worked at Waikato University (NZ), UNSW, CSIRO, UCL (UK), NICTA/ANU, and several tech start-ups (one as a Founder/CTO). Paul has an MSc in Machine Learning and a BSc (Computer Science and Philosophy).