No More Silos: Integrating Databases and Apache Kafka
No More Silos: Integrating Databases and Apache Kafka
Robin Moffatt
A presentation from ApacheCon @Home 2020
https://apachecon.com/acah2020/
Companies new and old are all recognising the importance of a low-latency, scalable, fault-tolerant data backbone, in the form of the Apache Kafka streaming platform. With Kafka, developers can integrate multiple sources and systems, which enables low latency analytics, event-driven architectures and the population of multiple downstream systems. In this talk, we’ll look at one of the most common integration requirements - connecting databases to Kafka. We’ll consider the concept that all data is a stream of events, including that residing within a database. We’ll look at why we’d want to stream data from a database, including driving applications in Kafka from events upstream. We’ll discuss the different methods for connecting databases to Kafka, and the pros and cons of each. Techniques including Change-Data-Capture (CDC) and Kafka Connect will be covered. Attendees of this talk will learn: * That all data is event streams; databases are just a materialised view of a stream of events. * The best ways to integrate databases with Kafka. * Anti-patterns of which to be aware. * The power of ksqlDB for transforming streams of data in Kafka.
Robin is a Senior Developer Advocate at Confluent, the company founded by the original creators of Apache Kafka, as well as an Oracle ACE Director (Alumnus). He has been speaking at conferences since 2009 including QCon, Devoxx, Strata, Kafka Summit, and Øredev. You can find many of his talks online at http://rmoff.net/talks/, and his blog articles at http://cnfl.io/rmoff and http://rmoff.net/. Outside of work he enjoys drinking good beer and eating fried breakfasts, although generally not at the same time.