What Could Go Wrong with a Kafka JDBC Connector?
https://cnfl.io/podcast-episode-227 | Java Database Connectivity (JDBC) is the Java API used to connect to a database. As one of the most popular Kafka connectors, it’s important to prevent issues with your integration.
In this episode, we’ll cover how a JDBC connection works, and common issues with your database connection
Why the Kafka JDBC Connector?
When it comes to streaming database events into Apache Kafka®, the JDBC connector usually represents the first choice for its flexibility and the ability to support a wide variety of databases without requiring custom code. As an experienced data analyst, Francesco Tisiot (Senior Developer Advocate, Aiven) delves into his experience of streaming Kafka data pipeline with JDBC source connector and explains what could go wrong. He discusses alternative options available to avoid these problems, including the Debezium source connector for real-time change data capture.
The JDBC connector is a Java API for Kafka Connect, which streams data between databases and Kafka. If you want to stream data from a rational database into Kafka, once per day or every two hours, the JDBC connector is a simple, batch processing connector to use. You can tell the JDBC connector which query you’d like to execute against the database, and then the connector will take the data into Kafka.
The connector works well with out-of-the-box basic data types, however, when it comes to a database-specific data type, such as geometrical columns and array columns in PostgresSQL, these don’t represent well with the JDBC connector. Perhaps, you might not have any results in Kafka because the column is not within the connector’s supporting capability. Francesco shares other cases that would cause the JDBC connector to go wrong, such as:
Infrequent snapshot times
Out-of-order events
Non-incremental sequences
Hard deletes
To help avoid these problems and set up a reliable source of events for your real-time streaming pipeline, Francesco suggests other approaches, such as the Debezium source connector for real-time change data capture. The Debezium connector has enhanced metadata, timestamps of the operation, access to all logs, and provides sequence number for you to speak the language of a DBA.
They also talk about the governance tool, which Francesco has been building, and how streaming Game of Thrones sentiment analysis with Kafka started his current role as a developer advocate.
EPISODE LINKS
► Kafka Connect Deep Dive – JDBC Source Connector: https://cnfl.io/jdbc-source-connector...
► JDBC Source Connector: What could go wrong?: https://cnfl.io/jdbc-source-connnecto...
► Metadata parser: https://github.com/aiven/metadata-parser
► Debezium Documentation: https://debezium.io/documentation/ref...
► Database Migration with Apache Kafka and Apache Kafka Connect: https://aiven.io/blog/db-technology-m...
► Francesco Tisiot’s Twitter: / ftisiot
► Kris Jenkins’ Twitter: / krisajenkins
► Streaming Audio Playlist: • Streaming Audio Podcast | Apache Kafka®, C...
► Join the Confluent Community: https://cnfl.io/join-community-episod...
► Learn more with Kafka tutorials, resources, and guides: https://cnfl.io/confluent-developer-e...
► Live demo: Intro to Event-Driven Microservices with Confluent: https://cnfl.io/event-driven-microser...
► Use PODCAST100 to get $100 of free Confluent Cloud usage: https://cnfl.io/try-cloud-episode-227
► Promo code details: https://cnfl.io/podcast100-episode-227
TIMESTAMPS
0:00 - Intro
6:48 - Game of Thrones Sentiment Analysis
11:34 - Kafka Integration with JDBC Connector
16:28 - JDBC Connector – Polling Time
20:18 - Change Data Capture with Debezium
30:01 - Manage Data Flows with ksqlDB
32:41 - metadata-parser
34:54 - Tips on Getting Started with Debezium
39:22 - It's a wrap
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#jdbc #kafkaconnect #apachekafka #kafka #confluent