Flinking Enrichment: Shouldn't This Be Easier? (David Anderson, Confluent) | Current 24
More: https://current.confluent.io | A common pattern in stream processing applications is to enrich incoming events with contextual information. For example, a fraud detection pipeline might enrich incoming transactions with information about the customer, the merchant, previous transactions, etc, as part of building up a feature vector to hand off to a fraud detection model.
Apache Flink offers a bewildering array of possibilities for how to do enrichment. With the SQL/Table API, a temporal join, a lookup join, or an AsyncScalarFunction can all be good choices, depending on the details of your use case, while the DataStream API offers KeyedCoProcessFunctions, BroadcastProcessFunctions, AsyncFunctions, and more. Which enrichment method you choose can be affected by where the enrichment data lives, the size of that dataset, and how often it changes, among other things.
In this talk we’ll bring structure to this landscape. You’ll learn the most useful techniques for implementing streaming enrichment with Flink, and see concrete examples for where each technique is appropriate. To keep you on track, we’ll then explore some pitfalls you may encounter, and offer best practice coping strategies for dealing with those challenges. Along the way you’ll learn something about Flink SQL, the DataStream API, and tricky corner cases with watermarks.
LEARN MORE
► Confluent Developer: https://developer.confluent.io
CONNECT
Subscribe on YouTube: / @confluentdeveloper
Community Slack: confluentcommunity.slack.com
X: https://x.com/confluentinc
Linkedin: / confluent
GitHub: https://github.com/confluentinc
Site: https://developer.confluent.io
ABOUT CONFLUENT DEVELOPER
Confluent Developer provides comprehensive resources for developers looking to learn about Apache Kafka®, Apache Flink®, Confluent Cloud, Confluent Platform, and any other technology related to the broader Data Streaming Platform. Content on Confluent Developer includes courses, getting started guides, topical deep-dives, patterns, tutorials, and listings of community events. Learn more at https://developer.confluent.io.
#apachekafka #apacheflink #confluent