AWS Developers | Real-Time Streaming Data Enrichment with Database CDC | 2/5 @awsdevelopers | Uploaded 6 months ago | Updated 3 hours ago
Learn how to keep reference data up to date while simultaneously enriching your data streams, with Apache Flink. We’ll take an in-depth look at how Apache Flink streaming join works with real-time event data and the database row level, using Change Data Capture (CDC).
In this series, Anand Shah (Data Analytics and Streaming Specialist at AWS) will help you build a modern data streaming architecture for a real-time gaming leaderboard. This architecture includes data ingestion, real-time enrichment with database change data capture (CDC), data processing, as well as computing, storing and visualizing the results. You will also learn advanced streaming analytics techniques, such as the control channel method for A/B testing, updating features and parameters with zero downtime, and how to handle late arrival of data. Anand will also talk you through the process of data de-duplication, as well as how you can store historical data for replay on-demand. 🎉
🌟 Get started with Amazon Managed Service for Apache Flink today, to build and run your fully managed Apache Flink applications on AWS! 👉 aws.amazon.com/managed-service-apache-flink
🔗 Github repository: github.com/build-on-aws/real-time-gaming-leaderboard-apache-flink
Resources used in this video:
🔗 AWS CDK Overview: docs.aws.amazon.com/cdk/v2/guide/home.html
🔗 Apache Flink CDC Connectors: github.com/apache/flink-cdc
🔗 Apache Flink Joins: nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/joins
🔗 Modern Streaming Data Architecture on AWS: docs.aws.amazon.com/whitepapers/latest/build-modern-data-streaming-analytics-architectures/what-is-a-modern-streaming-data-architecture.html
Follow AWS Developers:
👾 Twitch: twitch.tv/aws
🐦 Twitter: twitter.com/awsdevelopers
💻 LinkedIn: linkedin.com/showcase/aws
Follow Anand Shah:
🐦 Twitter: twitter.com/anandshah110
💻 LinkedIn: linkedin.com/in/anandshah110
00:00 Intro
00:35 What will you learn?
01:28 What is Change Data Capture (CDC)?
02:33 Keeping Apache Flink state up-to-date
03:20 Demo: CDK source code walkthrough and deploy
06:56 Demo: Building the CDC connector and using Managed Flink Notebooks
09:16 Demo: Challenge 2 - Querying player demographics and CDC join
10:15 Conclusion
#FlinkCDC, #ManagedServiceForApacheFlink, #StateManagement
Learn how to keep reference data up to date while simultaneously enriching your data streams, with Apache Flink. We’ll take an in-depth look at how Apache Flink streaming join works with real-time event data and the database row level, using Change Data Capture (CDC).
In this series, Anand Shah (Data Analytics and Streaming Specialist at AWS) will help you build a modern data streaming architecture for a real-time gaming leaderboard. This architecture includes data ingestion, real-time enrichment with database change data capture (CDC), data processing, as well as computing, storing and visualizing the results. You will also learn advanced streaming analytics techniques, such as the control channel method for A/B testing, updating features and parameters with zero downtime, and how to handle late arrival of data. Anand will also talk you through the process of data de-duplication, as well as how you can store historical data for replay on-demand. 🎉
🌟 Get started with Amazon Managed Service for Apache Flink today, to build and run your fully managed Apache Flink applications on AWS! 👉 aws.amazon.com/managed-service-apache-flink
🔗 Github repository: github.com/build-on-aws/real-time-gaming-leaderboard-apache-flink
Resources used in this video:
🔗 AWS CDK Overview: docs.aws.amazon.com/cdk/v2/guide/home.html
🔗 Apache Flink CDC Connectors: github.com/apache/flink-cdc
🔗 Apache Flink Joins: nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/joins
🔗 Modern Streaming Data Architecture on AWS: docs.aws.amazon.com/whitepapers/latest/build-modern-data-streaming-analytics-architectures/what-is-a-modern-streaming-data-architecture.html
Follow AWS Developers:
👾 Twitch: twitch.tv/aws
🐦 Twitter: twitter.com/awsdevelopers
💻 LinkedIn: linkedin.com/showcase/aws
Follow Anand Shah:
🐦 Twitter: twitter.com/anandshah110
💻 LinkedIn: linkedin.com/in/anandshah110
00:00 Intro
00:35 What will you learn?
01:28 What is Change Data Capture (CDC)?
02:33 Keeping Apache Flink state up-to-date
03:20 Demo: CDK source code walkthrough and deploy
06:56 Demo: Building the CDC connector and using Managed Flink Notebooks
09:16 Demo: Challenge 2 - Querying player demographics and CDC join
10:15 Conclusion
#FlinkCDC, #ManagedServiceForApacheFlink, #StateManagement
