Change Data Capture for DynamoDB Streams using Amazon Kinesis in-depth intuition
With Amazon Kinesis Data Streams for Amazon DynamoDB, you can capture item-level changes in your DynamoDB tables as a Kinesis data stream.
In this video I have explained about this concept in-depth.
Prerequisite:
---------------------
Capture and Process Item-level Changes Using DynamoDB Streams and AWS Lambda
• Capture and Process Item-level Change...
Simplify Amazon DynamoDB data extraction and analysis by using AWS Glue and Amazon Athena
• Simplify Amazon DynamoDB data extract...
Building Serverless Data Stream pipeline using Kinesis data streams and Firehose for Snowflake
• Building Serverless Data Stream pipel...
Lambda for Firehose Transformation:
------------------------------------------------------------
import json
import boto3
import base64
output = []
def lambda_handler(event, context):
print(event)
for record in event['records']:
payload = base64.b64decode(record['data']).decode('utf-8')
print('payload:', payload)
row_w_newline = payload + "\n"
print('row_w_newline type:', type(row_w_newline))
row_w_newline = base64.b64encode(row_w_newline.encode('utf-8'))
output_record = {
'recordId': record['recordId'],
'result': 'Ok',
'data': row_w_newline
}
output.append(output_record)
print('Processed {} records.'.format(len(event['records'])))
return {'records': output}
SQL Queries for DynamoDB:
----------------------------------------------
INSERT INTO "orders" value {'roll_no' : 5,'price' : 5, 'quantity': 235 }
INSERT INTO "orders" value {'roll_no' : 2,'price' : 10, 'quantity': 10 }
INSERT INTO "orders" value {'roll_no' : 3,'price' : 20, 'quantity': 100 }
INSERT INTO "orders" value {'roll_no' : 4,'price' : 30, 'quantity': 15 }
UPDATE orders SET price=90 WHERE roll_no=5
DELETE FROM "orders" WHERE "roll_no" = 3
Check this playlist for more AWS Projects in Big Data domain:
• Demystifying Data Engineering with Cl...