A totally serverless legacy to AWS cloud and Snowflake data warehouse project
Do you have data languishing in a mainframe legacy system, do you want to be able to analyse that data in a modern cloud platform, do you want to achieve that using AWS serverless technologies ? This is the talk for you ! Covers the technologies & patterns used in the end to end serverless solution.
Like many large organisations a legacy mainframe system is at the heart of the company’s IT and users would like to analyse the data in a modern platform to realise the value of the data.
ELT - Extracts the data from the legacy IBM DB2 mainframe, Loads the data into Snowflake and Transforms it using AWS services to run the workloads using serverless technologies.
Qlik replicate product running in an ECS Fargate container extracts the data from IBM hosted mainframe over AWS Direct Connect and loads it into Snowflake in near realtime over AWS PrivateLink meaning data never traverses the public internet.
AWS EventBridge schedules trigger Step Functions that call ECS Fargate containers running dbt core to transform the data using the data vault 2.0 pattern.
Snowflake Snowsight UI used to run analytical queries on the transformed Snowflake data.
GitHub runners & actions used to deploy both AWS and Snowflake Infrastructure through terrafom.
AWS EventBridge rules matching Security Hub and Inspector findings use lambda to format finding and publish to SNS for Microsft Teams alerts and email.
Microsoft Entra ID used for SAML 2.0 & SCIM integration for SSO & provisioning of end users and groups.