Architecting for Big Data Processing on AWS
Let's talk a little bit about Big Data on AWS. What are my options?
Big Data on AWS - you have a lot of options. It all starts with Amazon Simple Storage Service (S3), with 11 9s of durability, where you can essentially start building a data lake where we are pulling in all your data sources into one place to be analyzed by a greater processing system. From there, you can use something like Amazon Elastic MapReduce (EMR) Service to analyze and process all that data, aggregate it, apply your data models and build your data lake with streams through Amazon Kinesis. Amazon Kineses allows you to stream data in real-time to Amazon S3 with consumers or with Kinesis Firehose what you stream to S3 out of the box. Or you can do a true real-time streaming by using Kinesis in EMR. You can you can start processing a stream in a matter of minutes with those tools available in AWS.
What do you predict is the future of Big Data on AWS?
First it makes sense to look back and figure out why they're making that investment. And I would say it all starts with this cloud adoption story where people are first moving, shutting down data centers and going into AWS and now, at their fingertips, they have all these tools at their disposal that they either can never afford or would take a ton of time and resources to have that in their data center. Now that a lot of people have adopted AWS and are going all in, it's time that they start thinking about how do I leverage this unlimited storage where I can keep data virtually forever and make predictions and extract competitive business insights. Lowering the barrier of entry for that, Amazon Machine Learning is available, so I would predict that they're going to continue to advance that and provide new algorithms that weren’t available before, things like QuickSite that AWS is coming out with, a Big Data visualization tool that’s one-tenth the cost of other big players out there. I predict that Amazon is going to just basically continue to move themselves up the stack to simplify things for the end-user.
Just over the last few months CorpInfo achieved the Big Data consulting competency from AWS, we are very honored and very proud to achieve that. That puts us in a league where we've delivered Big Data Solutions for customers. We’ve had verifiable case studies and solutions and analysis to say: this is where the customer is coming from and this is what they were able to achieve as an outcome of CorpInfo’s involvement in their Big Data initiatives.
What are some of your favorite things that you've done within the realm of Big Data for some of our clients?
You have to design something that essentially is pluggable where adding one data source is no different from adding the next source or with very minimal or low amount of friction. A Lambda Architecture is essentially the Big Data processing framework that gives you that kind of scalability and pluggable architecture. You can play around with the Lambda architecture to create its own thing: that's where you have a speed layer for streaming, fast moving data where you need real-time updates and then you have a batch layer for huge volumes of data. This is more for business reports, complex analytics - things that don't have to be in a matter of seconds. That’s where the batch layer is. That all comes together in a query layer where you can even combine those things at different points in time.
When you're creating a Big Data based architecture and you're pulling in data from hundreds of different places and leveraging third-party tools, you're opening yourself up to potential security risk How do you mitigate this risk to give our clients the confidence that there actually going to be secure even though we're enabling a mass amount of data to come in to their business?
So what do we think about when we're building a Big Data Architecture in AWS, it definitely helps to think about things more holistically, you have to really start with the business process with things like data governments and master data management to understand what data is coming in, why is it coming in, what am I using it for downstream and does it contain private data, does it need to be anonymized or so forth. So that definitely needs to be in place when it comes to data security. But technologically we're utilizing the AWS Services, that brings us again higher in the stack where AWS is responsible for a lot of the security and we don't have to think about it as much versus building our own data centers and hosting our own services. Most of AWS Services are audited by third parties for things like PCI compliance, HIPAA compliance and so forth. You can just worry about, okay, this end point, it only needs to be open to XYZ businesses and source IP addresses. We’re able to just focus and make sure that those things are in place while AWS maintains the rest.