Data Platform Architecture, Evolution, and Philosophy at Netflix - @Scale 2014 - Data
Justin Becker, Senior Software Engineer at Netflix and Kurt Brown, Director of Data Platform at Netflix
The data platform at Netflix has evolved from a traditional BI stack to a modern, cloud-based architecture. Some current production components include Hadoop 2, Pig (on Tez), Hive, Presto, Teradata (Cloud), and S3 as our central data hub, along with lots of custom tooling (including Netflix open sourced techs like Inviso, Lipstick, and Aegisthus). We'll run through our data platform evolution, what our current architecture looks like and why, and the philosophical principles which drive how we get things done.