Yahoo Case Study: Automated Canary Analysis using Spinnaker Kayenta - Palash Agrawal
Canary Analysis is an essential part of any production deployment pipeline. As part of Yahoo/Verizon Media, we have many properties like Yahoo Homepage, Yahoo Mail, Yahoo Finance, Yahoo News, Yahoo Sports, TechCrunch, HuffPost, etc. that serve a large number of users. It is crucial to introduce changes in a consistently safe manner to minimize user impact minutes. At Yahoo/Verizon Media, we are running Spinnaker Kayenta in a Kubernetes Deployment and have integrated it with Screwdriver (https://screwdriver.cd/), an open source build platform for continuous delivery. We have extended Kayenta to support Splunk and an OpenTSDB based internal metrics service. We also built tools to facilitate easy onboarding for applications and designed a configuration syntax based on the Hashicorp Configuration Language (HCL) for running canary analysis. A deployment of Nike’s Referee (https://github.com/Nike-Inc/referee) tool is also set up to allow on-demand canary analysis. In this talk, we’ll share how we adopted, implemented and extended Spinnaker Kayenta to perform automated canary analysis in the Screwdriver CD solution used at Verizon Media.