Systems @Scale 2019 - Enabling next generation models for PYMK Scale
Peter Chng, Senior Software Engineer, LinkedIn
Gaojie Liu, Staff Software Engineer, LinkedIn
https://code.fb.com/core-data/systems...
The People You May Know (PYMK) recommendation service helps LinkedIn’s members identify other members that they might want to connect to and is the major driver for growing LinkedIn’s social network. The principal challenge in developing a service like PYMK is dealing with the sheer scale of computation needed to make precise recommendations with a high recall. This talk presents the challenges LinkedIn faced when bringing its next generation of models for PYMK to production. PYMK relies on Venice, a key-value store, for accessing derived data online in order to generate recommendations. However, the increasing amount of data that had to be processed in real time with our next-generation models required us to collaborate and codesign our systems with the Venice team to generate recommendations in a timely and agile manner while still being resource efficient. Peter and Gaojie describe this journey to LinkedIn’s current solution with an emphasis on how the Venice architecture evolved to support computation at scale, the lessons learned, and the plan to tackle the scalability challenges for the next phase of growth.