Text By the Bay 2015: Nima Sarshar, Using Big Data to Identify the World's Top Experts
ai.bythebay.io Nov 2025, Oakland, full-stack AI conference Scale By the Bay 2019 is held on November 13-15 in sunny Oakland, California, on the shores of Lake Merritt: https://scale.bythebay.io. Join us!
-----
In this talk, we report on our implementation of a big data system that is able to automatically identify and rank experts in a large number of categories by ingesting and analyzing millions of pieces of content published across the Web every day.
We adopt a principled approach to defining who an expert is. An expert is someone who (a) writes consistently about a small set of tightly related topics; if you are an expert in everything, you are an expert in nothing, and (b) who has a loyal following that engages with her contents consistently and finds them useful, and © who actually expresses opinions on the topics he writes about rather than merely breaking the news.
Formulating the above criteria, and implementing it at scale, is a daunting big data task. Firstly we needed to form a rather comprehensive picture of the body of works published by authors that often write on many different outlets and at times under different aliases. Secondly, we had to create a dynamic topical model that learns the relationship between tens of thousands of topics by analyzing millions of documents. Thirdly, we had to come up with a formula that results in a stable, consistent, ranking, that is robust to fluctuations in publishing patterns and engagement data, yet is adaptable to allow in for new experts and their voices to be heard.
-- Experts vs. Influencers: defining who an expert is
-- Unifying identities of authors across sites
-- A dynamic topical model that scales
-- Projection of topics onto authors
-- Opinion vs. Sentiment vs. Statement of Facts
-- Putting it all together
-- A note on architecture
As inPowered’s CTO, Nima leads the development of the core technologies at the heart of inPowered, and oversees all aspects of engineering and product development. Once a tenured professor with 50+ peer-reviewed publications, Nima left academia to found Haileo Inc., a startup specializing in visual search. He was a Principal Data Scientist at Intuit, before joining inPowered. Nima hold’s an M.Sc. from UCLA and a Ph.D. from McMaster University, both in Electrical Engineering.