Writing a distributed crawler architecture - Nenad Tičarić, TNT Studio
Recorded during Algolia Search Party - Crawling edition - June 2019, Paris
Writing a distributed crawler architecture - Nenad Tičarić, TNT Studio
While at first glance writing a crawler might seem like a simple traversal of the web graph, in praxis it gets complicated by the many demands of the system. It has to be distributed, scalable, polite, robust and extensible but still performant. This talk will cover a PHP based solution to some of those problems.