Scraping the Dungeons. Find your perfect D&D game using Scrapy and Bokeh visualisation
Scraping the Dungeons. Find your perfect D&D game using Scrapy and Bokeh visualisation - Viktor Zagranovskyy - PyCon Italia 2024
Elevator Pitch:
Together we will explore the new horisons of D&D world. Where the BBEG is not always a lich or a dragon, but a lack of good company and a stable schedule. In this sesstion we will create a simple data-driven app, that will use power of Scrapy to extract the data from open web and Bokeh visualisation
Description:
🐉 Introduction to the D&D Gaming Landscape:
Embark on a journey through the mystical world of Dungeons & Dragons (D&D). This introduction sets the stage 🎭 for understanding the challenge: navigating a sea 🌊 of game data to find the perfect match for each player. We'll discuss the need for an automated, user-friendly solution here.
🕸️ Scrapy: The Data Harvesting Powerhouse:
Dive deep into the capabilities of Scrapy, a leading open-source web crawling framework, perfect for extracting unstructured data. This segment covers the basics of setting up a crawler 🤖, navigating through web pages, and extracting key game information. We emphasize ethical scraping practices to responsibly collect data 🛡️.
🧹 Data Cleansing and Transformation:
After gathering raw data, we move to cleansing and transforming it. This crucial phase involves techniques for structuring and preparing data for visualization, ensuring that our data is clean and insightful 🌟.
🎨 Bokeh: Illuminating Data with Interactive Visuals:
The highlight of our tale is Bokeh, a Python library for creating interactive and engaging visualizations. Watch as we turn the D&D data into captivating, interactive plots and dashboards 📊, allowing users to explore and filter through games to find their ideal adventure. This part includes a live demonstration 🖥️ of a Bokeh application, brimming with interactive elements like graphs, sliders, and dropdowns.
🌍 Real-World Application and Outcomes:
Although we do a small amount of data harvesting, enrichment and visualisation - we will see the powerfull and scalable base for a production-ready batteries-included solution for data-oriented businesses
💬 Q&A and Interactive Session:
Concluding with a Q&A session, this segment invites audience engagement, offering a chance to discuss and envision future advancements in web scraping in Python.
Learn more: https://2024.pycon.it/event/scraping-...
#Bokeh #BestPractice #DistributedSystems #Architecture #Scaling