Scrapy selectors : XPATH 'contains' and 'starts-with' | Web Scraping Tips
Web scraping using Scrapy and Python - Some tips you may find useful.
Scrapy lets you use CSS or XPATH for the selectors, and here we look at how powerful XPATH can be when used with "contains" and "starts-with" inside the predicate to pick out parts of the page that we need to scrape.
Another benefit is that if the CSS changes on the web page our code may still pick out the parts we need.
See my code at : https://github.com/RGGH/Scrapy5
eg.
response.xpath("//a[starts-with(@href, 'https://www.yelp')]")
response.xpath('//*[contains(@href,"biz")]/@href')
Credit : Thanks to Aaron S for the suggestion/tip.
Dr P.