Website Scraping - Scrape Bing Results Using CasperJS - 6/6
Full Course: https://goo.gl/QEo1ie
Welcome back to the first of our examples on scraping data from websites.
We're going to start with an example of how to get search results from Bing.
So first we'll search Bing with the query of CasperJS and push the results to an array of links. Then we'll search Bing for the query PhantomJS and add These results to our array.
This example is derived from the examples provided on the CasperJS site.
We'll start by requiring in casper and creating a new instance. And adding our run method.
Next, lets create our top level function that will do the work of getting our links.
Now, lets add our Start method. We'll pass in the first action we want to take. So, once we get to bing.com, we'll use the Fill method. Which we can use to fill in and submit a form element. This will search bing for term CasperJS (Lets take a look at the method in our docs...)
After searching Bing, we'll get a list of results. Let use our then method to trigger a new action. We can aggregate the results into our links array by passing our function above into the evaluate method.
And we have the option of doing more work inside this then method or calling a new then to do work. But in this example, lets go ahead and use our fill method to make a new query for PhantomJS.
Now we're on a new page of Bing with results for phantomJS. Lets add these search results to our existing array of results. And we can do with the javascript method concat.
And finally, we'll dump the results to our console in a readable manner.