The Web Requests Plugin: A Powerful Tool for Scrapping Data from the Web
The Web Requests plugin is a powerful tool for scraping data from the web. It can be used to extract data from a variety of sources, including HTML, PDF, JSON, XML, CSV, and images. The plugin is easy to use and can be integrated with a variety of other tools and services.
Here's a basic guide on how to use it:
1. scrape_url: This is the main function of the plugin. It takes a number of parameters:
● url (required): The URL to scrape or perform a Google search if 'is_search' is set to true. When is_search is set to true, the 'url' parameter will be treated as a search query for Google.
● page (optional): The page number (of data chunks, not web pages) to retrieve, based on the page_size that was chosen. Default is 1.
● page_size (optional): The maximum number of characters that will be returned with each page (chunk). Defaults to 10000.
● is_search (optional): Indicates whether the request is a search query. If set to true, the 'url' parameter will be treated as a search query for Google. Default is false.
● follow_links (optional): Only relevant when 'is_search' is true. Indicates whether to return the content of the search result's underlying page (when true) or just the result's metadata (when false). Default is false
● num_results_to_scrape (optional): Only relevant when 'is_search' is true. The number of search results to return. Default is 3.
● job_id (optional): Job ID's are generated when we get your initial request for something new, and are returned in the response data for your reference and so you can either parse the data or paginate through the response pages/chunks.
● refresh_cache (optional): Indicates whether to refresh the cache for the content at the URL in this request. If set to true, a new request to the URL will be made and the cache will be
updated. This is useful if you're requesting an endpoint that is frequently updated. Default is
false.
● no_strip (optional): Indicates whether to skip the stripping of HTML tags and clutter. Use this flag if you want to preserve the original HTML structure, such as when specifically looking for something in source code. When 'no_strip' is set to false (by default), HTML content will be sanitized and certain tags (e.g., script and style tags) may be removed for security reasons.
Here's an example of how to use the scrape_url function:
web_requests.scrape_url({
url: "https://www.example.com",
page: 1,
page_size: 10000,
is_search: false,
follow_links: false,
num_results_to_scrape: 3,
refresh_cache: false,
no_strip: false
})
Example Prompts
Here are some example prompts that you can use to demonstrate the power of the Web Requests plugin:
Research how to get any product in Costco as a vendor and how long it usually takes.
Research https://prometheusapartments.com and list out the names of the executives in a table. Then identify which one would be best to pitch digital marketing services and craft an introduction email listing service offering with estimated prices.
Give me a list of vendors who supply food for birthday parties in Bangalore. Can you also create a table for the above result with cost of food per person?
Link to presentation: https://www.slideshare.net/AvinashRul...
Medium Article : / web-requests-chatgpt-plugin-goodbye-knowle...
Subscribe to LinkedIn Weekly Newsletter: https://www.linkedin.com/build-relati...
This video is part of the ChatGPT plugin series. To learn more, check out the following playlist: • How to add Plugins to ChatGPT