A useful web scraper that can be used in an array of different development situations, including gathering product specifications from E-Commerce sites that do not provide an API or gathering useful information related to SEO techniques.

I had to develop a web scraper for a proof-of-concept application a few months ago in PHP and since have had a few questions on how to implement them. I decided to make a post and include the web scraper free of charge for other developers to download and utilize in their scripts.

What This Scraper Does

This scraper uses the cURL PHP library in combination with 10 of the most popular search engines to gather information. First, it takes a string as a query provided by the user and searches the internet for sites relevant to that search query. It then visits and scrapes these sites for whatever regular expression you provide to it and stores these results into an array that can be placed into a database, xml file, or anything similar.