Tuesday, April 29, 2014

Crawling - Scrapy

What is Scrapy?

Scrapy is a fast high-level screen scraping and web crawling framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.


OS Ubuntu

Installing Dependencies 
sudo apt-get install build-essential libssl-dev libffi-dev python-dev

Install scrapy
sudo pip install Scrapy

The above scripts will install scrapy

No comments:

Post a Comment