A scalable frontier for web crawlers
各种爬虫---大众点评,安居客，58，人人贷，拍拍贷, IT桔子,拉勾网，豆瓣,搜房网,ASO100,气象数据,猫眼电影,链家,PM25.in...
simple python implementation of web_crawler using threading and Queue
Scrapy, a fast high-level web crawling & scraping framework for Python.