set timeout while spidering a site

Though I heavily depend on urllib2 module to develop web crawler, but sometimes the crawlers just stuck ... :(. So it's necessary to set a timeout but unfortunately urllib2 doesn't provide anything for this purpose. So we have to depend on socket module. here is the code that I use:


import socket

timeout = 300 # seconds
socket.setdefaulttimeout(timeout)

Comments

redbaron said…
What perfomance (sites/day) did you achieve with your crawlers?
September 7, 2008 at 2:36 AM
Post a Comment

Popular posts from this blog

Python all any built-in function

I have been using Python for a long time, still learning lot of simple and new things. Recently I came to know about two built-in functions named all and any . Today while I was solving a hackerrank problem , I used both functions which makes the code a bit nicer, in my opinion. The functions compares two singly linked lists and if they are equal, returns 1, otherwise returns 0. Here is my code - def compare_lists(llist1, llist2): while all([llist1, llist2]): if llist1.data != llist2.data: return 0 llist1 = llist1.next llist2 = llist2.next if any([llist1, llist2]): return 0 return 1 Sometimes, this kind of simple thing brings joy to me. Let me know your thoughts. :)
Read more

Accept-Encoding 'gzip' to make your cralwer faster

Image
If you are looking for a way to make your web crawler / spider work faster, then you must be looking for reducing the time to fetch data from the Internet. Fortunately many websites provide the content in compressed format if the server gets such type of request. If you see the request header of Firefox (you can use firebug to see the request / response headers), you can find that it always asks for gzip content. If the server provides gzip, then it sends data in gzip form otherwise it sends html. gzip data reduces bandwidth and thus make the program to scrape / harvest data faster. So you can do the same from you Python code. Just add ('Accept-Encoding', 'gzip,deflate') in the request header. Check the following code chunk: opener = urllib2.build_opener() opener.addheaders = [('Referer', referer), ('User-Agent', uagent), ('Accept-Encoding', 'gzip,deflate')] usock = opener.open(url) url = usock.geturl() data = decode(usock) usock.c...
Read more

lambda magic to find prime numbers

Find prime number up to 100 with just 3 lines of code. The 4th line is for printing ;) nums = range(2, 100) for i in range(2, 10):     nums = filter(lambda x: x == i or x % i, nums) print nums Isn't it amazing? Check my new post about Prime number: http://love-python.blogspot.com/2010/11/prime-number-generator-in-python-using.html
Read more