Use Proxy in your Spider

Using proxy, you can minimize the chance of getting blocked for your crawlers/spiders. Now let me tell you how to use proxy ip address in your spider in python. First load the list from a file:

fileproxylist = open('proxylist.txt', 'r')
proxyList = fileproxylist.readlines()
indexproxy = 0
totalproxy = len(proxyList)

Now for each proxy in list, call the following function:

def get_source_html_proxy(url, pip):

    proxy_handler = urllib2.ProxyHandler({'http': pip})
    opener = urllib2.build_opener(proxy_handler)
    opener.addheaders = [('User-agent', 'Mozilla/5.0')]
    urllib2.install_opener(opener)
    req=urllib2.Request(url)
    sock=urllib2.urlopen(req)
    data = sock.read()
    return data

Hope your spidering experience will be better with proxies :-)

You can use this list to test your crawler.

Comments

Nicola Montecchio said…
Hi
Very interesting article!
How do I know if a proxy is not responding? I mean, is there a simple way to set a timeout?
Thanks!
Nicola
April 7, 2008 at 6:31 PM
Tamim Shahriar said…
Yes, you can set timeout. Check this: http://love-python.blogspot.com/2008/02/set-timeout-while-spidering-site.html

Thanks.
April 7, 2008 at 7:58 PM
Nicola Montecchio said…
THANKS!!!
I'm definitely trying that.
thanks again
Nicola
April 7, 2008 at 8:06 PM
Unknown said…
This script picks a random proxy from a list, then changes your IP so you can spider it up right?
August 3, 2009 at 1:01 PM
Tamim Shahriar said…
@Heather, you are almost correct. You can choose the proxy one by one from the list or can take a random proxy. And using proxy you can hide your IP. Useful for the spiders. :)
August 3, 2009 at 9:52 PM
Unknown said…
So how exactly do i get it to choose a random proxy every time

I dont get the def url pip
September 20, 2009 at 9:09 AM
Kamil said…
Thank you! Now I can scrape sites from google without ban for ip :D
August 10, 2011 at 5:56 AM
Alock Roy said…
Hi,
I want to scrape a page using python. But the problem is when i want to go next page i have to submit a form. This form is submitted with 10 hidden values. So how i submit the form programmatically. The link is "https://www.jobs.lbhf.gov.uk/paplve_webrecruitment/wrd/run/ETREC106GF.display_srch_all?WVID=52561500BT&LANG=USA" . Please give me some suggestion or guide line.
Thanks
Sushanta
July 20, 2012 at 2:30 PM
Alock Roy said…
hi,
i want to scrape a site with python but the problem is that when i want to go the page i want to submit a form. This form is submitted with 10 hidden values. So how i submit the form programmatically. The link is https://www.jobs.lbhf.gov.uk/paplve_webrecruitment/wrd/run/ETREC106GF.display_srch_all?WVID=52561500BT&LANG=USA.please give me some suggestion or guide line.
thanks
July 20, 2012 at 2:37 PM
Post a Comment

Popular posts from this blog

lambda magic to find prime numbers

Find prime number up to 100 with just 3 lines of code. The 4th line is for printing ;) nums = range(2, 100) for i in range(2, 10):     nums = filter(lambda x: x == i or x % i, nums) print nums Isn't it amazing? Check my new post about Prime number: http://love-python.blogspot.com/2010/11/prime-number-generator-in-python-using.html
Read more

Strip HTML tags using Python

We often need to strip HTML tags from string (or HTML source). I usually do it using a simple regular expression in Python. Here is my function to strip HTML tags: def remove_html_tags(data): p = re.compile(r'<.*?>') return p.sub('', data) Here is another function to remove more than one consecutive white spaces: def remove_extra_spaces(data): p = re.compile(r'\s+') return p.sub(' ', data) Note that re module needs to be imported in order to use regular expression. Here you can find an updated code that gets the text from html: http://love-python.blogspot.com/2011/04/html-to-text-in-python.html
Read more

Accept-Encoding 'gzip' to make your cralwer faster

Image
If you are looking for a way to make your web crawler / spider work faster, then you must be looking for reducing the time to fetch data from the Internet. Fortunately many websites provide the content in compressed format if the server gets such type of request. If you see the request header of Firefox (you can use firebug to see the request / response headers), you can find that it always asks for gzip content. If the server provides gzip, then it sends data in gzip form otherwise it sends html. gzip data reduces bandwidth and thus make the program to scrape / harvest data faster. So you can do the same from you Python code. Just add ('Accept-Encoding', 'gzip,deflate') in the request header. Check the following code chunk: opener = urllib2.build_opener() opener.addheaders = [('Referer', referer), ('User-Agent', uagent), ('Accept-Encoding', 'gzip,deflate')] usock = opener.open(url) url = usock.geturl() data = decode(usock) usock.c...
Read more