Get Original URL

Once I got into trouble while crawling some websites. Some of the URL I had wasn't the original URL, rather they were redirecting to some other URL. Then I came up with a function to get the original URL. Here I share it with you:


def get_original_url(url):
"""This function takes an url and returns the original url with cookie (if any)
"""
    cj = cookielib.CookieJar()
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
    opener.addheaders = [('User-agent', 'Mozilla/5.0')]
    usock = opener.open(url)
    url = usock.geturl()
    usock.close()
    return url, cj


Please send me your comments on this piece of code.

Comments

aatiis said…
Hi,
Could you please make it clearer how to use the CookieJar? I've coded a neat little script a few days ago that gets a random token from a website and uses urllib2 to get the cookie string: the string is response.info()['set-cookie']. I tried to use this string by adding it to the headers (Cookie: the_string), but it won't work.
So to be clear here, I have faked a user-agent, logged in, got the set-cookies header, and have no clue how to use it. Now I need to retrieve some pages using that cookie.
If you write a post about this, I'll write about your blog on mine (if it's ok). Thanks anyway.
May 11, 2008 at 2:14 PM
Tamim Shahriar said…
Thanks for your post.

I shall try to write a post about how to use cookiejar soon (hopefully by this week), so stay tuned! :)
May 11, 2008 at 9:02 PM
Mabel @ Myo Myo said…
I tried your code by using different proxies but the server responds only one html source every time different urls are requested. I want to try this but there is no solution for cookiejar. And the website I want to crawl is not required to use authentication. Please post the code or any alternatives?
March 19, 2013 at 9:13 AM
Post a Comment

Popular posts from this blog

Python all any built-in function

I have been using Python for a long time, still learning lot of simple and new things. Recently I came to know about two built-in functions named all and any . Today while I was solving a hackerrank problem , I used both functions which makes the code a bit nicer, in my opinion. The functions compares two singly linked lists and if they are equal, returns 1, otherwise returns 0. Here is my code - def compare_lists(llist1, llist2): while all([llist1, llist2]): if llist1.data != llist2.data: return 0 llist1 = llist1.next llist2 = llist2.next if any([llist1, llist2]): return 0 return 1 Sometimes, this kind of simple thing brings joy to me. Let me know your thoughts. :)
Read more

Accept-Encoding 'gzip' to make your cralwer faster

Image
If you are looking for a way to make your web crawler / spider work faster, then you must be looking for reducing the time to fetch data from the Internet. Fortunately many websites provide the content in compressed format if the server gets such type of request. If you see the request header of Firefox (you can use firebug to see the request / response headers), you can find that it always asks for gzip content. If the server provides gzip, then it sends data in gzip form otherwise it sends html. gzip data reduces bandwidth and thus make the program to scrape / harvest data faster. So you can do the same from you Python code. Just add ('Accept-Encoding', 'gzip,deflate') in the request header. Check the following code chunk: opener = urllib2.build_opener() opener.addheaders = [('Referer', referer), ('User-Agent', uagent), ('Accept-Encoding', 'gzip,deflate')] usock = opener.open(url) url = usock.geturl() data = decode(usock) usock.c...
Read more

lambda magic to find prime numbers

Find prime number up to 100 with just 3 lines of code. The 4th line is for printing ;) nums = range(2, 100) for i in range(2, 10):     nums = filter(lambda x: x == i or x % i, nums) print nums Isn't it amazing? Check my new post about Prime number: http://love-python.blogspot.com/2010/11/prime-number-generator-in-python-using.html
Read more