get content (html source) of an URL by HTTP POST method in Python

To retrieve or get content (html source) of an URL, sometimes we need to POST some values. Here I show you a sample python code that uses POST.


import urllib, urllib2, time

url = 'http://www.example.com' # write ur URL here

values = {'key1' : 'value1', #write ur specific key/value pair
'key2' : 'value2',
'key3' : 'value3',
}

try:
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
print the_page
except Exception, detail:
print "Err ", detail

Hope it will be useful for you while writing web crawler/spider, specially where values must be submitted using HTTP POST method to get or extract content from an URL.

Please write your comments.

Comments

ajaksu said…
Uh, no, that's still an HTTP GET.
Tamim Shahriar said…
I think it's HTTP POST. And I tested it.
Why do you think it's HTTP GET? Any reason behind this?

BTW, you can check my latest post here.
Unknown said…
Excellent code! But LOL what is the time module for?
Tamim Shahriar said…
time module was used for another purpose in the script that I didn't show here. :)
LimitCracker said…
Could you extend it so it shows how to make a POST request with Cookies in order to retrieve the html code of a protected page?
Tamim Shahriar said…
There is another post that shows how to use cookies. Please search.

Thanks.
Anonymous said…
I'm trying to retrieve a List of result submniting a form (POST). And the scraping it with BeautifulSoup.

But if i debug the returned html. I don see any html of the list i want. The Page is the same before a i made the POST http request.

This is my code:
http://gist.github.com/625996

Popular posts from this blog

Strip HTML tags using Python

lambda magic to find prime numbers

Convert text to ASCII and ASCII to text - Python code