get content (html source) of an URL by HTTP POST method in Python
To retrieve or get content (html source) of an URL, sometimes we need to POST some values. Here I show you a sample python code that uses POST.
Hope it will be useful for you while writing web crawler/spider, specially where values must be submitted using HTTP POST method to get or extract content from an URL.
Please write your comments.
import urllib, urllib2, time
url = 'http://www.example.com' # write ur URL here
values = {'key1' : 'value1', #write ur specific key/value pair
'key2' : 'value2',
'key3' : 'value3',
}
try:
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
print the_page
except Exception, detail:
print "Err ", detail
Hope it will be useful for you while writing web crawler/spider, specially where values must be submitted using HTTP POST method to get or extract content from an URL.
Please write your comments.
Comments
Why do you think it's HTTP GET? Any reason behind this?
BTW, you can check my latest post here.
Thanks.
But if i debug the returned html. I don see any html of the list i want. The Page is the same before a i made the POST http request.
This is my code:
http://gist.github.com/625996