get remote file size through http

Couple of days ago I wrote this code.

The requirement is to get the size of a remote file (http). For example: the Python program needs to find the size of the file, http://abc.com/dir/file1.mp3

Now here is a very stupid solution for this:

import urllib2

url = 'http://abc.com/dir/file1.mp3'
usock = urllib2.urlopen(url)
data = usock.read()
size = data.__len__() # size in bytes
size = size / 1024.0 # in KB (Kilo Bytes)
size = size / 1024.0 # size in MB (Mega Bytes)
...

The stupidity happens in this line: data = usock.read() where the whole file is being read to get it's size! This solution came to my mind first. But soon I understood that the file size can be found from the http response header. Here is a much better solution:

import urllib2

url = 'http://abc.com/dir/file1.mp3'
usock = urllib2.urlopen(url)
size =  usock.info().get('Content-Length')
if size is None:
    size = 0
size = float(size) # in bytes
size = size / 1024.0 # in KB (Kilo Bytes)
size = size / 1024.0 # size in MB (Mega Bytes)
...

Comments

Anonymous said…
Hi. Good idea to use the header. One word of warning though: a webpage does not have to set the content-length, it's not mandatory
November 28, 2008 at 2:19 AM
Unknown said…
Hi, your solution still isn't very good as you're still putting unnecessary load on the server. Instead of an HTTP GET request, use HTTP HEAD which will return the same response header without actually transmitting any part of the file. That's exactly the purpose of HEAD.
August 6, 2009 at 11:41 AM
Tamim Shahriar said…
Steve / Marina, thanks for your comments.Can you give the code example to call HTTP HEAD In Python?
August 6, 2009 at 1:49 PM
Ben said…
This comment has been removed by the author.
October 12, 2009 at 12:56 PM
Ben said…
#Call HTTP HEAD in Python
import httplib
conn=httplib.HTTPConnection("www.abc.com")
conn.request("HEAD", "/dir/file1.mp3")
res=conn.getresponse()
fileSize=res.getheader('content-length')
#or res.getheaders() for all headers
conn.close()
October 12, 2009 at 12:58 PM
Post a Comment

Popular posts from this blog

Strip HTML tags using Python

We often need to strip HTML tags from string (or HTML source). I usually do it using a simple regular expression in Python. Here is my function to strip HTML tags: def remove_html_tags(data): p = re.compile(r'<.*?>') return p.sub('', data) Here is another function to remove more than one consecutive white spaces: def remove_extra_spaces(data): p = re.compile(r'\s+') return p.sub(' ', data) Note that re module needs to be imported in order to use regular expression. Here you can find an updated code that gets the text from html: http://love-python.blogspot.com/2011/04/html-to-text-in-python.html
Read more

lambda magic to find prime numbers

Find prime number up to 100 with just 3 lines of code. The 4th line is for printing ;) nums = range(2, 100) for i in range(2, 10):     nums = filter(lambda x: x == i or x % i, nums) print nums Isn't it amazing? Check my new post about Prime number: http://love-python.blogspot.com/2010/11/prime-number-generator-in-python-using.html
Read more

Convert text to ASCII and ASCII to text - Python code

Python has a very simple way to convert text to ASCII and ASCII to text. To find the ASCII value of a character use the ord() function. Example: >>> ord('B') 66 To get a character from it's ASCII value use the chr() function. Example: >>> chr(65) 'A' Here is a program to get the list of all characters along with their values: # program to print the list of characters and their ASCII values for value in range(0, 255): print "ASCII Value:", value, "\t", "Character:", chr(value), "\n" Here is another simple ASCII encoding-decoding program: # Convertion from text to ASCII codes message = raw_input("Enter message to encode: ") print "Decoded string (in ASCII):" for ch in message: print ord(ch), print "\n\n" # Convertion from ASCII codes to text message = raw_input("Enter ASCII codes: ") decodedMessage = "" for item in message.split(): decodedMessage += chr
Read more