Python Job - Web Development

Python is slowly getting popular in Bangladesh. Today I have browsed a job site and found one job advertisement for Python programmer. A company named Roots Information Technology is looking for four Python developers. Click here for the job details.

Hopefully we will find more companies looking for Python programmers in Bangladesh in neat future.

Updated python code for get html source

Yesterday I made little update to my function get_html_source() that gets the content of a page. I did so because I found that my previous function didn't support HTTP POST. Now the code supports both HTTP GET and HTTP POST. It also returns the cookiejar along with the html content of the page.

def get_html_source(url, referer = '', data = 0, cj = 0, retry_counter = 0):
    if retry_counter > 0:
        print 'Trying Again...'
    if retry_counter > 3:
        print 'Could not get source from url:', url
        return '', ''
    try:
        if cj:
            opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
        else:
            opener = urllib2.build_opener()

        opener.addheaders = [('Referer', referer),
                 ('Content-Type', 'application/x-www-form-urlencoded'),
                 ('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14'),
                 ('Accept-Encoding', 'gzip,deflate')]

        if data:
            # HTTP POST
            usock = opener.open(url, data)
        else:
            # HTTP GET
            usock = opener.open(url)
        
        content = decode(usock) # I think I have already written the code of decode function
                                # in another post. If you can't find it, just leave a comment
                                # here and I shall post the code again.
        usock.close()                
        return content, cj
    except urllib2.HTTPError, e:
        print 'The server couldn\'t fulfill the request. for url: ', url
        print 'Error code: ', e.code
        return get_html_source(url, referer, data, cj, retry_counter + 1)            
    except urllib2.URLError, e:
        print 'We failed to reach a server.'
        print 'Reason: ', e.reason
        return get_html_source(url, referer, data, cj, retry_counter + 1)
            

Please suggest any necessary update / modification of this code.

Resize image before uploading

Where Internet connection is slow, it takes too much time to upload images. Few days ago I was trying to upload some images in facebook and it seemed to take forever. The I wrote the following python program to reduce the image size (around 4MB to around 100 KB!):

import Image
import os, sys

# adjust width and height to your needs
width = 800
height = 600

for root, dirs, files in os.walk('./'):
    for name in files:       
        filename = os.path.join(root, name)
        if filename.endswith('jpg'):
            print filename
            imin = Image.open(filename)
            imout = imin.resize((width, height), Image.BICUBIC) 
            imout.save(filename)

get remote file size through http

Couple of days ago I wrote this code.

The requirement is to get the size of a remote file (http). For example: the Python program needs to find the size of the file, http://abc.com/dir/file1.mp3

Now here is a very stupid solution for this:

import urllib2

url = 'http://abc.com/dir/file1.mp3'
usock = urllib2.urlopen(url)
data = usock.read()
size = data.__len__() # size in bytes
size = size / 1024.0 # in KB (Kilo Bytes)
size = size / 1024.0 # size in MB (Mega Bytes)
...

The stupidity happens in this line: data = usock.read() where the whole file is being read to get it's size! This solution came to my mind first. But soon I understood that the file size can be found from the http response header. Here is a much better solution:

import urllib2

url = 'http://abc.com/dir/file1.mp3'
usock = urllib2.urlopen(url)
size =  usock.info().get('Content-Length')
if size is None:
    size = 0
size = float(size) # in bytes
size = size / 1024.0 # in KB (Kilo Bytes)
size = size / 1024.0 # size in MB (Mega Bytes)
...

Remove duplicate items from a list using set

A common problem beginners face is to remove duplicate items from a list. In Python it can be done easily. First make a set from the list and then make a list for that set.

myList = list(set(myList))

Here is a python example:

>>> myList = [1, 2, 3, 3, 2, 2, 4, 5, 5]
>>> myList
[1, 2, 3, 3, 2, 2, 4, 5, 5]
>>> myList = list(set(myList))
>>> myList
[1, 2, 3, 4, 5]
>>>

Let me know if you use any different technique to remove duplicates from a list.

use proxy and cookie together in Python

Here is an example code in Python that fetches an URL and writes the content to a text file. The purpose of publishing the code is to demonstrate how to use Proxy and Cookie together in an opener.

import urllib2
import cookielib

# create cookiejar to store cookie
cj = cookielib.CookieJar()

# IP:PORT
proxy_address = '66.98.208.8:3128' # change the IP:PORT, this one is for example

# create the proxy handler
proxy_handler = urllib2.ProxyHandler({'http': proxy_address})

# create opener
opener = urllib2.build_opener(proxy_handler, urllib2.HTTPCookieProcessor(cj))

# install the opener
urllib2.install_opener(opener)

# url to browse / visit
url = "http://www....com/" # change the url

req=urllib2.Request(url)

data=urllib2.urlopen(req).read()

print data

# now write the data to a text file

# create file handler
fh = open('page.txt', 'w')

data = "".join(data)

# write to file
fh.write(data)

# close the file handler
fh.close()

Hope you will find the code useful.

Py2SIS create sis file to make standalone application

You have coded the program and tested it using PyS60 emulator or in your mobile phone. Now you want to make standalone application out of your Python code to deploy in mobile phones. You have to create SIS (Symbian Installation System) file.

Format of the command is:

py2sis  [sisfile] [--uid=0x12345678] [--appname=myapp] 
[--presdk20] [--leavetemp]

I used the following command to create SIS file for my program:

F:\Nokia\Tools\Python_for_Series_60\py2sis>py2sis.exe 
dse_stock_tracker.py dse.sis --uid 0x12345678 
--appname=DSEStockPriceTracker

If you want to know details about py2sis, get the book (ebook is there) "Programming with Python for Series 60 Platform" and go to chapter 14. The book is freely available.