life is short - you need Python!

About Us Page Finder in Python

2011-12-01T02:32:00.000+06:00

Few months ago, I wrote a Python code that tries to find the about us page given a URL. You can test it here: http://208.43.85.103/cgi-bin/aboutus/main.py

I have used some heuristics from my observation to write the code. But as I have started to learn machine learning (http://www.ml-class.org), I am thinking to apply machine learning technique and rewrite this. I shall try to do it later, may be last week of December. Before doing this, I would like to improve my current program by incorporating more logic (but no ml technique). I shall publish the current algorithm soon here, but first it would be great if you can test it and give me some feedback and suggestions.

Pythonic way to convert a list into dictionary

2011-10-29T03:28:00.000+06:00

Here is a simple Python code that converts a list into a dictionary.

 def list_to_dict(li):  
     dct = {}  
     for item in li:  
         if dct.has_key(item):  
             dct[item] = dct[item] + 1  
         else:  
             dct[item] = 1  
     return dct  

 li = [1, 1, 1, 2, 3, 3, 4, 4, 4, 4, 4, 5, 6, 7, 7]  
 print list_to_dict(li)

Now I am looking for more Pythonic way to do this task. Any ideas?

Facebook scripting in Python

2011-09-06T13:28:00.000+06:00

I just came across this blog that shows an easy way to play with facebook api using python. It is

micro api client that implements the client-side authentication flow and has a few utility functions for accessing the graph api and fql.

But unfortunately I am getting error while trying with this code:


from urllib import urlretrieve
import imp
urlretrieve('https://raw.github.com/gist/1194123/fbconsole.py', '.fbconsole.py')
fb = imp.load_source('fb', '.fbconsole.py')

fb.AUTH_SCOPE = ['stream_publish']
fb.authenticate()

status = fb.graph_post("/me/feed", {"message":"Hello this is from python"})

Interactive Python Book

2011-08-30T09:24:00.000+06:00

An online interactive edition of the book How to Think Like a Computer Scientist (by Elkner et. al) is being developed by Brad Miller and David Ranum. I think it is going to be a great online resource to learn python, specially for the beginners. Students can do homework right in the browser (yes, and that's why for this ebook you will require html5 supported modern browser). The site is developed using Google App Engine and you will need a gmail account to access the ebook. They used Python 3 instead of Python 2.x in the book, which is a good thing in my opinion. So you can check the book here: http://thinkcspy.appspot.com.

The Zen of Python

2011-08-21T14:44:00.000+06:00

Well, if you are not sure about what I am talking, start the Python interpreter. Then import this, press enter and read the output. :)


subeen@subeen-Satellite-L300:~$ python
Python 2.7.1+ (r271:86832, Apr 11 2011, 18:05:24) 
[GCC 4.5.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import this

HTML to TEXT in Python

2011-04-29T03:07:00.001+06:00

I just wrote a small Python program. In the script there was a part where I needed to get the body of a web page and get rid of all the html tags, javascript, css styles, html comments etc. So I searched Google, found several threads in stackoverflow and then found this: http://www.aaronsw.com/2002/html2text/ This looks cool. But when I tested it against the 'about me' page of my blog, it didn't work because of some broken tags! Then I started to write the html to text function myself to get the plain text only. With help of regular expression I solved my problem (but may be I created more problems!). Here is my Python code:

def html_to_text(data):
    # remove the newlines
    data = data.replace("\n", " ")
    data = data.replace("\r", " ")

    # replace consecutive spaces into a single one
    data = " ".join(data.split())

    # get only the body content
    bodyPat = re.compile(r'< body[^<>]*?>(.*?)< / body >', re.I)
    result = re.findall(bodyPat, data)
    data = result[0]

    # now remove the java script
    p = re.compile(r'< script[^<>]*?>.*?< / script >')
    data = p.sub('', data)

    # remove the css styles
    p = re.compile(r'< style[^<>]*?>.*?< / style >')
    data = p.sub('', data)

    # remove html comments
    p = re.compile(r'')
    data = p.sub('', data)

    # remove all the tags
    p = re.compile(r'<[^<]*?>')
    data = p.sub('', data)

    return data

Note that in order to use the function, you need to remove some space characters, just copy-paste won't work. Guess why. ;)

Please share if you have better ideas or know useful libraries that can perform better than my code. Don't forget to test it first! :)

Replace consecutive whitespace with a single space

2011-04-16T02:50:00.001+06:00

Sometimes we need to replace consecutive whitespace in a string with a single space. This is a good practice while parsing html files. Let me show you two ways of doing this.

First one is to split the string and join. Here is the code snippet:

>>> s = "a      b       c d             e f"
>>> " ".join(s.split())
'a b c d e f'

You can check more string methods here: http://docs.python.org/release/2.5.2/lib/string-methods.html

Second method is to use regular expression. Here is the code:

>>> import re
>>> s = "a      b       c d             e f"
>>> p = re.compile(r'\s+')
>>> data = p.sub(' ', s)
>>> data
'a b c d e f'

Create Facebook Application in Python using App Engine

2011-04-03T23:17:00.000+06:00

So far I have used PHP for all the facebook applications that I (and my other team members) have developed. Today I was thinking about using Python to develop a facebook app. After Google search I found some links and at the same time I asked one of my ex-colleagues whether he used Python for any facebook application (as I knew that he was exploring Google App Engine and also Facebook app development). Then he told me that there is already a sample application in facebook that uses Python and most interestingly Google app engine! The application is named 'Run With Friends'. Though I have seen this page before but never looked at it closely. So, I think this page is the right place to get started creating facebook app in Python and GAE: https://developers.facebook.com/docs/samples/canvas/.

The project I am planning to do might take 4 to 6 months (if I get regular free time). Let me know if you have already done any interesting facebook app using Google app engine.

fast way to get input from stdin - Python

2010-11-28T01:11:00.000+06:00

I am looking for faster way for taking input from stdin in Python. So far I found three methods. input() - which is suitable for integers, raw_input() - reads the input as string and another one is sys.stdin.readline(). So far I have found sys.stdin.readline() is the fastest. Do you have any idea to read input (from console) in Python which is faster?

Python program to find the next palindrome

2010-11-25T06:16:00.001+06:00

Yesterday I wrote a Python program to find the next palindrome to solve this problem from SPOJ. My first accepted solution took four seconds to run. Today I improved the code and now it takes 1.13 seconds. In the rank-list I found people solving this problem with Python program that finishes execution in less than half seconds. Wondering whether I am missing any trick or not using Python properly. :-S

The problem is simple.

A positive integer is called a palindrome if its representation in the decimal system is the same when read from left to right and from right to left. For a given positive integer K of not more than 1000000 digits, write the value of the smallest palindrome larger than K to output. Numbers are always displayed without leading zeros.

As the number can be as large as 1000000 digits, stupid solutions won't finish it's execution in time (thus will get 'Time Limit Exceeded').

I am not going to explain my logic in detail, as I don't want to deprive you of the fun of solving problem. What I basically did was to divide the number into two parts, did some comparison between the corresponding characters (digits) and applied some logics. As soon as I realize that there is some change which made the number larger than it was, I get out of the loop. I don't make any change to the second part of the string, as it's a palindrome, I can generate it from the first part.

If you want to solve the problem, don't forget to test your program using the following numbers:


0
9
808
2133
199
1999
319887788993
333321
111
1111
999
9999
99999

Here is my code:

def palin(s):        
    word_len = len(s)
    if word_len == 1:
        if s == "9":
            return "11"
        else:
            return str(int(s)+1)                
    
    if word_len % 2 == 0:        
        m = ''
    else:         
        m = s[word_len/2]
    
    h1 = (word_len / 2) - 1           
    h2 = (word_len + 1) / 2
    
    i = h1
    j = h2
    
    if m:    
        while s[i] == s[j]:
            i -= 1
            j += 1
            if i < 0: break
        
        if i < 0 or s[i] < s[j]:
            if m == '9':
                m = "0"
            else:
                m = str(int(m) + 1)
                return s[:h1+1] + m + s[h1::-1]
                    
    
    while i >= 0:
        if s[i] > s[j]:
            break
        elif s[i] < s[j]:            
            s = s[:i] + str(int(s[i]) + 1) + "0" * h1              
            break
        elif i > 0: 
            k = i - 1
            l = j + 1
            while s[k] == s[l]:
                k -= 1
                l += 1
                if k < 0: break
            if k >= 0:
                if s[k] < s[l]:
                    if s[i] == '9':
                        s = s[:i] + '0' + s[i+1:]
                    else:
                        x = str(int(s[i]) + 1)
                        s = s[:i] + x + s[i+1:]
                        break
                i = k + 1
                j = l - 1
            else:
                if s[i] == '9':
                    s = s[:i] + '0' + s[i+1:]                    
                else:
                    x = str(int(s[i]) + 1)
                    s = s[:i] + x + s[i+1:]                    
                    break
        
        else:
            if s[i] == '9':
                s = s[:i] + '1' + s[i+1:]
                m += '0'                
                break            
                                
        i -= 1
        j += 1
                
    
    return s[:h1+1] + m + s[h1::-1]
    
t = int(raw_input())

while t:
    t -= 1
    s = raw_input()        
    print palin(s)

Let me know if you can come up with faster and / or more Pythonic solution.

Simple Infix to Postfix in Python

2010-11-24T00:00:00.000+06:00

Today I wrote a Python program to transform an expression to reverse polish notation in order to solve the problem Transform the Expression and it worked fine. Here is my code:

def rpn(s):
    var_list = []
    symb_list = []
    for c in s:
        if c >= 'a' and c <= 'z':
            var_list.append(c)
        elif c in ['^', '*', '/', '+', '-']:
            symb_list.append(c)
        elif c == ')':
            x = var_list.pop()
            y = var_list.pop()
            z = symb_list.pop()
            var_list.append(y + x + z)
        #print var_list, symb_list
        
    return var_list[0]
        
t = int(raw_input())

while t:
    t -= 1
    s = raw_input()        
    print rpn(s)

The algorithm is very simple which you should be able to follow if you study my code. You can uncomment the line #print var_list, symb_list to see a simulation. Note that the Python program I wrote is not the perfect infix to postfix converter but it worked for the problem. The ideal program should work without taking the advantage of brackets. :-)

Prime Number Generator in Python using Sieve Method

2010-11-23T00:30:00.004+06:00

I have implemented a prime number generator in Python using Sieve of Eratosthenes. Here is my code:

import math

high = 10000
root = int(math.sqrt(high) + 1.0)

ara = [x % 2 for x in xrange(0, high)]

ara[1] = 0
ara[2] = 1

x = 3
while x <= root:        
    if ara[x]:
        z = x * x
        while z < high:             
            ara[z] = 0
            z += (x + x)        
    x += 2
        
prime = [x for x in xrange(2, len(ara)) if ara[x] == 1]
print prime

I am looking for ideas to make my code faster. I tested my Python program using the code for solving a problem named Prime Generator in SPOJ. I could solve the problem correctly that took 3 seconds time and 3.8 MB memory.I am posting another code I found here. This one is similar implementation to mine but more Pythonic and much faster:

nroot = int(math.sqrt(n))
sieve = [True] * (n+1)
sieve[0] = False
sieve[1] = False

for i in xrange(2, nroot+1):
    if sieve[i]:
        m = n/i - i
        sieve[i*i: n+1:i] = [False] * (m+1)

sieve = [i for i in xrange(n+1) if sieve[i]]

I updated the above code and now it's 20% faster! :)

nroot = int(math.sqrt(n))
sieve = [True] * (n+1)
sieve[0] = False
sieve[1] = False
sieve[4: n+1:2] = [False] * (n / 2 - 1)
for i in xrange(2, nroot+1):
    if sieve[i]:
        m = (n/i - i) / 2
        sieve[i*i: n+1:i+i] = [False] * (m+1)

sieve = [i for i in xrange(n+1) if sieve[i]]

Teach Yourself Python using Google's Python class

2010-11-05T18:59:00.000+06:00

Recently I found an interesting video and later found this! It's a collection of well thought-out tutorials and videos with exercises (right now not very useful for me though :p). It's done under Google Edu project. I have checked couple of videos and looked at the exercises. I think it's a very good resource for someone who wants to learn Python. So, if you are a python beginner you will find Google's Python Class useful, but if you already know these basic Python stuffs, what you can do is to share so that another Python newbie can get help. :)

Facebook like button Added

2010-10-28T06:18:00.000+06:00

Just added facebook like button in my blog. It was simple. First I Googled using the term 'add facebook like in blogspot'. Then got this link: http://www.wikihow.com/Add-Facebook-Like-to-Blogger and followed the instructions. But I got an error while changing the html source. XML error message: The reference to entity "layout" must end with the ';' delimiter.. I needed to replace the &layout with & amp;layout (you need to delete the space). That's all.

Now you can Like my posts. ;)

Python code to generate dates in a range

2010-09-30T12:53:00.000+06:00

Today I needed to write a Python script where a part of it was to generate all dates in a range. I am sharing a simplified version of my code.


import datetime

def generate_dates(start_date, end_date):
    td = datetime.timedelta(hours=24)
    current_date = start_date
    while current_date <= end_date:
        print current_date
        current_date += td

start_date = datetime.date(2010, 1, 25)
end_date = datetime.date(2010, 3, 5)
generate_dates(start_date, end_date)

As always, please share if you know other methods that serve the purpose.

Insert an item to a tuple in Python

2010-09-27T08:32:00.000+06:00

Tuples, like strings, are immutable: it is not possible to assign to the individual items of a tuple ...

[from python doc].

Say you create a tuple t:


>>> t = 1, 2, 3
>>> t
(1, 2, 3)
>>>

Now you need to assign value 4 to the 4th element of the tuple:


>>> t[3] = 4
Traceback (most recent call last):
  File "", line 1, in 
TypeError: 'tuple' object does not support item assignment

Now you try append like a list:


>>> t.append(4)
Traceback (most recent call last):
  File "", line 1, in 
AttributeError: 'tuple' object has no attribute 'append'
>>>

No luck yet. So better covert the tuple to a list:


>>> l = list(t)
>>> l
[1, 2, 3]
>>> l.append(4)
>>> l
[1, 2, 3, 4]
>>>

And now convert the list to a tuple:


>>> t = tuple(l)
>>> t
(1, 2, 3, 4)
>>>

Do you know of more ways to do this?

Create child process in Python - Subprocess module

2010-09-04T03:09:00.000+06:00

Recently in one of my projects I had to create subprocess (child process) in Python. Actually I had to run multiple instances and didn't want to wait till those processes finish their job. I used the subprocess module to create those process. I am not sharing my actual code rather giving some sample code here that I wrote to understand the use of subprocess module myself which show you how to create child process (for which parent process need not wait for all the child processes to finish execution) using subprocess module.

1. create a file named test.py


#!/usr/bin/env python

import os, subprocess

print "hello"

urlList = ["http://python.org",
"http://muktosoft.com",
"http://bdosn.org",
"http://matholympiad.org.bd",
"http://bbc.co.uk",
"http://cricinfo.com", 
"http://bdpy.org"]


for url in urlList:
    subprocess.Popen(["python", "test2.py", url])

print "good bye\n"

2. now create another file in the same directory named test2.py.


import urllib2
import sys
from urlparse import urlparse

url = sys.argv[1]

print "\nURL: ", url

usock = urllib2.urlopen(url)
data = usock.read()
usock.close()

filename = urlparse(url).netloc + ".html"
fp = open(filename, "w")
fp.write(data)
fp.close()

Now run the program test.py.

test.py program doesn't wait for all the child processes to be finished. It just starts those and terminated. Each child process is terminated when they finish execution. Don't forget to notice the order in which html files are created in your directory. :)

You might need to make the test2.py executable (chmod +x test2.py) but I am not sure whether it was necessary.

python code to retrive links from web page

2010-09-03T00:19:00.000+06:00

There are several ways to extract / retrieve links (URL) from web page using Python. Let me discuss few ways.

1. Using Beautiful Soup.
You can find code to retrieve links from web page using beautiful soup in it's documentation.
The code is simple:


import BeautifulSoup
import urllib2

print "Enter the URL: "
url = raw_input("> ")
usock = urllib2.urlopen(url)
soup = BeautifulSoup.BeautifulSoup(html_source)
links = BeautifulSoup.SoupStrainer('a')
for link in BeautifulSoup.BeautifulSoup(response, parseOnlyThese=links): 
 if link.has_key('href'):
     print link['href']

2. I saw another code in the book dive into python that uses a html parser to extract url.


from sgmllib import SGMLParser

class URLLister(SGMLParser):
 def reset(self):
     SGMLParser.reset(self)
     self.urls = []   

 def start_a(self, attrs):
     href = [v for k, v in attrs if k=='href']
     if href:
         self.urls.extend(href)

if __name__ == "__main__":
 import urllib2
 print "Enter the URL: "
 url = raw_input("> ")

 usock = urllib2.urlopen(url) 
 parser = URLLister() 
 parser.feed(usock.read()) 
 parser.close()
 usock.close()

 for url in parser.urls:
     print url

Now try this URL for the above two programs: http://www.vworker.com
For the first one you will get few links (not all) and for the second one you will get an error message. :(
The reason is, there is an incorrect tag in the html source code (search the source code with the term 'strong')

3. So I had to write this python script using regular expression.


import re
import urllib2

def get_hyperlinks(url, source): 
 if url.endswith("/"):
     url = url[:-1]

 urlPat = re.compile(r'<a [^<>]*?href=("|\')([^<>"\']*?)("|\')')

 result = re.findall(urlPat, source)

 urlList = []

 for item in result:
     link = item[1]     
     if link.startswith("http://") and link.startswith(url):
         if link not in urlList:
             urlList.append(link)
     elif link.startswith("/"):
         link = url + link
         if link not in urlList:
             urlList.append(link)
     else:
         link = url + "/" + link
         if link not in urlList:
             urlList.append(link)
 
 return urlList

print "Enter the URL: "
url = raw_input("> ")
usock = urllib2.urlopen(url)
data = usock.read()
usock.close()
print get_hyperlinks(url, data)

This code also takes care that there is no duplicate links. Now test this with http://www.vworker.com :)

Prevent sql injection in python using cursor.execute correctly

2010-08-26T04:57:00.000+06:00

Today I did some searching about how to prevent sql injection while using Python and MySQLdb. Then I found some links with tips and also did some experiment and found something interesting.

Usually I used to write mysql query in python this way:

query = ".."
cursor.execute(query)

But it can't prevent sql injection. Check this example: (It will be better if you create a table named user_info with the fields: id, name, email and populate the table with some data.)


import MySQLdb

try:
    conn = MySQLdb.connect (host = "localhost",
    user = "uname",
    passwd = "pass",
    db = "mydb")
    cursor = conn.cursor()
    
    email = "' OR '1'='1"
    query = "SELECT * FROM user_info WHERE email = '" + email + "'"
    print query
    cursor.execute(query)
    
    if cursor.rowcount > 0:
        print cursor.fetchall()
    else:
        print "no item found"
        
except MySQLdb.Error, e:
    print "Error %d: %s" % (e.args[0], e.args[1])

It will show all records from user_info table! Because the query SELECT * FROM user_info WHERE email = '' OR '1'='1' gets executed.

Now if you run the following code this won't happen:


import MySQLdb

try:
    conn = MySQLdb.connect (host = "localhost",
    user = "uname",
    passwd = "pass",
    db = "mydb")
    cursor = conn.cursor()

    email = "' OR '1'='1"
    
    cursor.execute("SELECT * FROM user_info WHERE email = %s", email)
    
    if cursor.rowcount > 0:
        print cursor.fetchall()
    else:
        print "no item found"
        
except MySQLdb.Error, e:
    print "Error %d: %s" % (e.args[0], e.args[1])

It displays "no item found". It is secured as it lets the MySQLdb library to handle the necessary checking. Here is the format:

cursor.execute("... %s ...%s", (param1, param2))

Do you have any more suggestion to prevent sql injection while coding in Python thus improving security?

Using Eclipse for Python with Pydev

2010-07-17T19:43:00.000+06:00

So far I have been using gedit in my Ubuntu machine for Python coding. And I am very happy with it. :) But next week I am going to start another Python project (using Django) and I have decided to use Eclipse as the IDE. I have heard good things about it. So just installed pydev in Eclipse. Let me share some helpful links.

1. Pydev in Eclipse market place.

2. Step by step solution for configuring Eclipse for Python.

3. Pydev home page.

Hope you will find this post useful.

Python for Kids

2010-06-27T01:25:00.000+06:00

I am volunteering in a pilot project that targets to teach programming to school kids (class VIII-IX). My responsibility is to design the course. I have been researching about this and decided to use Python (not because it's my favorite language but because I think teaching Python would be a good idea).

If anybody can share any useful resource, links or suggestions it will be highly appreciated.

I shall keep you updated about the outcome of the project.

Design Pattern in Python

2010-04-21T16:10:00.000+06:00

I am exploring some design patters in Python and found some links that might be helpful for you to learn design pattern in Python. Though most of the design patterns are not language specific, but it's nice to see the examples in Python! :)

Check the links:

Patterns in Python

Design Patterns in Python

Design Patterns in Python (presentation by Alex Martelli)

Video on youtube!

Please share if you know about any useful resource / link related to design pattern in Python.

switch-case alternative in Python

2010-04-03T14:46:00.000+06:00

Python has so 'switch - case' support unlike most other common languages. Of course you can use If-else block to serve your purpose. But there is an elegant alternative to 'switch-case' in python, using dictionary. Go though the following code (you should try running it).


# this is the default function
def errorMessage():
    print "Incorrect input. Please enter a number between 0 and 3."

def fnc0():
    print "This is function 0"

def fnc1():
    print "This is function 1"

def fnc2():
    print "This is function 2"

def fnc3():
    print "This is function 3"


fncDict = {'0': fnc0, '1': fnc1, '2': fnc2, '3': fnc3}

num = raw_input("Enter a number between 0 and 3")
# if num is found as a key in the dictionary fncDict, then corresponding function is called
# else the function errorMessage is called
fncDict.get(num, errorMessage)()

Hope you will find it interesting!

Python code to generate random string

2010-04-03T02:22:00.000+06:00

Today I wrote a small piece of Python code to generate random string. The string will have only the characters: 0-9, A-Z, a-z.


    word = ''
    for i in range(wordLen):
        word += random.choice('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789')

Here is the complete source code that generates a large file that has a random word one per line.


import random

def get_random_word(wordLen):
    word = ''
    for i in range(wordLen):
        word += random.choice('ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789')
    return word
    
def main():
    fout = open('word_list.txt', 'w')
    for i in range(100000000):
        wordLen = random.randint(5, 15)
        word = ran_word(wordLen)
        if i % 500000 == 0:
            fout.close()
            fout = open('word_list.txt', 'a')
            print i
        fout.write(word+"\n")
    fout.close()

main()

getch() in Python - get a single character from keyboard

2010-03-30T22:01:00.000+06:00

Today I needed to write a Python code for 'Press any key to continue' feature. First I wrote the code: raw_input("Press Any Key to Continue: ")
But actually it's not any key, you have to press enter followed by any key (or just enter). Then I started searching Google, came across several links, tried some of those and finally got something which is very close to what I need and modified the code to fit into my need. Here is the link that was almost exactly what I needed: http://pyfaq.infogami.com/how-do-i-get-a-single-keypress-at-a-time

Here is my getch function in Python:


import sys    
import termios
import fcntl

def myGetch():
    fd = sys.stdin.fileno()

    oldterm = termios.tcgetattr(fd)
    newattr = termios.tcgetattr(fd)
    newattr[3] = newattr[3] & ~termios.ICANON & ~termios.ECHO
    termios.tcsetattr(fd, termios.TCSANOW, newattr)

    oldflags = fcntl.fcntl(fd, fcntl.F_GETFL)
    fcntl.fcntl(fd, fcntl.F_SETFL, oldflags | os.O_NONBLOCK)

    try:        
        while 1:            
            try:
                c = sys.stdin.read(1)
                break
            except IOError: pass
    finally:
        termios.tcsetattr(fd, termios.TCSAFLUSH, oldterm)
        fcntl.fcntl(fd, fcntl.F_SETFL, oldflags)

And this one is also interesting but I couldn't make it work: getch()-like unbuffered character reading from stdin on both Windows and Unix (Python)

How to define python source code encoding?

2009-12-06T20:59:00.000+07:00

While using non-ASCII characters in your code, you might get this error: "SyntaxError: Non-ASCII character '\xc3' in file myprog.py on line 101, but no encoding declared;".

Solution is to define encoding at the beginning of your source code.

Syntax: # -*- coding: desired_encoding -*-
Example: # -*- coding: iso-8859-1 -*-

For details check http://www.python.org/dev/peps/pep-0263/.

Python Console in gedit

2009-12-01T03:02:00.000+07:00

Being an Ubuntu user gedit is my default editor for programming (sometimes I use netbeans though). Today I found a post where the author wrote about gedit plugins. I just played with it a bit and found that it has a Python Console! This is really interesting.

You can also try installing the plugin:

sudo apt-get install gedit-plugins

Now go to:

edit > preferences > plugins

and select which plugins you want and enable the side pane & bottom pane from the view menue.

Update twitter status using Python script

2009-11-14T23:56:00.000+07:00

Recently I have written a Python script to update status message in twitter using their api. Let me share the code here:


import urllib
import urllib2
import base64
import httplib
import socket

## authenticate module is used for http basic authentication. found in 'urllib2 missing manual'
def authenticate(username, password):
   password_mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()
   top_level_url = 'http://twitter.com/'
   password_mgr.add_password(None, top_level_url, username, password)
   handler = urllib2.HTTPBasicAuthHandler(password_mgr)
   opener = urllib2.build_opener(handler)

   return opener

## updates twitter status given the status, user name (id or email) and password 
def update_status(status, user, password):   
   data = {'status' : status}   
   data = urllib.urlencode(data)
  
   url = 'http://twitter.com/statuses/update.xml'
  
   opener = authenticate(user, password)
   result = ''
   try:
       handle = opener.open(url, data = data)
       result = handle.read()
       handle.close()   
       return
   except Exception, detail:
       print "Err ", detail

#program starts from here   
main()

Hope you will find it useful. Let me know if you have any suggestion to make my code better. It will be appreciated.

Python UNICODE encode / decode error

2009-11-12T01:01:00.000+07:00

Today I was trying to scrape a Spanish site and got into trouble with some Spanish characters. I had to parse some messages from that Spanish website and post into twitter using my Python script. But for some reasons Spanish characters didn't show up in twitter status updates.

I was in some trouble with the following error messages:
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 5778-5781: invalid data
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 5778: ordinal not in range(128)

Then I did Googling for some time and got this very useful link.

I used this code to get the content of the website:


import codecs
import urllib2

url = '' # put the URL here
usock = urllib2.urlopen(url)
Reader = codecs.getreader("latin_1")
fh = Reader(usock)
data = fh.read()
fh.close()
usock.close()
data = data.encode("latin_1")

Though I first used utf-8 encoding rather than latin_1, but when I got this error: "UnicodeDecodeError: 'utf8' codec can't decode bytes in position 5778-5781: invalid data" I found that the website is using latin_1 character set (from html source) not utf-8.

Don't forget to check the codecs module. Btw, I am using Python 2.5.2. :)

Python programmer in Malaysia

2009-10-04T00:18:00.000+07:00

One of my friend is looking for python developers located in Malaysia. If you are interested please contact him: yfaizal at gmail.com

Thanks.

Google friend connect in Love Python

2009-08-22T12:09:00.000+07:00

I have added Google friend connect widget in this blog so the followers can socialize with each other. You can also do it using the 'follower' gadget. You can read about Google friend connect here: http://www.google.com/intl/en/press/annc/20080512_friend_connect.html

randomize elements in a file or list using Python

2009-08-02T00:37:00.000+07:00

Last week I had to populate a mysql table with some random user names. First problem was to get those names - which I managed writing a crawler with Python. Using the program I got a text file with around 500 names listed in alphabetical order. But I didn't want to insert those in the same order as it will look odd (you don't want to get a list of opponent player's name all starting with 'A' in a game, right?). Then I decided to write some python code to generate another file with names in random order. The logic will be simple. First load all the names in a list, and from that list get a random element and put in another list. Delete the element from the initial list. Before I started coding, I googled for a while and found an interesting solution!


import random
...
count = names.__len__() 
# names is the list containing names in alphabetical order
names = random.sample(names, count) 
# now names contain names in random order!
...

Another Python magic!

Should a Python programmer be paid higher than a PHP / Perl / Java programmer ?

2009-07-07T00:30:00.000+07:00

This question might sound stupid, but I have been wondering about this. Say I hire two developers with similar experience and expertise in my company. One is for a Python project and another one is for Java project. Should they get same compensation? A couple of years back, when I heard about Python from one of my senior friend (who is a CTO of a software firm, a very experienced guy), he told me that he likes Python because, development speed is lot faster in this language. He, from his experience believes that if a project takes six months to be completed using Java, four months should be good enough if it's done using Python. He was also planning for a project where he choose Python, and his logic was, he will use four Python developers for the project and if he choose Java instead of Python, it would take 10 developers to complete the project (same time frame). Then I said to him, "so, are you going to pay higher to the Python devs?" ... he just smiled :) ...

So folks, what do you think? Should the Python devs get more? Share your thoughts.

Python Job - Web Development

2009-06-22T01:24:00.000+07:00

Python is slowly getting popular in Bangladesh. Today I have browsed a job site and found one job advertisement for Python programmer. A company named Roots Information Technology is looking for four Python developers. Click here for the job details.

Hopefully we will find more companies looking for Python programmers in Bangladesh in neat future.

Updated python code for get html source

2009-03-10T13:30:00.000+06:00

Yesterday I made little update to my function get_html_source() that gets the content of a page. I did so because I found that my previous function didn't support HTTP POST. Now the code supports both HTTP GET and HTTP POST. It also returns the cookiejar along with the html content of the page.


def get_html_source(url, referer = '', data = 0, cj = 0, retry_counter = 0):
    if retry_counter > 0:
        print 'Trying Again...'
    if retry_counter > 3:
        print 'Could not get source from url:', url
        return '', ''
    try:
        if cj:
            opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
        else:
            opener = urllib2.build_opener()

        opener.addheaders = [('Referer', referer),
                 ('Content-Type', 'application/x-www-form-urlencoded'),
                 ('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14'),
                 ('Accept-Encoding', 'gzip,deflate')]

        if data:
            # HTTP POST
            usock = opener.open(url, data)
        else:
            # HTTP GET
            usock = opener.open(url)
        
        content = decode(usock) # I think I have already written the code of decode function
                                # in another post. If you can't find it, just leave a comment
                                # here and I shall post the code again.
        usock.close()                
        return content, cj
    except urllib2.HTTPError, e:
        print 'The server couldn\'t fulfill the request. for url: ', url
        print 'Error code: ', e.code
        return get_html_source(url, referer, data, cj, retry_counter + 1)            
    except urllib2.URLError, e:
        print 'We failed to reach a server.'
        print 'Reason: ', e.reason
        return get_html_source(url, referer, data, cj, retry_counter + 1)

Please suggest any necessary update / modification of this code.

Resize image before uploading

2009-01-05T10:16:00.000+06:00

Where Internet connection is slow, it takes too much time to upload images. Few days ago I was trying to upload some images in facebook and it seemed to take forever. The I wrote the following python program to reduce the image size (around 4MB to around 100 KB!):


import Image
import os, sys

# adjust width and height to your needs
width = 800
height = 600

for root, dirs, files in os.walk('./'):
    for name in files:       
        filename = os.path.join(root, name)
        if filename.endswith('jpg'):
            print filename
            imin = Image.open(filename)
            imout = imin.resize((width, height), Image.BICUBIC) 
            imout.save(filename)

get remote file size through http

2008-10-28T23:20:00.000+06:00

Couple of days ago I wrote this code.

The requirement is to get the size of a remote file (http). For example: the Python program needs to find the size of the file, http://abc.com/dir/file1.mp3

Now here is a very stupid solution for this:


import urllib2

url = 'http://abc.com/dir/file1.mp3'
usock = urllib2.urlopen(url)
data = usock.read()
size = data.__len__() # size in bytes
size = size / 1024.0 # in KB (Kilo Bytes)
size = size / 1024.0 # size in MB (Mega Bytes)
...

The stupidity happens in this line: data = usock.read() where the whole file is being read to get it's size! This solution came to my mind first. But soon I understood that the file size can be found from the http response header. Here is a much better solution:


import urllib2

url = 'http://abc.com/dir/file1.mp3'
usock = urllib2.urlopen(url)
size =  usock.info().get('Content-Length')
if size is None:
    size = 0
size = float(size) # in bytes
size = size / 1024.0 # in KB (Kilo Bytes)
size = size / 1024.0 # size in MB (Mega Bytes)
...

Remove duplicate items from a list using set

2008-09-27T22:02:00.000+06:00

A common problem beginners face is to remove duplicate items from a list. In Python it can be done easily. First make a set from the list and then make a list for that set.


myList = list(set(myList))

Here is a python example:


>>> myList = [1, 2, 3, 3, 2, 2, 4, 5, 5]
>>> myList
[1, 2, 3, 3, 2, 2, 4, 5, 5]
>>> myList = list(set(myList))
>>> myList
[1, 2, 3, 4, 5]
>>>

Let me know if you use any different technique to remove duplicates from a list.

use proxy and cookie together in Python

2008-09-08T22:24:00.000+06:00

Here is an example code in Python that fetches an URL and writes the content to a text file. The purpose of publishing the code is to demonstrate how to use Proxy and Cookie together in an opener.


import urllib2
import cookielib

# create cookiejar to store cookie
cj = cookielib.CookieJar()

# IP:PORT
proxy_address = '66.98.208.8:3128' # change the IP:PORT, this one is for example

# create the proxy handler
proxy_handler = urllib2.ProxyHandler({'http': proxy_address})

# create opener
opener = urllib2.build_opener(proxy_handler, urllib2.HTTPCookieProcessor(cj))

# install the opener
urllib2.install_opener(opener)

# url to browse / visit
url = "http://www....com/" # change the url

req=urllib2.Request(url)

data=urllib2.urlopen(req).read()

print data

# now write the data to a text file

# create file handler
fh = open('page.txt', 'w')

data = "".join(data)

# write to file
fh.write(data)

# close the file handler
fh.close()

Hope you will find the code useful.

Py2SIS create sis file to make standalone application

2008-09-04T20:44:00.000+06:00

You have coded the program and tested it using PyS60 emulator or in your mobile phone. Now you want to make standalone application out of your Python code to deploy in mobile phones. You have to create SIS (Symbian Installation System) file.

Format of the command is:

py2sis  [sisfile] [--uid=0x12345678] [--appname=myapp] 
[--presdk20] [--leavetemp]

I used the following command to create SIS file for my program:

F:\Nokia\Tools\Python_for_Series_60\py2sis>py2sis.exe 
dse_stock_tracker.py dse.sis --uid 0x12345678 
--appname=DSEStockPriceTracker

If you want to know details about py2sis, get the book (ebook is there) "Programming with Python for Series 60 Platform" and go to chapter 14. The book is freely available.

The A-Z of programming languages: Python

2008-09-01T01:49:00.000+06:00

Computer world published an interesting article - The A-Z of programming languages: Python. Actually it's an interview of Guido van Rossum - the creator of Python language. There he answers various questions regarding Python. I think it's a must read for Python fans / enthusiasts. So can't help but sharing the link with you.

Regular Expression in Python

2008-08-29T01:43:00.000+06:00

'How to learn regular expression in Python?' - a very common question from the beginners or Python newbies. Well... if you can use regular expression in Perl or PHP, then it's not very different in Python.

And if you are completely new to regular expression and want to learn it, then you can follow the following guideline:

[Update on Oct 30, 2010]: I think you should start with this video on regular expression.

At first, you should be to read Regular Expression HowTo by A.M. Kuchling. Read it carefully and read it at least twice :)

So... now you have finished reading Regular Expression How To and still not very confident. Don't worry. It's time to dive into regular expression. Just go through Chapter 7 Regular Expressions of Dive Into Python. Don't forget to code each and every example. There are some nice case studies. Those will help (it was really helpful in my case).

And now it's time for you to check details about the re module from python doc.

By this time you should be a good player of regular expressions and can use it to solve practical problems.

Still want to be a master? Then you should read this: Mastering Regular Expressions (I haven't read it yet though ;)

If you have to work with regular expressions frequently, you can check this one: Regular Expression Pocket Reference: Regular Expressions for Perl, Ruby, PHP, Python, C, Java and .NET (Pocket Reference (O'Reilly))

Hopefully learning regular expressions will empower you more!

Looking for Python Jobs? Check Job Boards

2008-08-27T09:41:00.000+06:00

Sometime I see people wondering about the Job market of Python. Now days kids are more interested about their career :) ... So I have decided to collect Python Job board addresses and post the links here. Note that this page will be updated whenever I find any new link. You can also help me by posting comments.

Python Job Board: This is a community maintained page. It's frequently updated. Also there is a volunteer opportunity page.

Is Python only for web programming?

2008-08-26T23:38:00.000+06:00

As a Python enthusiast/advocate I always have to face some common questions about Python from friends, juniors, colleagues... One of those is, "Is Python used only for web programming?"

So I think I better write the answer here :)

And the answer is NO. Python is used in lots of places:

Web Programming: Python is heavily used in Web Programming. There is a great CMS named Plone, developed in Python. There are also some good web frameworks. Also the Google Application Engine is developed using Python
Desktop Computing: Python can be used to develop software for desktop. There are multiple ways to develop GUI for your python software. You can use Tkinter, wxPython etc.
Distributed Computing: It's another field where Python is used effectively. Once I heard about the RIVER project (I forgot the link) which is developed using Python.
Mobile Phone Programming: Yes, Python can be used in your mobile phones. Nokia developed PyS60 to run in Symbian S60 (series 60) phones. There is also a book available for PyS60 Mobile Python: Rapid prototyping of applications on the mobile platform (Symbian Press)
Game Programming!: There is PyGame. So if you are really want to have some fun while coding, want to develop an interesting game, Python is there for you with pygame. There are books available for pygame: Beginning Game Development with Python and Pygame: From Novice to Professional (Beginning from Novice to Professional), Game Programming With Python (Game Development Series)
Scientific computing: There is Python module available for numerical programming (numpy) and also there is another module that can be used for programming in bioinformatics. There are two complete books available for this: Python For Bioinformatics (Jones and Bartlett Series in Biomedical Informatics) and Bioinformatics Programming in Python: A Practical Course for Beginners

Also check the page from wikipedia List of Python Software

Last but not the least, Python is one of the three official programming languages of Google!

Please add if I missed anything important.

PyS60 - download file using urlretrieve

2008-08-25T21:58:00.000+06:00

The following function written in Python downloads a remote file to mobile phone.


import urllib

def download_file():
    remoteFileName = u"http://example.com/a.txt" # set the url here
    localFileName = u"c:\\items.txt"
    urllib.urlretrieve(remoteFileName, localFileName)

download_file()

The code was used in my 'DSE Price Tracker' application in order to download the list of companies.

Create selection list from a file in PyS60

2008-08-23T00:25:00.000+06:00

Let's come back to the project - DSE price tracker. The next step was to load all the company names from a file (stored in the mobile phone memory) and display in the selection list. The file contains the company names, one per line.

So I wrote the following code:


import appuifw

def get_data_from_file():
    fp = open('c:\\names.txt', 'r')
    li = fp.readlines()
    fp.close()
    compList = []
    for item in li:
        item = item.decode("utf-8")
        compList.append(item)   
    return compList

L = get_data_from_file()
index = appuifw.selection_list(choices=L , search_field=1)

Yes, it works. Note that you must decode each line to unicode (utf-8). (item = item.decode("utf-8"))

But wait, what are those small boxes beside each name? I guess it's the newline character ('\n') after each line. So I need to replace '\n' with a blank ''.
Here is the modified get_data_from_file()


def get_data_from_file():
    fp = open('c:\\names.txt', 'r')
    li = fp.readlines()
    fp.close()
    compList = []
    for item in li:
        item = item.replace('\n', '')
        item = item.replace('\r', '')
        item = item.decode("utf-8")
        compList.append(item)   
    return compList

Now it looks fine!

Next to-do is to grab the file from a remote URL and store it in mobile phone memory, then display the list.

Designing Class in Python - Pythonic way

2008-08-22T01:27:00.000+06:00

I found two very interesting discussions on writing / designing class in Python. If you still in search of the beauty of object oriented programming in Python, check the following links:

Enjoy Python :)

Create a selection list in PyS60

2008-08-21T22:51:00.000+06:00

Creating a selection list for my software - 'DSE Stock Tracker' was very important task. It will show the list of companies of Dhaka Stock Exchange. So for demo I tried to create a list of four companies. Here is the code:


import appuifw

L = ['DHAKABANK', 'AIMS1STMF', 'EXIMBANK', 'TRUSTBANK']
index = appuifw.selection_list(choices=L , search_field=1)

I tried to run the program but got the following error:
SymbianError: [Errno -6] KErrArgument
Oh, I forgot to add the 'u' (u for unicode) before each string. So I changed my code and it works.


import appuifw

L = [u'DHAKABANK', u'AIMS1STMF', u'EXIMBANK', u'TRUSTBANK']
index = appuifw.selection_list(choices=L , search_field=1)

Now, as there are hundreds of companies, it's not wise to write every name in the code. So the next to-do is load the list from a file.

'DSE Stock Tracker' in First BdOSN Code Sprint

2008-08-18T20:48:00.001+06:00

I am back from the first BdOSN code sprint - the first of it's kind in our country - Bangladesh. It was held at SUST (Shah Jalal University of Science & Technology), Sylhet - one of the most beautiful places in Bangladesh. I did my graduation from this university :)

There I lead a project named 'DSE Stock Tracker'. It shows real-time (4 minutes delay) stock prices from Dhaka Stock Exchange to your mobile. The mobile client was developed in Python (PyS60).

The whole system has three parts:

A spider that harvests data from dse website and stores in mysql database
An API for mobile application/client
A mobile client

All the parts are developed using Python.

I shall write my experience, challenges faced and how I solved them in next several posts. Stay tuned!

How to send email from a Python script

2008-08-16T01:21:00.002+06:00

Today I needed to send email from a Python script. As always I searched Google and found the following script that fits to my need.


import smtplib

SERVER = "localhost"

FROM = "sender@example.com"
TO = ["user@example.com"] # must be a list

SUBJECT = "Hello!"

TEXT = "This message was sent with Python's smtplib."

# Prepare actual message

message = """\
From: %s
To: %s
Subject: %s

%s
""" % (FROM, ", ".join(TO), SUBJECT, TEXT)

# Send the mail

server = smtplib.SMTP(SERVER)
server.sendmail(FROM, TO, message)
server.quit()

[Source: http://effbot.org/pyfaq/how-do-i-send-mail-from-a-python-script.htm]

But when I tried to run the program, I got the following error message:

Traceback (most recent call last):

File "email.py", line 1, in 

import smtplib

File "/usr/lib/python2.5/smtplib.py", line 46, in 

import email.Utils

File "/home/dsebd/email.py", line 24, in 

server = smtplib.SMTP(SERVER)

AttributeError: 'module' object has no attribute 'SMTP'

I got rid of the problem when I renamed the file email.py to another file!

Can anyone explain the reason?

Update [21 August 2011]
If you want to use it as function you can use the following code:



import smtplib



def send_email(sender, receiver, subject, body):

    msg = ("From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n%s" 

        %(sender, receiver, subject, body))

    s = smtplib.SMTP('localhost')

    s.sendmail(sender, [receiver], msg)

    s.quit()

You should also check this link for more examples (won't work in older version of python like 2.4.x): http://docs.python.org/library/email-examples.html

Create thumbnail by resizing image in Python

2008-08-09T00:11:00.000+06:00

Sometimes we need to create thumbnails of certain size from image. In this post I show you how to create thumbnail by resizing image in Python.
Here is the Python code for creating thumbnail / resizing an image:


import Image

def generate_and_save_thumbnail(imageFile, h, w, ext):
    image = Image.open(imageFile)
    image = image.resize((w, h), Image.ANTIALIAS)
    outFileLocation = "./images/"
    outFileName = "thumb"
    image.save(outFileLocation + outFileName + ext)

# set the image file name here
myImageFile = "myfile.jpg"
# set the image file extension
extension = ".jpg"
# set height
h = 200
# set width
w = 200

generate_and_save_thumbnail(myImageFile, h, w, extension)

The second argument of image.resize() is the filter. You have several options there:

Image.ANTIALIAS
Image.BICUBIC
Image.BILINEAR
Image.NEAREST

Each uses different algorithm. For downsampling (reducing the size) you can use ANTIALIAS. Other filers may increase the size.

You can also look at the following Python modules:

Strip HTML tags using Python

2008-07-26T18:34:00.002+06:00

We often need to strip HTML tags from string (or HTML source). I usually do it using a simple regular expression in Python. Here is my function to strip HTML tags:

def remove_html_tags(data):
    p = re.compile(r'<.*?>')
    return p.sub('', data)

Here is another function to remove more than one consecutive white spaces:

def remove_extra_spaces(data):
    p = re.compile(r'\s+')
    return p.sub(' ', data)

Note that re module needs to be imported in order to use regular expression.

Here you can find an updated code that gets the text from html: http://love-python.blogspot.com/2011/04/html-to-text-in-python.html

HTTP Response Header Information in Python

2008-07-25T01:55:00.000+06:00

Sometimes we need to get the HTTP Response header from our program. You can check the following python code, that prints the http response header:


>>> import urllib2
>>> usock = urllib2.urlopen('http://bdosn.org')
>>> print usock.info()
Date: Thu, 24 Jul 2008 19:52:53 GMT
Server: Apache/1.3.39 (Unix) PHP/5.2.3 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 mod_gzip/1.3.26.1a FrontPage/5.0.2.2634a mod_ssl/2.8.30 OpenSSL/0.9.7a
X-Powered-By: PHP/5.2.3
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html

>>>

If you want to get the value of a particular item, use the get() method:


>>> print usock.info().get('Date')
Thu, 24 Jul 2008 19:52:53 GMT
>>>

Hope you will find this tips useful!

check status proxy address

2008-07-22T02:45:00.000+06:00

Often we need to use proxy addresses in our web spiders / crawler. But most of the time the proxies don't work. So I made a little python program to test the proxy IPs. Let's look into the code:


import urllib2, socket

socket.setdefaulttimeout(180)

# read the list of proxy IPs in proxyList
proxyList = ['125.76.226.9:80', '213.55.87.162:6588'] # there are two sample proxy ip

for item in proxyList:
    if is_bad_proxy(item):
        print "Bad Proxy", item
    else
        print item, "is working"
    

def is_bad_proxy(pip):    
    try:        
        proxy_handler = urllib2.ProxyHandler({'http': pip})        
        opener = urllib2.build_opener(proxy_handler)
        opener.addheaders = [('User-agent', 'Mozilla/5.0')]
        urllib2.install_opener(opener)        
        req=urllib2.Request('http://www.your-domain.com')  # change the url address here
        sock=urllib2.urlopen(req)
    except urllib2.HTTPError, e:        
        print 'Error code: ', e.code
        return e.code
    except Exception, detail:

        print "ERROR:", detail
        return 1
    return 0

Hope the proxy checker will be useful to you!

Accept-Encoding 'gzip' to make your cralwer faster

2008-07-19T01:00:00.000+06:00

If you are looking for a way to make your web crawler / spider work faster, then you must be looking for reducing the time to fetch data from the Internet. Fortunately many websites provide the content in compressed format if the server gets such type of request. If you see the request header of Firefox (you can use firebug to see the request / response headers), you can find that it always asks for gzip content. If the server provides gzip, then it sends data in gzip form otherwise it sends html. gzip data reduces bandwidth and thus make the program to scrape / harvest data faster.

So you can do the same from you Python code. Just add ('Accept-Encoding', 'gzip,deflate') in the request header. Check the following code chunk:


opener = urllib2.build_opener()
opener.addheaders = [('Referer', referer),
   ('User-Agent', uagent),
   ('Accept-Encoding', 'gzip,deflate')]
usock = opener.open(url)
url = usock.geturl()
data = decode(usock)
usock.close()                
return data

Note the decode() function used in the code. Yes, you have to decode the content (if it's compressed).


def decode (page):
    encoding = page.info().get("Content-Encoding")    
    if encoding in ('gzip', 'x-gzip', 'deflate'):
        content = page.read()
        if encoding == 'deflate':
            data = StringIO.StringIO(zlib.decompress(content))
        else:
            data = gzip.GzipFile('', 'rb', 9, StringIO.StringIO(content))
        page = data.read()

    return page

You can also have a look at this page from the book - Dive Into Python: http://diveintopython.org/http_web_services/gzip_compression.html

If you would like to buy a hard copy of this book, get it from here: Dive Into Python

Using Python to interact with MySQL

2008-07-13T15:49:00.000+06:00

I have to use MySQL database frequently. Interacting with MySQL is easy and simple. There is a nice module named MySQLdb is available for this purpose.

I am not writing details about using it as there is a good tutorial "Writing MySQL Scripts with Python DB-API" available online. I myself read this when I used MySQL with Python for the first time.

MySQL Cookbook also covers database programming in Python and other languages as well.

Regular Expression not working in scraper?

2008-07-05T12:18:00.000+06:00

This is a very common problem for the beginners who try to write web crawler / spider / scraper. The content is fetched but regex is not working right. :(

But the problem is not with the regular expression. You just need to add the following two lines after you fetch content of a web page:


content = content.replace("\n", "")
content = content.replace("\r", "")

Now the regex should work if everything else is ok!

Run Python program in windows command prompt

2008-06-19T22:20:00.000+06:00

To run Python program in windows, I have downloaded and installed python windows installer from http://www.python.org/download/. IDLE is installed by default when I install the python windows installer. So python programs can be written and/or executed using IDLE. Or you can just double click a program to run it. Another way is to run from the windows command prompt. Right click on my computer. Then click Properties > Advanced > Environment Variables. Now edit the 'path' variable. Add the location of python there (in my case it is F:\Python25). If there are more than one variables, each one should be separated by a semicolon (;). For example:
Variable name: path
Variable value: F:\TCWIN45\BIN;F:\Python25

Now you can run / execute a python program from windows command prompt using the following command:
python abc.py

You can also get the python interpreter if you just type python and press enter in the command prompt.

Function call in Python

2008-06-05T09:17:00.000+06:00

Check the following code:

def fibonacci(x):   
    if x < 2: return 1
    return (fibonacci(x-2) + fibonacci(x-1))
   
def factorial(x):
    if x < 2: return 1
    return (x * factorial(x-1))
   
def main():      
    funcs = [fibonacci, factorial]
    n = 10
    for i in range(len(funcs)):       
        print funcs[i](n)       
   
main()

Interesting part of the code is print funcs[i](n). Sometimes it is useful to call the function in this way.

List of files, directories in Python

2008-05-26T01:13:00.000+06:00

Here I present a short python program that prints the list of all files inside the current directory and subdirectories. It prints filename with relative path to the current directory.


import os

for root, dirs, files in os.walk('./'):
   for name in files:       
       filename = os.path.join(root, name)
       print filename

You can also print the list of directories only (just print the dirs). Actually I needed this stuff to solve a problem from Google treasure hunt. Let me know if you want me to share the full source code of the solutions of Google treasure hunt. Also share your code, if you have any different idea!

Check the python documentation to learn more about processing Files and Directories.

Myspace auto login using Python

2008-05-21T01:50:00.000+06:00

Today I am going to share with you an interesting program - a Python script that logs in to myspace given your username (email address) and password. If you study the script, you will get idea about the following things:

HTTP POST in Python
How to use cookiejar
How to add user-agent, referer etc. to the request header using Python
And of course, how to auto login to a site :)

Here is the code:


import urllib2, cookielib, re



url = 'http://secure.myspace.com/index.cfm?fuseaction=login.process'


email = "" #put your email address here
pwd = ""   #put your password here


data = "__VIEWSTATE=%2FwEPDwUKLTY1Mjc2MTMwMWQYAQUeX19Db250cm9sc1JlcXVpcmVQb3N0QmFja0tleV9fFgIFPWN0bDAwJGN0bDAwJE1haW4kY3BNYWluJFNwbGFzaERpc3BsYXkkY3RsMDAkUmVtZW1iZXJfQ2hlY2tib3gFPWN0bDAwJGN0bDAwJE1haW4kY3BNYWluJFNwbGFzaERpc3BsYXkkY3RsMDAkTG9naW5fSW1hZ2VCdXR0b24%3D&NextPage=&ctl00%24ctl00%24Main%24cpMain%24SplashDisplay%24ctl00%24Email_Textbox="+email+"&ctl00%24ctl00%24Main%24cpMain%24SplashDisplay%24ctl00%24Password_Textbox="+pwd+"&ctl00%24ctl00%24Main%24cpMain%24SplashDisplay%24ctl00%24Login_ImageButton.x=36&ctl00%24ctl00%24Main%24cpMain%24SplashDisplay%24ctl00%24Login_ImageButton.y=15&ctl00%24ctl00%24Main%24cpMain%24SplashDisplay%24ctl00%24nexturl=&ctl00%24ctl00%24Main%24cpMain%24SplashDisplay%24ctl00%24apikey=&ctl00%24ctl00%24Main%24cpMain%24SplashDisplay%24ctl00%24ContainerPage="



cj = cookielib.CookieJar()

opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))

opener.addheaders = [('Referer', 'http://www.myspace.com/index.cfm?fuseaction=splash'),

                   ('Content-Type', 'application/x-www-form-urlencoded'),

                   ('User-Agent', 'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14')]

usock = opener.open(url, data)

the_page = usock.read()

usock.close()

And check this link to get idea, how do I know details about request header and post values!

First problem I faced in PyS60

2008-05-19T17:20:00.001+06:00

When I tried to connect to a website using Python from my mobile phone, I got into trouble. Every time I tried a simple program I got the following error message:

IOError:[Errno socket error](0, 'getaddrinfo failed')

Here is the code I tried:


import urllib

usock = urllib.urlopen("http://example.com")
data = usock.read()
usock.close()

I tried Google but couldn't find any good solution. Then I did some trial-n-error myself and got rid of the problem!
Actually when I connect to Internet from my mobile phone I have 3 access points - GP-WAP, GP-MMS and GP-Internet. When I browse from my mobile phone (it's Nokia N70), I use GP-WAP. So when the python program asked for the option I selected that and got the error. When I selected GP-Internet, I got rid of the problem.

Now I can connect the Internet using Python from mobile!!!

py60 - Python for S60

2008-05-19T11:31:00.000+06:00

Yes, I have started using Python in my mobile phone !!!

Python for S60 is Nokia's port of the Python language to the S60 smartphone platform. In addition to the standard features of the Python language, PyS60 provides access to many of the phone's uniquely smartphone-y functions, such as camera, contacts, calendar, audio recording and playing, TCP/IP and Bluetooth communications and simple telephony.

(from http://wiki.opensource.nokia.com/projects/Python_for_S60)

You can use emulator or your mobile phone (S60 platform) to run/test python programs. I use my Nokia N70 to run python and so far it's great fun! You can write the program in your computer and transfer it to your mobile phone. There is an interactive console in the mobile phone so that you can write your program directly in the mobile phone. There is another blue-tooth based console which I haven't tried yet.

You can download the latest version of pyS60 from sourceforge.

And I am using this tutorial in my py60 adventure.

Let me know about your experience if you are using py60/pyS60.

Common problem in CSV file writing

2008-05-11T23:26:00.000+06:00

First let me tell you how I write CSV files :)
I don't use Pythons CSV module for writing. I just name the file xyz.csv (fp = open('xzy.csv', 'w')) and write using something like: fp.write(a+','+b+','+c)

But sometimes it can create a problem if any string contains a comma (,) within itself. For example if the string a is 'hello, how are you?', then it will be printed in two columns... so before writing to file I use:
a = a.replace(',', ' ')

Is there any other way to avoid such situation? If you know, please share.

Convert hex to ASCII string in Python

2008-05-11T01:01:00.000+06:00

Sometimes web pages use hexadecimal values of of ASCII characters. To parse them we need to convert them to ASCII string first. Today I faced a similar problem and came up with the following solution:


import binascii, re

def asciirepl(match):
  # replace the hexadecimal characters with ascii characters
  s = match.group()  
  return binascii.unhexlify(s)  

def reformat_content(data):
  p = re.compile(r'\\x(\w{2})')
  return p.sub(asciirepl, data)

hex_string = '\x70f=l\x26hl=en\x26geocode=\x26q\x3c'
ascii_string = reformat_content(hex_string)

print ascii_string

I took help of binascii module (and of course the re module). The logic was simple, I just replaced \x.. (here .. represents two characters) with their corresponding ASCII characters.

You can check my other post to convert ASCII values to ASCII characters: http://love-python.blogspot.com/2008/04/get-ascii-string-for-decimal-number-in.html

And please share if you have any other idea.

generate different file name each time you run a program

2008-05-09T23:31:00.000+06:00

I have written a script today which has an interesting behavior. Each time the program runs, it generates output in a different file (or in other word, the file name has to be different each time the program runs). So I decided to use current time as the prefix of the filename. I used the localtime() function of time module in Python.


>>> import time
>>> time.localtime()
(2008, 5, 9, 23, 32, 8, 4, 130, 0)
>>>

I made a string from localtime and added it before the filename. You can get the idea from the following code block. It's so simple!


import time

prefix = ''
for t in time.localtime():
  prefix += t.__str__()

fileName = prefix + "_result.csv"

print fileName

Keep enjoying Python!

getpass - Python module for portable password input

2008-05-07T09:44:00.000+06:00

Last night I found an interesting Python module, getpass. It has two functions:

getpass(): Prompt the user for a password without echoing.
getuser(): Return the login name of the user.

The getpass function was really useful for me. But sometimes you would want to avoid the getuser function, and provide user name on your own.

Please check the Python documentation for details: http://docs.python.org/lib/module-getpass.html

Convert text to ASCII and ASCII to text - Python code

2008-04-29T22:25:00.000+06:00

Python has a very simple way to convert text to ASCII and ASCII to text.

To find the ASCII value of a character use the ord() function.
Example:
>>> ord('B')
66

To get a character from it's ASCII value use the chr() function.
Example:
>>> chr(65)
'A'

Here is a program to get the list of all characters along with their values:


# program to print the list of characters and their ASCII values
for value in range(0, 255):
   print "ASCII Value:", value, "\t", "Character:", chr(value), "\n"

Here is another simple ASCII encoding-decoding program:


# Convertion from text to ASCII codes

message = raw_input("Enter message to encode: ")

print "Decoded string (in ASCII):"
for ch in message:
   print ord(ch),
print "\n\n"

# Convertion from ASCII codes to text

message = raw_input("Enter ASCII codes: ")

decodedMessage = ""

for item in message.split():
   decodedMessage += chr(int(item))   

print "Decoded message:", decodedMessage

Hope you got a clear understanding of how to convert between text and ASCII code.

PyHP - Python Hypertext Preprocessor

2008-04-29T01:41:00.000+06:00

Couple of months ago, I was trying to embed Python code into HTML. After given the problem to solve, I was planning for a solution and looking into Python libraries. Suddenly I thought, why not spend some time in google to see if there already exists something similar. Actually I was looking for something very simple and I got one!!! It's called PyHP - Python Hypertext Preprocessor. Thank God, I didn't have to reinvent the wheel, as it was almost the similar thing what I was thinking to build. It's a small, fast, hypertext preprocessor in my favorite language - Python. And of course it is open source. The only thing I needed to change in the code is, I made the code to print output in a html file rather than in console. It was just a simple tweak.

Download and play with PyHP - Python Hypertext Preprocessor. You can use it as a standalone program, or use it with Apache's mod-python.

Enjoy Programming, Enjoy Python!

Copy Compound object lists using deepcopy

2008-04-24T21:10:00.000+06:00

Sometimes you might get into trouble while copying a compound object list to another in Python. If this is so, then you should use the copy module of Python. To know details about shallow and deep copy operations in Python please visit http://docs.python.org/lib/module-copy.html

To copy simple lists without using copy module of python, you can use the method described in another post of my blog http://love-python.blogspot.com/2008/04/how-to-copy-list-in-python.html
One of my friend found that this method wasn't working for him while copying a graph into another (he used adjacency matrix to represent graph), then he tried deepcopy and got rid of the problem.

Locate Firefox Cache - Python Code

2008-04-20T14:34:00.000+06:00

Here I share my python code to locate firefox cache directory.


# program to locate firefox cache directory in Linux
from subprocess import Popen, PIPE

name = raw_input('Enter the top-level direcotry ( example: /home ): ')

li = Popen(['find', name, '-iname', 'cache'], stdout=PIPE).stdout.read().split('\n')

flag = 1

for item in li:
   if item.find('firefox') != -1:
       print 'Cache Directory: ', item
       flag = 0
       break
      
if flag:
   for item in li:
       if item.find('mozilla') != -1:
           print 'Cache Directory: ', item
           break

It will work on Linux (*nix). To use it in windows with minimum modification, you need to install UnxUtil in your machine. Please visit http://speed-dev.blogspot.com/2008/03/unxutil-run-unix-linux-command-in.html to know more about UnxUtil.

Adjacency Matrix (Graph) in Python

2008-04-19T22:35:00.000+06:00

Adjacency matrix is a widely used graph data structure used to store graph. Couple of days ago I had to write a Python code that reads user input from console and stores the graph in an adjacency matrix. I used List for this purpose. Here is my code:-



def graph_input():
    V = int(raw_input("Enter the number of Nodes: "))
    E = int(raw_input("Enter the number of Edges: "))

    print "Enter Edges with weights:\n"

    adj_matrix = []
 # vertex numbering starts from 0
    for i in range(0, V):
        temp = []
        for j in range(0, V):
            temp.append(0)
        adj_matrix.append(temp)
    
    for i in range(0, E):
        s = raw_input()
        u, v, w = s.split()
        u = int(u)
        v = int(v)
        w = int(w)
        adj_matrix[v][u] = adj_matrix[u][v] = w

 # print the adjacency matrix
    for i in range (0, V):
        print adj_matrix[i]

    return adj_matrix

graph = graph_input()

Let me know if you have any other idea to represent a graph using adjacency matrix in Python.

Python Web Frameworks - A Short List

2008-04-18T23:04:00.000+06:00

Python is heavily used for web application development. Web frameworks make your life easier while developing a web application - doesn't matter it's small, medium or huge. Here I give a short list of popular python web frameworks:

Django - Django is a high-level Python Web framework that encourages rapid development and clean, pragmatic design.
Pylons - Pylons is a lightweight web framework emphasizing flexibility and rapid development.
TurboGears - the rapid Web development megaframework
web.py - web.py is a web framework for python that is as simple as it is powerful.

I have little idea about Pylons. But haven't looked at other python web frameworks. So I am not going for any comparison among the python web frameworks :-)

For a more detail list of python web framework please visit here: http://wiki.python.org/moin/WebFrameworks

get content (html source) of an URL by HTTP POST method in Python

2008-04-16T09:51:00.000+06:00

To retrieve or get content (html source) of an URL, sometimes we need to POST some values. Here I show you a sample python code that uses POST.


import urllib, urllib2, time

url = 'http://www.example.com' # write ur URL here

values = {'key1' : 'value1', #write ur specific key/value pair
          'key2' : 'value2',
          'key3' : 'value3',
          }

try:
    data = urllib.urlencode(values)          
    req = urllib2.Request(url, data)
    response = urllib2.urlopen(req)
    the_page = response.read() 
    print the_page 
except Exception, detail: 
    print "Err ", detail

Hope it will be useful for you while writing web crawler/spider, specially where values must be submitted using HTTP POST method to get or extract content from an URL.

Please write your comments.

Python code to scrape email address using regular expression

2008-04-10T07:13:00.000+06:00

Programmers when learn writing web spiders or crawlers, try to write a script to parse/collect email address from website. Here I post a class in Python that can harvest email address. It uses regular expression.


import re

class EmailScraper():
    def __init__(self):        
        self.emails = []

    def reset(self):
        self.emails = []

    def collectAllEmail(self, htmlSource):
        "collects all possible email addresses from a string, but still it can miss some addresses"
        #example: t.s@d.com
        email_pattern = re.compile("[-a-zA-Z0-9._]+@[-a-zA-Z0-9_]+.[a-zA-Z0-9_.]+")
        self.emails = re.findall(email_pattern, htmlSource)
        
    def collectEmail(self, htmlSource):
        "collects all emails that starts with mailto: in the html source string"
        #example: <a href="mailto:t.s@d.com">
        email_pattern = re.compile("<a\s+href=\"mailto:([a-zA-Z0-9._@]*)\">", re.IGNORECASE)
        self.emails = re.findall(email_pattern, htmlSource)

Note that it's not a bullet proof program and don't use it for any bad purpose ;)

Read FireFox Cache with Python

2008-04-08T08:17:00.000+06:00

Do you know the easiest way to view the disk cache data of your firefox browser? Just paste "about:cache?device=disk" (without the quotes) in the address bar of your firefox browser. If you see it for the first time, you will be amazed.

Anyway, I had to write a program that does similar thing (locate and read firefox cache and show cache data) in Python. First I tried to automate the about:cache mechanism, but couldn't make it. Later I went for processing raw cache data.

In the Cache directory, the files _CACHE_001_, _CACHE_002_, and _CACHE_003_ contain a mix of data and metadata. The metadata contains information such as the URL requested, timestamps, size, and other details. You can view the files (I used VI in my Ubuntu to view those).

The file structure of each _CACHE_00n file is as follows:


 ______________________________
|                              |
|       4096 byte free         |
|        block bit-map         |
|------------------------------|
|                              |
|            Blocks            |
|             ...              |
|                              |
|                              |
|______________________________|
The size of each block varies depending on the file. 
_CACHE_001_          256 byte blocks
_CACHE_002_         1024 byte blocks
_CACHE_003_         4096 byte blocks

Data in the cache is always encoded in a group of 1-4 blocks and always starts exactly on a block boundary (an exact multiple of 256, 1024, or 4096 bytes). For cache data that corresponds to metadata, it is encoded in this very specific data format.


      _________________
0-3  |       magic      |    unsigned int   (0x00010008)   (Big Endian)
     |------------------| 
4-7  |    location      |    unsigned int  (Big Endian)
     |------------------|
8-11 |    fetchcount    |    unsigned int  (Big Endian)
     |------------------|
12-15|    fetchtime     |    unsigned int  (Big Endian)
     |------------------|
16-19|    modifytime    |    unsigned int  (Big Endian)
     |------------------|
20-23|    expiretime    |    unsigned int  (Big Endian)
     |------------------|
24-27|    datasize      |    unsigned int  (Big Endian)
     |------------------|
28-31|    requestsize   |    unsigned int  (Big Endian)
     |------------------|
32-35|    infosize      |    unsigned int  (Big Endian)
     |------------------|
36-  |  request string  |    char[].  Length is determined by
     |                  |    the value of the requestsize field
     |       ...        |    above.  Terminated by a \x00 byte.
     |                  |
     |                  |
     |------------------|
n-   |   info string    |    char[].  Length is determined by
     |                  |    the value of the infosize field
     |       ...        |    above. 
     |                  |
     |                  |
     |                  |
     |__________________|

But interestingly I didn't need to use all these knowledge. Just used regular expression to parse the data :-)

How to copy a list in Python?

2008-04-06T00:12:00.001+06:00

Let me tell about a common mistake many python beginners do while trying to copy a list to another list. Suppose to copy a list listA to listB, they use listB = listA, now if you change something in one list, the other list is changed automatically as they refer to same list - actually listB points to listA! So the proper way should be

listB = []
listB.extend(listA)

Here I paste some experiments I made:

>>> listA = [1, 2, 3, 4, 5]
>>> listB = listA
>>> listC = []
>>> listC.extend(listA)
>>> listB
[1, 2, 3, 4, 5]
>>> listC
[1, 2, 3, 4, 5]
>>> listA
[1, 2, 3, 4, 5]
>>> listB[4] = 0
>>> listB
[1, 2, 3, 4, 0]
>>> listA
[1, 2, 3, 4, 0]
>>> listC
[1, 2, 3, 4, 5]
>>> listC[4] = 10
>>> listC
[1, 2, 3, 4, 10]
>>> listA
[1, 2, 3, 4, 0]
>>> listB
[1, 2, 3, 4, 0]
>>>

Hope you get the point!

Get the ASCII string for Decimal number in your data - reformat with Python

2008-04-05T02:45:00.000+06:00

Once while writing a spider for a site, I came across some data that were not ASCII, rather had the ASCII value in the format '&#123' ... so I wrote a function myself to reformat the data (replace those with their corresponding ASCII values). But I had a feel that it was not Pythonic. So I searched Google, and found the following function that does the same task in a Pythonic way. Unfortunately I forgot from where I got it :(

def asciirepl(self, match):
   "Return the ascii string for a decimal number"
   s = match.group()
   value = int(s[2:-1])
   if value > 255:
       return ''
   return chr(value)   

def reformat_content(self, data):
   p = re.compile(r'&#(\d+);')
   return p.sub(self.asciirepl, data)



How to think like a (Python) programmer - Free eBook
2008-04-03T07:40:00.000+06:00
I just found this book couple of days ago. Though it won't be very useful for me to learn Python (as I already familiar with Python), I think it's an excellent book for Python newbies even with no prior programming experience. It can also be used for teaching a one semester programming course - the author wrote the book for this purpose!

The book is divided into 19 chapters that covers programming fundamentals, python fundamentals, python data structures, object oriented programming, and the last chapter introduces to GUI programming with Python (Tkinter).

The book is released under the GNU Free Documentation License!

You can get the it from here. Download is free!

Post your comments on the book.


Leap Year in Python
2008-03-28T23:10:00.000+06:00
Year 2008 is a leap year. The leap year thing is always interesting. Sometimes I also think about the unfortunate people who have birthday on 29th February! :)

Let me write two modules that determines whether a year is leap year or not, (I also write test module for this).



def is_leap_year2(year):
    if year % 400 == 0:
        return True
    elif year % 100 == 0:
        return False
    elif year % 4 == 0:
        return True
    else:
    return False

def is_leap_year(year):
    if year % 100 != 0 and year % 4 == 0:
        return True
    elif year % 100 == 0 and year % 400 == 0:
        return True
    else:
        return False

def test_is_leap_year():
    years = [1900, 2000, 2001, 2002, 2020, 2008, 2010]

    for year in years:
        if is_leap_year2(year):
            print year, "is leap year"
        else:
            print year, "is not a leap year"

#program starts from here
test_is_leap_year()


After writing the code I found a nice discussion in python google group. Click here.

You can also take a look at the wikipedia article on leap year.


Use Beautiful Soup for screen scraping in Python
2008-03-23T22:55:00.000+06:00
Tired of writing web spiders/crawler/scrapers? You can try Beautiful Soup in Python. I also have decided to use it in my next spiders.

From their website:
Beautiful Soup is a Python HTML/XML parser designed for quick turnaround projects like screen-scraping. Three features make it powerful:

   1. Beautiful Soup won't choke if you give it bad markup. It yields a parse tree that makes approximately as much sense as your original document. This is usually good enough to collect the data you need and run away.
   2. Beautiful Soup provides a few simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree: a toolkit for dissecting a document and extracting what you need. You don't have to create a custom parser for each application.
   3. Beautiful Soup automatically converts incoming documents to Unicode and outgoing documents to UTF-8. You don't have to think about encodings, unless the document doesn't specify an encoding and Beautiful Soup can't auto detect one. Then you just have to specify the original encoding. 


The reasons I have decided to learn it:
1. It will be fun to learn a new style for my web spiders
2. It can be used to parse both HTML and XML
3. Sometimes it in buyer's requirement that "Beautiful Soup" should be used in the Python spider

You can download it from here: http://www.crummy.com/software/BeautifulSoup/

Ruby programmers can enjoy Rubyful Soup

Hope the soup will help you to have a better scraping experience (for me too) :-)


Use Proxy in your Spider
2008-03-17T21:48:00.000+06:00
Using proxy, you can minimize the chance of getting blocked for your crawlers/spiders. Now let me tell you how to use proxy ip address in your spider in python. First load the list from a file:

fileproxylist = open('proxylist.txt', 'r')
proxyList = fileproxylist.readlines()
indexproxy = 0
totalproxy = len(proxyList)

Now for each proxy in list, call the following function:

def get_source_html_proxy(url, pip):

    proxy_handler = urllib2.ProxyHandler({'http': pip})
    opener = urllib2.build_opener(proxy_handler)
    opener.addheaders = [('User-agent', 'Mozilla/5.0')]
    urllib2.install_opener(opener)
    req=urllib2.Request(url)
    sock=urllib2.urlopen(req)
    data = sock.read()
    return data

Hope your spidering experience will be better with proxies :-)

Let me know about any alternate idea.


Install IDLE in Linux
2008-03-14T22:19:00.000+06:00
IDLE is a very nice and common tool for python programming. In windows, it comes with default python installation. But in Linux it doesn't come built in. So in Ubuntu I used Synaptic Package Manager to download/install IDLE. Fedora systems comes with Python but dont come with IDLE.. In fact if you search for a package or rpm named idle, you wont find that. IDLE is in python-tools package. So the following command (as root) will install IDLE in Fedora or other RedHat systems.

yum install python-tools

To download Python: http://python.org/download/

Enjoy your ride on Python !


run python script as a background process in linux
2008-03-07T22:19:00.000+06:00
So, you have a server to which you connect remotely, upload a python script and want to run it and logout from the server keeping the program running. If you frequently work with spiders, you surely want to do it. But how to do it? For example if your script's name is script.py, then the command is:

nohup python script.py &

And sometimes you may be interested to see the output is that being generated. Then you should view the nohup.out file! This command can be useful:

tail -f nohup.out

So run the program in background and enjoy your time :-)


RentACoder.com - another place to look for freelance Python projects
2008-03-06T22:46:00.000+06:00
Yes, in rentacoder.com you will find lots of projects including some python projects. If you are serious about freelance work, I think you should visit here for a good start. I have seen lots of people get interested in freelance works, registers themselves and get frustrated as they can't win any bid. So read the blog post and get some ideas. As the writer in the blog explained everything step-by-step.

Happy bidding!


What is your OS?
2008-03-06T22:22:00.000+06:00
Another poll ended. This was the 2nd poll in this site. I asked my visitors about their operating system. In reply I got that 43% uses Windows, 37% Linux, 12% MAC.

Now another poll is running. It's about your browser. Please vote.


extract domain name from url
2008-03-01T20:11:00.000+06:00
Sometimes I need to find domain name from url in my program for various purposes (most of the time in my crawlers). So far I used the following function that takes an url and returns the domain name:


def find_domain(url):
    pos = url[7:].find('/')
    if pos == -1:
        pos = url[7:].find('?')
        if pos == -1:
            return url[7:]
        url = url[7:(7+pos)]
        return url


But today I found a module named urlparse. So my function now looks like this:


def find_domain2(url):
    return urlparse(url)[1]

The new one is much better I think.

Check urlparse for details.


swap values - the Python way
2008-02-27T23:48:00.000+06:00
It's a very well known problem given to the beginners, 'swap values of two variables'. In our first introductory programming course (structured programming in C) we solved it in different ways. Most of us used another temporary variable. Some of us did some math tricks. I remember that one of my friend wrote the following code (in C):
int a, b;
scanf("%d %d", &a, &b);
printf("%d %d\n", b, a);

And it made us laugh :-D

Here is the pythonic way of doing this. Try the following code:
a = 2
b = 3
print a, b
a, b = b, a
print a, b

:-)


Links
2008-02-27T10:20:00.000+06:00
High Speed Software Development
A cool place to learn some tricks with Perl, PHP, Python, C# ... It also has lots of links to take the user at the right place.

Link2Me Link Exchange Directory - The Best Link Exchange for SEO Professionals
Quality directory of webmasters actively seeking link exchange. Improve your search engine rankings and link popularity the easy way. Work clever not hard.

PeatSoft - Basic Programming and Software
GWBasic, QBasic, Visual Basic and ZX Spectrum programming, DESKTOP.INI info, games, tools and David Bowie midis

XML News Editor (SoftXML)
SoftXML as it sounds was born when XML was firstly introduced on web. We have been participating in XML development since its creation. It has been amazing to see how quickly the XML standard has been developed, and how quickly a large number of soft

Directory of Computers/Tech Blogs

Technology Blog Directory

Web Directory

bsd magazine
BSD magazine is a quarterly totally devoted to different distros of BSD. The magazine comes with a DVD with the latest BSD distribution.

Designer Sunglasses at Discount. Christian Dior Sunglasses, Bvlgari Sunglasses, Gucci Sunglasses and more.
Discount Sun Glasses. Designer Glasses Online. Chanel, Gucci, Prada... 2006 2007 New Authentic Shades & Eyewear. Define your style with great buys on the hottest trends in mens & womens fashion.

Free software download, Get free license by post a review at BA-Software
BA-Software - Website provider the best software solution, Multimedia software, Security software, Graphics & Design, Internet software, Developer tools, Utility tools, Ebook, software for Business and Personal.

ASP.net Computer Programming
ASP.net Computer Programming

ResellRightsDownloads- Turnkey Websites, Turn Key Business, Home Based Business.
resell rights, turnkey websites, turn key business for just £49.99, start your own home based business today, and we allow you free use of our downloads area for your own customers when you resell the package, make big money in no time at all.

thedownloadswebstore- Turnkey Websites, Software, Scripts, Graphics, Ebooks.
Turnkey Websites, Software, Scripts, Graphics, Ebooks And More, All At The Lowest Prices, And With Secure Paypal Payments, Instant Downloads And Full Resell Rights.

Programotion. Aprende a programar paso a paso
Aqui dispondras de las nociones basicas para empezar, implementaciones de algoritmos y podras descubrir lenguajes de programacion como Pascal, Eiffel, C, o el ensamblador IA-32.

Freelance.dbPerl for freelance custom web design
Programmers, coders and consultants do work on bid contract basis for offshore projects in USA, Europe, India, Ukraine, Romania, Asia. Freelance custom web designers welcome

Reading Glasses
Discount Designer Reading Glasses &Sunglasses: auth. optical frames, safety lenses: Chanel Dior Dolce e Gabbana Gucci Bvlgari Prada Oakley.. Newest 2008, most popular 2007, popular old fashioned models. SMART EYEGLASSES SHOPPING-add lenses online

Hot World Softwares
There are some downloads,  games, SEO, everything you can find in there

A complete software solutions company
Fortuna InfoSolutions, is a software development and business consulting company. Expertise in CRM, Web based softwares, sites, desktop based systems.

Point to Share
trying to write about sharepoint, .Net framerwork, OOP, and UML

Kinky Lingerie
Lingerie, basques and underwear

christmas gifts
planetvision offers a wide variety of gadgets and gifts for christmas and birthdays all ages catered for

Authentic Prada SunGlasses.
Draw attention to really inexpensive authentic discount designer sunglasses and eyewear online such as Oakley, Chanel, Chopard, Christian Dior, Dolce Gabbana, Gattinoni, Gianni Versace, Giorgio Armani, Gucci, Prada, Revo...

VigRX Plus
UKs Cheapest VigRX Plus, free discreet delivery available on all orders.

Orange County Web Design
Orange County Web Design, Search Engine Optimization (SEO), Affordable Custom Web Design Company servicing businesses and individuals nationwide.

Alargamiento Del Pene
vigrx -  no1 en Alargamiento de Pene en USA e Inglaterra

Special Freelancers for Projects
Online market place for freelancers and projects owners

Miami Limousine Florida - airport port transportation
Luxury sedan and limo service for Miami, Ft Lauderdale, Palm Beach and all Florida Keys
Miami airport and Port of Miami transportation. Covering South Beach and Key West.

Mobile Broadband
Compare mobile broadband available in the uk. Compare these packages to standard broadband packages and see the advantages and disadvantages of mobile broadband.

Add Your Website To The Best Web Directory Network.
Add your website to our network, your website will be accessed by more than 10,000 user everyday.

restresor
Sista minuten resor, eller restresor, blir ofta betydligt billigare an vanliga charterresor. Med restresor reser du billigt.

Special Freelancers for Projects
Online market place for freelancers and projects owners

Home Contractor, House Renovations.
Revovations/contract work for houses/homes in Toronto. Installation and maintenance services include kitchens, basements, bathrooms, finishings, decks, fences, excavation, masonry, flagstone, framing and more. Call today for a free estimate.

Compare Broadband Packages
Compare broadband packages provides detailed analysis of the broadband packages currently available in the UK

Discount Mens Jewelry, Mens Rings, Gifts For Him, Mens Bracelets
Shop for great jewelry and gifts for men, including stainless rings, earrings, bracelets, and pendants, at incredibly low prices with free shipment.

Website Design - Web Design Tutorials
Web design hosting & tutorial website. Thousands of free tutorial videos.

Sexy Underwear
Basques, Underwear and Fancy Dress Costumes. Sexy Clothing, Crotchless Knickers, Naughty Nightwear, Lingerie Kinky Lingerie and Sexy Underwear. Free Delivery.

physiotherapy websites
Physio123 provide physiotherapy website design and seo for physiotherapy clinics and physiotherapy  providers.

Rent A Coder - Hire freelancers - Outsource projects - Webdesign - Work From home - Programming
Rent A Coder, Outsourcing jobs, A place to list your projects, Freelancers market, How to Hire a coder, How to Rent for programming, Freelance programmer, Work from Home, Cheap quality coders, Low project & bid comision, Find a coder for your project

O sitio do Quintas
Videos de travestis e transformistas,nacionais e internacionais. O melhor de cada familia no sitio do quintas.Video,music and fun for all.

Get 50% Discount on Web Hosting with FREE Domain, Website Design, Website Templates, Search Engine Submission, SEO.
Slash.IN Bangalore based Company specialized in Website Designing, Website Hosting, Windows Hosting, Linux Hosting, SEO, Search Engine Submissions, Selling Premium Website Templates, Templates Customizing, Get 40% Discount on all Hosting Plans & more

Special Freelancers for Projects
Online market place for freelancers and projects owners

Festlokal
Hitta din festlokal hos oss. En rolig fest ar beroende av ratt festlokal, stor eller liten. Se till att skaffa en bra festlokal idag.

 Craft Village
Craft Village contains five areas of exploration. They are cooking, sewing, quilting, home decor, and knitting. We offer free recipes and projects for all abilities. There is also a resource page and more. The projects change weekly. Archived.

Russian Wife
Hi Welcome to Russian Rendezvous’s updated website. Run by myself Nick & my Russian wife Lilya. We have been putting men and ladies together now for over 10 years

eaZycomp-Computer Services London
Professional PC and Laptop Repairs, Installation, Support, Maintenance, 24 Hour Service for Homes and Businesses, Wireless networking and more

Chime Cottage Music - specialist online CD shop.
Online CD/MP3 shop for progressive rock, synth rock, electronica, ambient and New Age sounds.

Createwebsite.biz - Build a Website, Create a Website, web hosting, web design, templates, CMS, Ad Free Hosting, Free Domain Name, Make Your Website with our Builder
Make your own website with our website builder - website templates, content management system (CMS), ad-free hosting, ecommerce and real estate solutions, website design and web page design services to create a website at Createwebsite.biz


How to download a file using Python?
2008-02-27T09:13:00.000+06:00
Couple of weeks ago, I had to write a spider that harvest data from a website into a csv file and download the images. First I was thinking how to do the download... then I came up with a simple idea and wrote a function save_image that takes the url of the jpg image and filename, downloads the file and saves it with the name given in filename.

import urllib2

def save_image(url, filename):
    usock = urllib2.urlopen(url)
    data = usock.read()
    usock.close()  
    fp = open(filename, 'wb')
    fp.write(data)
    fp.close()


Actually I just write the file in binary mode. Now post your code that performs this task is a different manner.


GetACoder for freelance Python projects
2008-02-26T00:37:00.000+06:00
GetACoder.com is a good place for freelancers where you can search for Python projects (of course you can search for other projects also, not only Python :)

Go here for Python projects.

There are lot of websites / blogs that can help you to be a successful freelancer. So I don't want to write these things here. Just one line, 'have patience and be honest'.

Have fun with Python!


Use user agent in your spider
2008-02-23T11:57:00.000+06:00
Some websites don't allow your spider to scrape the pages unless you use an user-agent in your code. You can fool the websites using user-agent so that they understand that the request is coming from a browser. Here is a piece of code that use user agent 'Mozilla 5.0' to get the html content of a website:


import urllib2

url = "http://www.example.com" #write your url here
opener = urllib2.build_opener()
opener.addheaders = [('User-agent', 'Mozilla/5.0')]
usock = opener.open(url)
url = usock.geturl()
data = usock.read()
usock.close()               
print data


You can use other user agent as well. For example, the user agent my Firefox browser uses:
"Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.12) Gecko/20061201 Firefox/2.0.0.12 (Ubuntu-feisty)"

What is your user agent?


Python 2.5.2 released!
2008-02-23T03:24:00.000+06:00
Final version of Python 2.5.2 is released!

Download it from here: http://www.python.org/download/releases/2.5.2/

From python.org:
This is the second bugfix release of Python 2.5. Python 2.5 is now in bugfix-only mode; no new features are being added. According to the release notes, over 100 bugs and patches have been addressed since Python 2.5.1, many of them improving the stability of the interpreter, and improving its portability.

Read the release notes for more: http://www.python.org/download/releases/2.5.2/NEWS.txt


Execute Linux commands in Python
2008-02-21T23:35:00.000+06:00
For last few days, I am doing an interesting project, for which I require to execute some Linux commands from Python. So I searched a bit for it and found an way to do it. For example if I want to run the command 'ls -lt', I can use the following code:

import os
os.system('ls -lt')

But problem is I want to store the result in a list. Then I thought of writing the output to a file and read the file into a list. My code was:

import os
os.system('ls -lt > output.txt')



Can you suggest a better way?


which Python version do you use?
2008-02-20T23:58:00.001+06:00
Yes, this was the first poll in this blog. 9 people voted for Python 2.5 and 1 voted for 3.0

I myself still use Python 2.5.1, waiting for the final release of 3.0... :)


How to round a floating point number?
2008-02-19T23:07:00.000+06:00
It's another problem I faced few days ago. I needed to round a floating point number to two digits after decimal place. Then I found a function named 'round()'. Look at the following code block written in Python:


d = 10.0 / 3
print d
d = round(d, 2)
print d



Get Original URL
2008-02-19T00:32:00.000+06:00
Once I got into trouble while crawling some websites. Some of the URL I had wasn't the original URL, rather they were redirecting to some other URL. Then I came up with a function to get the original URL. Here I share it with you:


def get_original_url(url):
"""This function takes an url and returns the original url with cookie (if any)
"""
    cj = cookielib.CookieJar()
    opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
    opener.addheaders = [('User-agent', 'Mozilla/5.0')]
    usock = opener.open(url)
    url = usock.geturl()
    usock.close()  
    return url, cj


Please send me your comments on this piece of code.


Python projects in GetAFreelancer
2008-02-16T23:58:00.000+06:00
If you are interested in some freelance works using Python, there are several websites where you can search for projects. GetAFreelancer.com is one of them. You can find python projects here: http://getafreelancer.com/projects/by-job/Python.html
If you don't have time to work on projects, still, browsing the projects will help you to get some ideas about the real world application of Python.


Command Line Arguments in Python
2008-02-16T02:40:00.000+06:00
Another problem the beginner in Python faces 'to take input from the command line arguments' - maybe you want to give a text file name as a command line argument or you need it for another purpose (in the first time, my necessity was the earlier one).

The solution is to use sys.argv you need to import the sys module first. then sys.argv[0] contains the python filename itself, sys.argv[1] contains the next argument and so on...


import sys

print 'I am ', sys.argv[0]
print sys.argv[1]

Run the program: python my.py file.txt
Now you may assign the sys.argv[1] to a variable and do whatever you want.

To know more about sys module, Check http://docs.python.org/lib/module-sys.html

Hope you find this tip useful :)


Search Python code in Krugle
2008-02-13T02:33:00.000+06:00
I found an interesting search engine that searches for source code. Visit http://www.krugle.org/kse/files and try your search for your desired programming language. 

Interesting thing about krugle is not only that you can search code of your desired language, you can specify where it should search the keyword - in source code or comment or documentation ...

Hope you will find the site useful :-)