Python code to search tweets in twitter

Recently I wrote a code to search keywords in twitter and extract the tweets as a part of a larger program. It uses twitter search api. I guess some of you might find this code snippet useful. :)

 import requests  
 import sys  
 import time  
 from datetime import datetime  
   
   
   
 def store_tweets(keyword, results, max_id):  
   tweet_list = []  
     
   for item in results:  
     tweet_id = item['id_str']  
     tweet_text = item['text'].encode('utf-8', 'ignore')  
     if len(tweet_text) > 255:  
       tweet_text = tweet_text[0:255]  
     from_user = item['from_user'].encode('utf-8', 'ignore')  
     from_user_id = item['from_user_id']  
     from_user_name = item['from_user_name'].encode('utf-8', 'ignore')  
     profile_image = item['profile_image_url'].encode('utf-8', 'ignore')  
     created_at = datetime.strptime(item['created_at'][:-6], "%a, %d %b %Y %H:%M:%S")  
       
     tweet_list.append((keyword, tweet_id, tweet_text, from_user, from_user_id, from_user_name, profile_image, created_at))  
       
   # Now store the tweet list in a database or file  
   return  
   
   
   
 def search_tweets(keyword, max_id = '', result_type = 'mixed'):  
   url = 'http://search.twitter.com/search.json'  
     
   params = {'q': keyword, 'rpp': '100'}  
     
   if len(max_id) > 0:  
     params['since_id'] = max_id  
       
   params['result_type'] = result_type  
   r = requests.get(url, params=params)  
   time.sleep(5)  
     
   if r.status_code != 200:  
     print r.status_code  
     print r.text  
     time.sleep(30)  
     return r.status_code  
   
   json_content = r.json  
     
   max_id = json_content['max_id_str']  
     
   while True:  
     results = json_content['results']  
     if len(results) == 0:  
       print "no result"  
       break  
         
     store_tweets(keyword, results, max_id)  
       
     if 'next_page' in json_content.keys():  
       next_page = json_content['next_page']  
     else:  
       break  
       
     r = requests.get(url + next_page)  
     time.sleep(5)  
     print r.status_code  
     if r.status_code != 200:  
       print r.status_code  
       print r.text  
       time.sleep(30)  
       return r.status_code  
   
     json_content = r.json  
       
     
     
 if __name__ == "__main__" :  
   keyword = 'python'  
   search_tweets(keyword)  
     
     
   

Generator and yield explained simply

I know though some of you are already using Python for some time, you still get confused when you hear the term generator and yield. Let me show you a simple program that explains it easily:

my_li = ['a', 'b', 'c', 'd']

def generator_example(my_li):
    for item in my_li:
        yield item, 1

x = generator_example(my_li)
print type(x)
print x
for item in x:
    print item

Now, if you run the program, you will see the following output:

('a', 1)
('b', 1)
('c', 1)
('d', 1)

So, you see, the function generator_example returned a generator! We are used to see return in functions that returns a value (variable, list, dictionary etc.). But what if you want to return a series (sequence) of items instead of a single value? There you need to use yield. So in the loop (for item in my_li:) for every item in the list you return (actually yield) a tuple (the item and 1) and calling yield you don't get out of the function generator_example, rather you continue yielding for all the items in the list and then get out of the function. Now the return value of the function is stored in x. If you print the type of x, you will see it is and if you print the value of x, you see that it's a generator object. What do you do with this? You can iterate over x using loop and get the actual values. But you can iterate only once. Now you need another program to get it more clear? This example is even simpler:

my_li = ['a', 'b', 'c', 'd']

def generator_example(my_li):
    for item in my_li:
        yield item.upper()

x = generator_example(my_li)
print type(x)
print x
for item in x:
    print item

This page also has some good explanations: http://stackoverflow.com/questions/231767/the-python-yield-keyword-explained

dict object is not callable on json, requests module

So, you are using the amazing requests module of Python! So checked their sample code in the website: http://docs.python-requests.org/en/latest/ and the URL you are calling returns json content. So you followed their example and came up with a code like this:

import requests
url = '' # put your url here
r = requests.get(url)
json_content = r.json()
print json_content

When you run the program, you get the following error:

json_content = r.json()
TypeError: 'dict' object is not callable

Now you are wondering what is this! An example from the home page doesn't work. So you searched Google and came to my website. Now you solve the problem using json_content = r.json instead of json_content = r.json()
So this code with work (Python 2.7):

import requests
url = '' # put your url here
r = requests.get(url)
json_content = r.json
print json_content

If you are wondering why you got the error in the first code, let me give you a hint. Add the following lines to your Python code and see the output:
print type(r)
print type(r.json)

generate all subsets using recursion | how to

Yesterday I wrote another program to generate all subsets of a set (array / list) during my preparation for today's class on backtracking at National Programming Camp. I wrote the code in C but just converted it into Python for the readers of the blog, so that they can understand and appreciate the beauty of recursion / backtracking.

 taken = []  
     
 def process(X):  
     print X  
   
 def generate_all_subsets(li, X, i, k, m):      
     process(X[0:i])  
   
     for j in range(k, m):  
         if taken[j] == False:              
             if i > 0:  
                 if li[j] <= X[i-1]:  
                     continue  
   
             taken[j] = True  
             X.append(li[j])  
             generate_all_subsets(li, X, i+1, j+1, m)  
             taken[j] = False  
             X.pop(i)  
           
 if __name__ == "__main__":  
     li = [1, 2, 3, 4]  
     X = []  
     taken.extend([False, False, False, False])  
     generate_all_subsets(li, X, 0, 0, li.__len__())  
       

generate permutation using recursion | how to

Here is a Python program to generate all permutations using recursion. I just wrote the code while preparing for a class in national programming camp. If you study it carefully, you will have some idea about backtracking.

 
 taken = []  
 
 def process(X):  
      print X  
 
 def recurse(li, X, i, n):  
      if i == n:  
           process(X[0:i])  
      else:  
           for j in range(0, n):  
                if taken[j] == False:  
                     taken[j] = True  
                     X.append(li[j])  
                     recurse(li, X, i+1, n)  
                     taken[j] = False  
                     X.pop(i)  
 
 if __name__ == "__main__":  
      li = [1, 2, 3, 4]  
      X = []  
      taken.extend([False, False, False, False])  
      print taken  
      recurse(li, X, 0, li.__len__())  

MongoDB for Python Developers - online course

10genEducation is offering couple of free online course on mongodb for developers. One is going to start on January (2013) and another will start on February (2013). They are going to use Python for the January course. So if you are a Python programmer and interested in NoSQL (mongodob), just sign up here.

Concatenate Two Dictionaries | Python How To?

If you want to concatenate two dictionaries in Python (add a dictionary to another dictionary), there is a simple way to do it. Use the update() method which is much like the extend() function used to concatenate lists.

>>> dict_1 = {1: 'a', 2: 'b', 3: 'c'}
>>> dict_2 = {4: 'd', 5: 'e', 6: 'f'}
>>> dict_1
{1: 'a', 2: 'b', 3: 'c'}
>>> dict_2
{4: 'd', 5: 'e', 6: 'f'}
>>> dict_1.update(dict_2)
>>> dict_2
{4: 'd', 5: 'e', 6: 'f'}
>>> dict_1
{1: 'a', 2: 'b', 3: 'c', 4: 'd', 5: 'e', 6: 'f'}
>>>

What is your other tricks for concatenating Python dictionaries?