scrape macy's deals using beautiful soup

Let me show you a tiny real example on how to use the bs4 (beautiful soup version 4) module of Python. Say we want to collect information about the hot deals from macy's. The URL is here. Well, you can see all the info in one page and copy-paste, but that's not our purpose. First you have to get the content of the page using the cute requests module. 
import requests

url = 'http://bit.ly/19zWmQT'
r = requests.get(url)
html_content = r.text
Now start cooking the soup: 
from bs4 import BeautifulSoup

soup = BeautifulSoup(html_content)
Now look at the html code (page source code) of the url. You will see that the offers are in a list (li) that has 'offer' as a css class name (and some other class names). So you can write the code in the following way: 
offer_list = soup('li', 'offer')
Or you can write: 
offer_list = soup.find_all('li', 'offer')
Another way to write this is: 
offer_list = soup.select('li.offer')
Now run this loop: 
for offer in offer_list:
    title = offer.find('h3').text
    url = offer.find('a')['href']
    description = offer.find('span').text
    promo_code = offer.find('span', class_='promo-code').text
    promo_date = offer.find('span', class_='end-date').text
    print title, url, description, promo_date, promo_code
You are done! :)

Comments

Unknown said…
I tried using this code with no luck. Specifically, I tried writing 'offer_list' in all three ways, and for each one the 'for loop' simply produces nothing. And I have Beautiful Soup loaded.

December 3, 2013 at 12:56 AM
Unknown said…
I tried using this code with no luck. Specifically, when I ran the 'for loop' I got no output. And I do have Beautiful Soup loaded and running.
December 3, 2013 at 12:57 AM
Post a Comment

Popular posts from this blog

Strip HTML tags using Python

We often need to strip HTML tags from string (or HTML source). I usually do it using a simple regular expression in Python. Here is my function to strip HTML tags: def remove_html_tags(data): p = re.compile(r'<.*?>') return p.sub('', data) Here is another function to remove more than one consecutive white spaces: def remove_extra_spaces(data): p = re.compile(r'\s+') return p.sub(' ', data) Note that re module needs to be imported in order to use regular expression. Here you can find an updated code that gets the text from html: http://love-python.blogspot.com/2011/04/html-to-text-in-python.html
Read more

lambda magic to find prime numbers

Find prime number up to 100 with just 3 lines of code. The 4th line is for printing ;) nums = range(2, 100) for i in range(2, 10):     nums = filter(lambda x: x == i or x % i, nums) print nums Isn't it amazing? Check my new post about Prime number: http://love-python.blogspot.com/2010/11/prime-number-generator-in-python-using.html
Read more

Convert text to ASCII and ASCII to text - Python code

Python has a very simple way to convert text to ASCII and ASCII to text. To find the ASCII value of a character use the ord() function. Example: >>> ord('B') 66 To get a character from it's ASCII value use the chr() function. Example: >>> chr(65) 'A' Here is a program to get the list of all characters along with their values: # program to print the list of characters and their ASCII values for value in range(0, 255): print "ASCII Value:", value, "\t", "Character:", chr(value), "\n" Here is another simple ASCII encoding-decoding program: # Convertion from text to ASCII codes message = raw_input("Enter message to encode: ") print "Decoded string (in ASCII):" for ch in message: print ord(ch), print "\n\n" # Convertion from ASCII codes to text message = raw_input("Enter ASCII codes: ") decodedMessage = "" for item in message.split(): decodedMessage += chr
Read more