Sunday, March 23, 2008

Use Beautiful Soup for screen scraping in Python

Tired of writing web spiders/crawler/scrapers? You can try Beautiful Soup in Python. I also have decided to use it in my next spiders.

From their website:
Beautiful Soup is a Python HTML/XML parser designed for quick turnaround projects like screen-scraping. Three features make it powerful:

1. Beautiful Soup won't choke if you give it bad markup. It yields a parse tree that makes approximately as much sense as your original document. This is usually good enough to collect the data you need and run away.
2. Beautiful Soup provides a few simple methods and Pythonic idioms for navigating, searching, and modifying a parse tree: a toolkit for dissecting a document and extracting what you need. You don't have to create a custom parser for each application.
3. Beautiful Soup automatically converts incoming documents to Unicode and outgoing documents to UTF-8. You don't have to think about encodings, unless the document doesn't specify an encoding and Beautiful Soup can't auto detect one. Then you just have to specify the original encoding.

The reasons I have decided to learn it:
1. It will be fun to learn a new style for my web spiders
2. It can be used to parse both HTML and XML
3. Sometimes it in buyer's requirement that "Beautiful Soup" should be used in the Python spider

You can download it from here:

Ruby programmers can enjoy Rubyful Soup

Hope the soup will help you to have a better scraping experience (for me too) :-)


socalspiders said...

what are you using , other than python, for spiders? (Php, perl?)

subeen said...

If it's my personal project or I have the freedom to choose language, I always use Python. But in my previous Job I had to use Perl to spider video websites (more than 50 major websites) in order to build a video search engine. Perl is also very good for writing spiders. I also used PHP for some small crawlers specially when it needs a web interface.

omii said...

I recently got into Python and just used Beautiful Soup to do some screen scraping (I wanted to get the IMDB top 250 movies list in a text file). It was incredibly easy! Here's a quick post on what I did with sample code: Screen Scraping with Python & Beautiful Soup