Regular Expression not working in scraper?

This is a very common problem for the beginners who try to write web crawler / spider / scraper. The content is fetched but regex is not working right. :(

But the problem is not with the regular expression. You just need to add the following two lines after you fetch content of a web page:

content = content.replace("\n", "")
content = content.replace("\r", "")


Now the regex should work if everything else is ok!

Comments

Shiplu said…
Well, I always use multiline regular expression. they work.
Beside this Its good to use domxml to parse the content. It wont fail.
July 13, 2008 at 12:10 AM
Shiplu said…
Another good technique would be using tidy and xslt for proper scrapping . . .
July 13, 2008 at 12:12 AM
Tamim Shahriar said…
I use urllib2 for fetching content from a website.
July 13, 2008 at 10:09 AM
Post a Comment

Popular posts from this blog

Python all any built-in function

I have been using Python for a long time, still learning lot of simple and new things. Recently I came to know about two built-in functions named all and any . Today while I was solving a hackerrank problem , I used both functions which makes the code a bit nicer, in my opinion. The functions compares two singly linked lists and if they are equal, returns 1, otherwise returns 0. Here is my code - def compare_lists(llist1, llist2): while all([llist1, llist2]): if llist1.data != llist2.data: return 0 llist1 = llist1.next llist2 = llist2.next if any([llist1, llist2]): return 0 return 1 Sometimes, this kind of simple thing brings joy to me. Let me know your thoughts. :)
Read more

Accept-Encoding 'gzip' to make your cralwer faster

Image
If you are looking for a way to make your web crawler / spider work faster, then you must be looking for reducing the time to fetch data from the Internet. Fortunately many websites provide the content in compressed format if the server gets such type of request. If you see the request header of Firefox (you can use firebug to see the request / response headers), you can find that it always asks for gzip content. If the server provides gzip, then it sends data in gzip form otherwise it sends html. gzip data reduces bandwidth and thus make the program to scrape / harvest data faster. So you can do the same from you Python code. Just add ('Accept-Encoding', 'gzip,deflate') in the request header. Check the following code chunk: opener = urllib2.build_opener() opener.addheaders = [('Referer', referer), ('User-Agent', uagent), ('Accept-Encoding', 'gzip,deflate')] usock = opener.open(url) url = usock.geturl() data = decode(usock) usock.c...
Read more

lambda magic to find prime numbers

Find prime number up to 100 with just 3 lines of code. The 4th line is for printing ;) nums = range(2, 100) for i in range(2, 10):     nums = filter(lambda x: x == i or x % i, nums) print nums Isn't it amazing? Check my new post about Prime number: http://love-python.blogspot.com/2010/11/prime-number-generator-in-python-using.html
Read more