Python UNICODE encode / decode error

Today I was trying to scrape a Spanish site and got into trouble with some Spanish characters. I had to parse some messages from that Spanish website and post into twitter using my Python script. But for some reasons Spanish characters didn't show up in twitter status updates.

I was in some trouble with the following error messages:
UnicodeDecodeError: 'utf8' codec can't decode bytes in position 5778-5781: invalid data
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 5778: ordinal not in range(128)

Then I did Googling for some time and got this very useful link.

I used this code to get the content of the website:

import codecs
import urllib2

url = '' # put the URL here
usock = urllib2.urlopen(url)
Reader = codecs.getreader("latin_1")
fh = Reader(usock)
data = fh.read()
fh.close()
usock.close()
data = data.encode("latin_1")


Though I first used utf-8 encoding rather than latin_1, but when I got this error: "UnicodeDecodeError: 'utf8' codec can't decode bytes in position 5778-5781: invalid data" I found that the website is using latin_1 character set (from html source) not utf-8.

Don't forget to check the codecs module. Btw, I am using Python 2.5.2. :)

Comments

Andrei Savu said…
... and don't forget to check chardet

http://chardet.feedparser.org/
November 12, 2009 at 2:03 AM
Tamim Shahriar said…
Thanks Andrei for the cool link.
November 12, 2009 at 2:06 AM
wingi said…
You should encode and decode the correct charsets - dont try the several charsets until it works (for you) ! Your spanish site with serve the correct encoding in Header or html. And the twitter api will accept utf-8.
April 5, 2010 at 2:07 AM
Unknown said…
i am new in python .i can't print Bengali in console.can you suggest any way or sample code?
August 1, 2014 at 8:24 AM
Post a Comment

Popular posts from this blog

Python all any built-in function

I have been using Python for a long time, still learning lot of simple and new things. Recently I came to know about two built-in functions named all and any . Today while I was solving a hackerrank problem , I used both functions which makes the code a bit nicer, in my opinion. The functions compares two singly linked lists and if they are equal, returns 1, otherwise returns 0. Here is my code - def compare_lists(llist1, llist2): while all([llist1, llist2]): if llist1.data != llist2.data: return 0 llist1 = llist1.next llist2 = llist2.next if any([llist1, llist2]): return 0 return 1 Sometimes, this kind of simple thing brings joy to me. Let me know your thoughts. :)
Read more

Accept-Encoding 'gzip' to make your cralwer faster

Image
If you are looking for a way to make your web crawler / spider work faster, then you must be looking for reducing the time to fetch data from the Internet. Fortunately many websites provide the content in compressed format if the server gets such type of request. If you see the request header of Firefox (you can use firebug to see the request / response headers), you can find that it always asks for gzip content. If the server provides gzip, then it sends data in gzip form otherwise it sends html. gzip data reduces bandwidth and thus make the program to scrape / harvest data faster. So you can do the same from you Python code. Just add ('Accept-Encoding', 'gzip,deflate') in the request header. Check the following code chunk: opener = urllib2.build_opener() opener.addheaders = [('Referer', referer), ('User-Agent', uagent), ('Accept-Encoding', 'gzip,deflate')] usock = opener.open(url) url = usock.geturl() data = decode(usock) usock.c...
Read more

lambda magic to find prime numbers

Find prime number up to 100 with just 3 lines of code. The 4th line is for printing ;) nums = range(2, 100) for i in range(2, 10):     nums = filter(lambda x: x == i or x % i, nums) print nums Isn't it amazing? Check my new post about Prime number: http://love-python.blogspot.com/2010/11/prime-number-generator-in-python-using.html
Read more