If you are looking for a way to make your web crawler / spider work faster, then you must be looking for reducing the time to fetch data from the Internet. Fortunately many websites provide the content in compressed format if the server gets such type of request. If you see the request header of Firefox (you can use firebug to see the request / response headers), you can find that it always asks for gzip content. If the server provides gzip, then it sends data in gzip form otherwise it sends html. gzip data reduces bandwidth and thus make the program to scrape / harvest data faster. So you can do the same from you Python code. Just add ('Accept-Encoding', 'gzip,deflate') in the request header. Check the following code chunk: opener = urllib2.build_opener() opener.addheaders = [('Referer', referer), ('User-Agent', uagent), ('Accept-Encoding', 'gzip,deflate')] usock = opener.open(url) url = usock.geturl() data = decode(usock) usock.c...
Comments
--Hans
it really helpful.
thanks.
or any link for explanation will be helpful.
thanks.
Example:
>>> foo = [2, 5, 1, 8]
Regular slice (returns all)
>>> foo[:]
[2, 5, 1, 8]
Slice with stride (returns every nth)
>>> foo[::2]
[2, 1]
Stride works inversely as well
>>> foo[::-2]
[8, 5]
Hence it can be used to reverse a string (stride backwards using step length of one)
>>> 'foobar'[::-1]
'raboof'
There is another way to reverse as well. This can be done using "reversed" generator:
>>> ''.join([char for char in reversed('foobar')])
'raboof'
As the list comprehension returns a list I had to join it separately. Normally the [::-1] trick should work just fine. "reversed" may be useful in some special cases, however.
Check out http://www.python.org/doc/2.3.5/whatsnew/section-slices.html for more comprehensive documentation.
s = s[::2]
print s
#for this code it returns ac
#how
Does anyone know what i should use to extract string/substring from a string?
print $string = reverse("ABC")
>>>print s[:]
spam
>>>print s[::-1]
maps
>>>print s[0:len(s)]
spam
>>>print s[0:len(s):-1]
(prints nothing)
Any reason, why ??
s='abc'
print s[::-2]
returns 'ca' only...
You could do this:
oct(12)
that would return
'014' as a string
To enter an octal number literal, precede with a 0:
0122 will return 82 in base 10
to go from an octal string to an int:
int('014',8)
>>> sentence = "one two three four"
>>> ' '.join( [ word[::-1] for word in sentence.split() ] )[::-1]
'four three two one'
>>> sentence = "one two three four"
>>> ' '.join( [ word for word in sentence.split() ] [::-1])
'four three two one'
[::-1]
[:] <-- this gets all of the string
while [::(value)] skips or reverse the string. if it is negative, the skipping starts from the last character of the string depends on the number.
for example, you can put -2 instead of -1.
s = 'abcdef'
s = s[::-2]
print s
if counted base on the slice number, it would creat a group of three which are fe, dc and ba. the first character of each group (f,d,b) will appended to formed a value and stored it to the variable 's'.
revword = ''
for letter in reversed(word): revword += letter
print revword
Full code solutions with test data and output, described in clear English and tested on the same page
print (" ".join(str.split()[::-1])) would work for reversing string word by word :)
Thought to share it . :)