Skip to content

fix description parsing and use user-agent that gives more results #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
lindi2 wants to merge 6 commits into
base: master
Choose a base branch
from
Open

fix description parsing and use user-agent that gives more results #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
80e42ba
Fix Google search description parsing
lindi2 Aug 1, 2012
104a0e1
Use MSIE user-agent to get more results
lindi2 Aug 1, 2012
4ef6ff2
support ftp and https urls
lindi2 Mar 8, 2013
6011093
support ftp and https urls
lindi2 Mar 8, 2013
0df4267
Handle google /interstitial urls (used for potentially malicious sites?)
lindi2 Mar 13, 2013
4eb2e04
Merge branch 'master' of horizon:/opt/xgoogle
lindi2 Mar 13, 2013
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion xgoogle/browser.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
# awk -F\" '{B[$6]++} END { for (b in B) { print B[b] ": " b } }' |
# sort -rn |
# head -20
'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)',
'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6',
'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.5; en-US; rv:1.9.0.6) Gecko/2009011912 Firefox/3.0.6',
'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6 (.NET CLR 3.5.30729)',
Expand All @@ -32,7 +33,6 @@
'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.5) Gecko/2008121621 Ubuntu/8.04 (hardy) Firefox/3.0.5',
'Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6; en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1 Safari/525.27.1',
'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)',
'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 2.0.50727)',
'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'
)

Expand Down
13 changes: 10 additions & 3 deletions xgoogle/search.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,7 +250,10 @@ def _extract_title_url(self, result):
title = ''.join(title_a.findAll(text=True))
title = self._html_unescape(title)
url = title_a['href']
match = re.match(r'/url\?q=(http[^&]+)&', url)
match = re.match(r'/url\?q=((http|ftp|https)[^&]+)&', url)
if match:
url = urllib.unquote(match.group(1))
match = re.match(r'/interstitial\?url=((http|ftp|https)[^&]+)&', url)
if match:
url = urllib.unquote(match.group(1))
return title, url
Expand All @@ -260,6 +263,10 @@ def _extract_description(self, result):
if not desc_div:
self._maybe_raise(ParseError, "Description tag in Google search result was not found", result)
return None
desc_span = desc_div.find('span', {'class': 'st'})
if not desc_span:
self._maybe_raise(ParseError, "Description tag in Google search result was not found", result)
return None

desc_strs = []
def looper(tag):
Expand All @@ -275,8 +282,8 @@ def looper(tag):
except AttributeError:
desc_strs.append(t)

looper(desc_div)
looper(desc_div.find('wbr')) # BeautifulSoup does not self-close <wbr>
looper(desc_span)
looper(desc_span.find('wbr')) # BeautifulSoup does not self-close <wbr>

desc = ''.join(s for s in desc_strs if s)
return self._html_unescape(desc)
Expand Down