Scraping with Scrapy and Selenium

0
0

I have a scrapy spider which crawls a site that reloads content via javascript on the page. In order to move to the next page to scrape, I have been using Selenium to click on the month link at the top of the site.

The problem is that, even though my code moves through each link as expected, the spider just scrapes the first month (Sept) data for the number of months and returns this duplicate data.

How can I get around this?

from selenium import webdriver
class GigsInScotlandMain(InitSpider):
name
= 'gigsinscotlandmain'
allowed_domains
= ["gigsinscotland.com"]
start_urls
= ["http://www.gigsinscotland.com"]
def __init__(self):
InitSpider.__init__(self)
self
.br = webdriver.Firefox()
def parse(self, response):
hxs
= HtmlXPathSelector(response)
self
.br.get(response.url)
time
.sleep(2.5)
# Get the string for each month on the page.
months
= hxs.select("//ul[@id='gigsMonths']/li/a/text()").extract()
for month in months:
link
= self.br.find_element_by_link_text(month)
link
.click()
time
.sleep(5)
# Get all the divs containing info to be scraped.
listitems
= hxs.select("//div[@class='listItem']")
for listitem in listitems:
item
= GigsInScotlandMainItem()
item
['artist'] = listitem.select("div[contains(@class, 'artistBlock')]/div[@class='artistdiv']/span[@class='artistname']/a/text()").extract()
#
# Get other data ...
#
yield item
  • You must to post comments
0
0

The problem is that you are reusing HtmlXPathSelector that was defined for the initial response. Redefine it from selenium browser source_code:

...
for month in months:
link
= self.br.find_element_by_link_text(month)
link
.click()
time
.sleep(5)
hxs
= HtmlXPathSelector(self.br.page_source)
# Get all the divs containing info to be scraped.
listitems
= hxs.select("//div[@class='listItem']")
...
  • You must to post comments
Showing 1 result
Your Answer
Post as a guest by filling out the fields below or if you already have an account.
Name*
E-mail*
Website