Selenium: retrieve data that loads while scrolling down

0
0

I’m trying to retrieve elements in a page that has an ajax-load scroll-down functionality alla Twitter. For some reason this isn’t working properly. I added some print statements to debug it and I always get the same amount of items and then the function returns. What am I doing wrong here?

wd = webdriver.Firefox()
wd
.implicitly_wait(3)
def get_items(items):
print len(items)
wd
.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# len(items) and len(wd.find_elements-by...()) both always seem to return the same number
# if I were to start the loop with while True: it would work, but of course... never end
while len(wd.find_elements_by_class_name('stream-item')) > len(items):
items
= wd.find_elements_by_class_name('stream-item')
print items
wd
.execute_script("window.scrollTo(0, document.body.scrollHeight);")
return items
def test():
get_page
('http://twitter.com/')
get_items
(wd.find_elements_by_class_name('stream-item'))
  • You must to post comments
0
0

Try putting a sleep in between

wd = webdriver.Firefox()
wd
.implicitly_wait(3)
def get_items(items):
print len(items)
wd
.execute_script("window.scrollTo(0, document.body.scrollHeight);")
# len(items) and len(wd.find_elements-by...()) both always seem to return the same number
# if I were to start the loop with while True: it would work, but of course... never end
sleep
(5) #seconds
while len(wd.find_elements_by_class_name('stream-item')) > len(items):
items
= wd.find_elements_by_class_name('stream-item')
print items
wd
.execute_script("window.scrollTo(0, document.body.scrollHeight);")
return items
def test():
get_page
('http://twitter.com/')
get_items
(wd.find_elements_by_class_name('stream-item'))

Note: The hard sleep is just for demonstrating that it works. Please use the waits package to wait for a smart condition instead.

  • You must to post comments
Showing 1 result
Your Answer
Post as a guest by filling out the fields below or if you already have an account.
Name*
E-mail*
Website