How can I parse a website using Selenium and Beautifulsoup in python? [closed]


New to programming and figured out how to navigate to where I need to go using Selenium. I’d like to parse the data now but not sure where to start. Can someone hold my hand a sec and point me in the right direction?

Any help appreciated –

  • You must to post comments

Assuming you are on the page you want to parse, Selenium stores the source HTML in the driver’s page_source attribute. You would then load the page_source into BeautifulSoup as follows:

In [8]: from bs4 import BeautifulSoup
In [9]: from selenium import webdriver
In [10]: driver = webdriver.Firefox()
In [11]: driver.get('')
In [12]: html = driver.page_source
In [13]: soup = BeautifulSoup(html)
In [14]: for tag in soup.find_all('title'):
....: print tag.text
Hacker News
  • You must to post comments
Showing 1 result
Your Answer
Post as a guest by filling out the fields below or if you already have an account.