How To Webscrape Dynamic Websites In Python (without Using Selenium)

Are there any libraries or alternative methods to Selenium to scrape data from dynamic (javascript-rendered) websites? The issue I've run into is that many websites can detect when

Solution 1:

If you don't want to use selenium to scrape dynamic website. Two ways I know:

1. Find the ajax API and send GET request.That's could only use requests module or urllib module could do that.(I recommend this but it needs to take some measure.)

2. If your python verions >= 3.6,you could try to use requests-html module.As far as I know,it could get some text that's rendered by JavaScript.

Solution 2:

You can try libraries like requests-html. That is a simple solution, but if it doesn't work, it is built on top of pyppeteer and it does all the heavy lifting.