These are some of the solutions for web-scraping toscrape.com.
This includes these endpoints/challenges:
- Default
- Scroll
- Delayed
You can remove the use of proxies if you want.
Run main.py to return a list of valid_proxies in 'valid_proxies.txt' from 'proxies.txt'
run sele.py and its various functions as the solutions
I used a combination of BeautifulSoup, Selenium, and requests to solve these challenges.