crawling-python

🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl! Don't be shy, join here: https://discord.gg/EMgGbDceNQ and follow here for daily tips and tricks: https://x.com/Scrapling_dev

List of libraries, tools and APIs for web scraping and data processing.

Transform Web Content into LLM-Ready Data

Scalable Python web scraping scripts for +40 popular domains

Script that crawls meta data from ICLR OpenReview webpage. Tutorials on installing and using Selenium and ChromeDriver on Ubuntu.

🕷 Automatically detects changes to the official Telegram sites, beta clients, MTProto servers and mini apps

TLS Requests is a powerful Python library for secure HTTP requests, offering browser-like TLS client, fingerprinting, anti-bot page bypass, and high performance.

The script for selenium in python. Make automated testing easier! 使用json脚本驱动selenium

A universal solution for web crawling lists. 抓取网页列表的通用解决方案

BFS and DFS BASED Project.A neural network-powered web crawler that intelligently extracts, classifies, and processes web content using deep learning

Python-based web crawling script with randomized intervals, user-agent rotation, and proxy server IP rotation to outsmart website bots and prevent blocking.

此项目分享安卓逆向的实战案例以及学习笔记，适合新手学习，随着作者逐渐变成大神，这个仓库也会适合大神学习~

Instagram user's photos and videos downloader. Download all media files from any username. Working 2022!

Free Facebook pages MetaData Scraping Library - Unlimited Calls

NewsCrap adalah alat scraping berita Google berbasis Command Line Interface (CLI) yang dirancang untuk riset, investigasi, dan pengumpulan data OSINT. Dengan fitur canggih seperti rotation proxy, scheduling otomatis, dan multi-format export, alat ini memudahkan pengumpulan data berita secara efisien dan andal.

Web scraping API for building AI applications.

API to parse tibia.com content into python objects.

File Crawler index files and search hard-coded credentials

🚀 OFFICIAL STARTER TEMPLATE FOR BOTASAURUS SCRAPING FRAMEWORK 🤖

项目已经移动至：https://github.com/BaiduSpider/BaiduSpider ！！ 一个爬取百度搜索结果的爬虫，目前支持百度网页搜索，百度图片搜索，百度知道搜索，百度视频搜索，百度资讯搜索，百度文库搜索，百度经验搜索和百度百科搜索。