NaiboWang/EasySpider
A visual no-code/code-free web crawler/spider一个可视化爬虫软件，可以无代码图形化设计和执行的爬虫任务
Language: JavaScript
Total stars: 3595
Stars trend:
12 May 2023
#javascript
#batch, #batchprocessing, #batchscript, #codefree, #crawler, #frontend, #gui, #html, #inputparameters, #layman, #parameters, #services, #spider, #visual, #visualization, #visualprogramming, #visulization, #web, #www
A visual no-code/code-free web crawler/spider一个可视化爬虫软件，可以无代码图形化设计和执行的爬虫任务
Language: JavaScript
Total stars: 3595
Stars trend:
12 May 2023
12am █▏ +9
1am █▊ +14
2am ██▊ +22
3am █▉ +15
4am █▊ +14
5am ███ +24
6am ██▊ +22
7am █▍ +11
8am ▉ +7
9am ▎ +2
10am ▉ +7
11am ▋ +5
#javascript
#batch, #batchprocessing, #batchscript, #codefree, #crawler, #frontend, #gui, #html, #inputparameters, #layman, #parameters, #services, #spider, #visual, #visualization, #visualprogramming, #visulization, #web, #www
adbar/trafilatura
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
Language: Python
Total stars: 1562
Stars trend:
14 Aug 2023
15 Aug 2023
#python
#articleextractor, #corpus, #corpusbuilder, #corpustools, #crawler, #htmltomarkdown, #html2text, #news, #newsaggregator, #newscrawler, #nlp, #readability, #rssfeed, #scraping, #tei, #textcleaning, #textextraction, #textmining, #textpreprocessing, #webscraping
Python & command-line tool to gather text on the Web: web crawling/scraping, extraction of text, metadata, comments
Language: Python
Total stars: 1562
Stars trend:
14 Aug 2023
7pm ██▎ +18
8pm ███▎ +26
9pm ███▊ +30
10pm ██▍ +19
11pm ██ +16
15 Aug 2023
12am ██ +16
1am █▊ +14
2am ███ +24
3am █▋ +13
4am ██▌ +20
5am ██▌ +20
#python
#articleextractor, #corpus, #corpusbuilder, #corpustools, #crawler, #htmltomarkdown, #html2text, #news, #newsaggregator, #newscrawler, #nlp, #readability, #rssfeed, #scraping, #tei, #textcleaning, #textextraction, #textmining, #textpreprocessing, #webscraping
apify/crawlee
Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.
Language:TypeScript
Total stars: 9760
Stars trend:
#typescript
#apify, #automation, #crawler, #crawling, #headless, #headlesschrome, #javascript, #nodejs, #npm, #playwright, #puppeteer, #scraper, #scraping, #typescript, #webcrawler, #webcrawling, #webscraping
Crawlee—A web scraping and browser automation library for Node.js that helps you build reliable crawlers. Fast.
Language:TypeScript
Total stars: 9760
Stars trend:
22 Nov 2023
12am █▍ +11
1am ▊ +6
2am █▏ +9
3am ▍ +3
4am ▎ +2
5am ▍ +3
6am ▎ +2
7am ▋ +5
8am ████▋ +37
9am ████▉ +39
10am ███▏ +25
11am ██ +16#typescript
#apify, #automation, #crawler, #crawling, #headless, #headlesschrome, #javascript, #nodejs, #npm, #playwright, #puppeteer, #scraper, #scraping, #typescript, #webcrawler, #webcrawling, #webscraping
iawia002/lux
👾 Fast and simple video download library and CLI tool written in Go
Language:Go
Total stars: 23594
Stars trend:
#go
#bilibili, #crawler, #download, #downloader, #go, #golang, #iqiyi, #qq, #scraper, #tumblr, #video, #youku, #youtube
👾 Fast and simple video download library and CLI tool written in Go
Language:Go
Total stars: 23594
Stars trend:
17 Mar 2024
8pm ▏ +1
9pm +0
10pm +0
11pm +0
18 Mar 2024
12am ██▏ +17
1am █▊ +14
2am ██▍ +19
3am █▌ +12
4am █▏ +9
5am ▊ +6#go
#bilibili, #crawler, #download, #downloader, #go, #golang, #iqiyi, #qq, #scraper, #tumblr, #video, #youku, #youtube
Evil0ctal/Douyin_TikTok_Download_API
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
Language:Python
Total stars: 6131
Stars trend:
#python
#aiohttp, #api, #asgi, #async, #asyncio, #crawler, #douyin, #douyinscraper, #douyintiktokapi, #douyintiktokdownload, #fastapi, #nowatermark, #onlineparsing, #python, #pywebio, #scraper, #spider, #tiktok, #tiktokscraper, #webscraping
🚀「Douyin_TikTok_Download_API」是一个开箱即用的高性能异步抖音、快手、TikTok、Bilibili数据爬取工具，支持API调用，在线批量解析及下载。
Language:Python
Total stars: 6131
Stars trend:
1 Apr 2024
1pm ██▍ +19
2pm ██▏ +17
3pm █▏ +9
4pm ▌ +4
5pm ▌ +4
6pm ▍ +3
7pm ▎ +2
8pm ▍ +3
9pm ▏ +1
10pm ▏ +1
11pm ▍ +3
2 Apr 2024
12am █▏ +9#python
#aiohttp, #api, #asgi, #async, #asyncio, #crawler, #douyin, #douyinscraper, #douyintiktokapi, #douyintiktokdownload, #fastapi, #nowatermark, #onlineparsing, #python, #pywebio, #scraper, #spider, #tiktok, #tiktokscraper, #webscraping
mendableai/firecrawl
🔥 Turn entire websites into LLM-ready markdown
Language:TypeScript
Total stars: 106
Stars trend:
#typescript
#ai, #crawler, #data, #htmltomarkdown, #llm, #markdown, #rag, #scraper, #scraping, #webcrawler
🔥 Turn entire websites into LLM-ready markdown
Language:TypeScript
Total stars: 106
Stars trend:
16 Apr 2024
5pm ▋ +5
6pm ██▍ +19
7pm ██▎ +18
8pm █▊ +14
9pm █▎ +10
10pm █ +8
11pm ▎ +2
17 Apr 2024
12am ██▎ +18#typescript
#ai, #crawler, #data, #htmltomarkdown, #llm, #markdown, #rag, #scraper, #scraping, #webcrawler
coder-hxl/x-crawl
x-crawl is a flexible Node.js AI-assisted crawler library. Making crawler work more efficient, intelligent and convenient. ------ x-crawl 是一个灵活的 Node.js AI 辅助爬虫库。使爬虫工作变得更加高效、智能和便捷。
Language:TypeScript
Total stars: 939
Stars trend:
#typescript
#ai, #aicrawl, #chromium, #crawl, #crawler, #fingerprint, #flexible, #javascript, #multifunction, #nodejs, #puppeteer, #spider, #typescript
x-crawl is a flexible Node.js AI-assisted crawler library. Making crawler work more efficient, intelligent and convenient. ------ x-crawl 是一个灵活的 Node.js AI 辅助爬虫库。使爬虫工作变得更加高效、智能和便捷。
Language:TypeScript
Total stars: 939
Stars trend:
19 Apr 2024
12am ▍ +3
1am █▉ +15
2am ██ +16
3am █▌ +12
4am ▋ +5
5am ▍ +3
6am █ +8
7am █▏ +9
8am ▍ +3
9am ▏ +1#typescript
#ai, #aicrawl, #chromium, #crawl, #crawler, #fingerprint, #flexible, #javascript, #multifunction, #nodejs, #puppeteer, #spider, #typescript
ngc660sec/NGCBot
A WeChat robot based on the ✨HOOK mechanism, supporting 🌱security news regular push [FreeBuf, Prophet, Security Guest, Qi Anxin Attack and Defense Community], 👯Kfc copywriting, ⚡️Registration query, ⚡️Mobile phone number location query, ⚡️WHOIS information query, 🎉 Horoscope query, ⚡️Weather query, 🌱Fishing calendar, ⚡️Microstep threat intelligence query, 🐛Beauty video, ⚡️Beauty pictures, 👯Help menu. 📫 Supports points function, ⚡️supports automatic pulling of people, ⚡️detects ads, 🌱automatic group sending, 👯Ai reply, 😄rich in customization, even beginners can easily get started!
Language:Python
Total stars: 691
Stars trend:
#python
#bot, #crawler, #security, #weixin, #weixin, #wxbot
A WeChat robot based on the ✨HOOK mechanism, supporting 🌱security news regular push [FreeBuf, Prophet, Security Guest, Qi Anxin Attack and Defense Community], 👯Kfc copywriting, ⚡️Registration query, ⚡️Mobile phone number location query, ⚡️WHOIS information query, 🎉 Horoscope query, ⚡️Weather query, 🌱Fishing calendar, ⚡️Microstep threat intelligence query, 🐛Beauty video, ⚡️Beauty pictures, 👯Help menu. 📫 Supports points function, ⚡️supports automatic pulling of people, ⚡️detects ads, 🌱automatic group sending, 👯Ai reply, 😄rich in customization, even beginners can easily get started!
Language:Python
Total stars: 691
Stars trend:
16 May 2024
3am ▍ +3
4am +0
5am ▏ +1
6am +0
7am ▎ +2
8am ▎ +2
9am ██▋ +21
10am █▍ +11
11am █▊ +14
12pm █▊ +14
1pm ▉ +7#python
#bot, #crawler, #security, #weixin, #weixin, #wxbot
projectdiscovery/katana
A next-generation crawling and spidering framework.
Language:Go
Total stars: 9079
Stars trend:
#go
#cli, #crawler, #gocrawler, #headless, #spiderframework, #webspider
A next-generation crawling and spidering framework.
Language:Go
Total stars: 9079
Stars trend:
18 Jun 2024
7am ▎ +2
8am ▏ +1
9am ▍ +3
10am ▌ +4
11am █▏ +9
12pm █ +8
1pm █▏ +9
2pm ██ +16
3pm █▊ +14
4pm █▏ +9#go
#cli, #crawler, #gocrawler, #headless, #spiderframework, #webspider
apify/crawlee-python
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Language:Python
Total stars: 208
Stars trend:
#python
#apify, #automation, #beautifulsoup, #crawler, #crawling, #headless, #headlesschrome, #pip, #playwright, #python, #scraper, #scraping, #webcrawler, #webcrawling, #webscraping
Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.
Language:Python
Total stars: 208
Stars trend:
9 Jul 2024
7am ██▏ +17
8am █▍ +11
9am █▎ +10
10am ███▏ +25
11am ██▊ +22
12pm ███ +24
1pm ███▍ +27#python
#apify, #automation, #beautifulsoup, #crawler, #crawling, #headless, #headlesschrome, #pip, #playwright, #python, #scraper, #scraping, #webcrawler, #webcrawling, #webscraping
D4Vinci/Scrapling
Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python
Language:Python
Total stars: 268
Stars trend:
#python
#automation, #crawler, #crawling, #crawlingpython, #css, #dommanipulation, #hacktoberfest, #lxml, #playwright, #python, #python3, #scraping, #selectors, #selenium, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
Undetectable, Lightning-Fast, and Adaptive Web Scraping for Python
Language:Python
Total stars: 268
Stars trend:
11 Nov 2024
6am ▌ +4
7am ▌ +4
8am ▏ +1
9am ▏ +1
10am ▏ +1
11am ▋ +5
12pm ▎ +2
1pm ▌ +4
2pm ▉ +7
3pm ██▋ +21
4pm █▊ +14
5pm ███▏ +25#python
#automation, #crawler, #crawling, #crawlingpython, #css, #dommanipulation, #hacktoberfest, #lxml, #playwright, #python, #python3, #scraping, #selectors, #selenium, #stealth, #webscraper, #webscraping, #webscrapingpython, #webscraping, #xpath
xiaoxiunique/x-kit
一个用于抓取和分析 X (Twitter) 用户数据和推文的工具。
Language:TypeScript
Total stars: 167
Stars trend:
#typescript
#crawler, #kols, #twitter, #x
一个用于抓取和分析 X (Twitter) 用户数据和推文的工具。
Language:TypeScript
Total stars: 167
Stars trend:
26 Dec 2024
6pm ▏ +1
7pm +0
8pm ▋ +5
9pm ▏ +1
10pm ▍ +3
11pm █ +8
27 Dec 2024
12am █▏ +9
1am █▎ +10
2am █▉ +15
3am █▎ +10
4am ▋ +5
5am █▏ +9#typescript
#crawler, #kols, #twitter, #x
mendableai/firecrawl
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
Language:TypeScript
Total stars: 21906
Stars trend:
#typescript
#ai, #aiscraping, #crawler, #data, #htmltomarkdown, #llm, #markdown, #rag, #scraper, #scraping, #webcrawler, #webscraping
🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
Language:TypeScript
Total stars: 21906
Stars trend:
20 Jan 2025
5pm ▏ +1
6pm ▎ +2
7pm ▉ +7
8pm ▊ +6
9pm ▉ +7
10pm █ +8
11pm ▋ +5
21 Jan 2025
12am ▉ +7
1am ▌ +4
2am ▍ +3
3am ██ +16
4am █▍ +11#typescript
#ai, #aiscraping, #crawler, #data, #htmltomarkdown, #llm, #markdown, #rag, #scraper, #scraping, #webcrawler, #webscraping