Add Scrapiq remote MCP server (URL to clean text/markdown/JSON for RAG) - #5169
Open
NG-PR0JECT wants to merge 1 commit into
Open
NG-PR0JECT wants to merge 1 commit into
NG-PR0JECT wants to merge 1 commit into
Conversation
Scrapiq turns any URL into clean text, markdown, or JSON for LLM and RAG pipelines. It is a remote streamable-http server, anonymous (no OAuth) and MIT licensed; tools are discovered dynamically. - Endpoint: https://scrapiq.io/mcp - Source: https://github.com/NG-PR0JECT/scrapiq-mcp-server - Docs: https://scrapiq.io/docs
Author
|
Gentle ping — the two runs on this branch (
Anything needed from my side for the runs to be approved, or a maintainer action to wait on? Happy to change the category or tag set if the catalogue prefers something else. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
Scrapiq — a remote MCP server that turns any URL into clean text, markdown, or JSON for LLM and RAG pipelines.
streamable-http, anonymous (no OAuth, no API key, no account)https://scrapiq.io/mcpWhy it is useful in the Toolkit
Feeding raw HTML into retrieval silently degrades retrieval quality and inflates token spend; most scrapers either drag in navigation/ads/boilerplate or fail silently and return an empty string with no error. Scrapiq exposes one tool,
scrapiq_extract(url, format, max_chars), which returns title, content, links and metadata for a page — no LLM in the loop, and no extraction keys to configure.It pairs with the local filesystem/git servers: an agent can pull a page (release notes, a docs page, a competitor's pricing page) into clean markdown and then write it to disk, with no API credentials to paste into the Toolkit UI.
Verification performed
initialize→ protocol2025-06-18;tools/list→ 1 tool (scrapiq_extract) against the public endpoint, verified with the official MCP Python SDK over plain HTTPS.POST /mcpreturns200/202(notifications),GET /mcpreturns405— stateless by design, which is expected for a remote server.trafilatura/readability-lxml/MarkItDownon 23 live pages (raw output, method and caveats published): https://github.com/NG-PR0JECT/scrapiq-benchNotes
tools.jsonis[]because tools are discovered dynamically from the remote endpoint.