Skip to content

Add Scrapiq remote MCP server (URL to clean text/markdown/JSON for RAG) - #5169

Open
NG-PR0JECT wants to merge 1 commit into
docker:mainfrom
NG-PR0JECT:add-scrapiq
Open

NG-PR0JECT wants to merge 1 commit into
docker:mainfrom
NG-PR0JECT:add-scrapiq

Conversation

@NG-PR0JECT

Copy link
Copy Markdown

What this adds

Scrapiq — a remote MCP server that turns any URL into clean text, markdown, or JSON for LLM and RAG pipelines.

Why it is useful in the Toolkit

Feeding raw HTML into retrieval silently degrades retrieval quality and inflates token spend; most scrapers either drag in navigation/ads/boilerplate or fail silently and return an empty string with no error. Scrapiq exposes one tool, scrapiq_extract(url, format, max_chars), which returns title, content, links and metadata for a page — no LLM in the loop, and no extraction keys to configure.

It pairs with the local filesystem/git servers: an agent can pull a page (release notes, a docs page, a competitor's pricing page) into clean markdown and then write it to disk, with no API credentials to paste into the Toolkit UI.

Verification performed

  • initialize → protocol 2025-06-18; tools/list → 1 tool (scrapiq_extract) against the public endpoint, verified with the official MCP Python SDK over plain HTTPS.
  • POST /mcp returns 200/202 (notifications), GET /mcp returns 405 — stateless by design, which is expected for a remote server.
  • Public benchmark vs trafilatura / readability-lxml / MarkItDown on 23 live pages (raw output, method and caveats published): https://github.com/NG-PR0JECT/scrapiq-bench

Notes

  • tools.json is [] because tools are discovered dynamically from the remote endpoint.
  • No secrets or environment variables are needed to enable the server.

Scrapiq turns any URL into clean text, markdown, or JSON for LLM and RAG
pipelines. It is a remote streamable-http server, anonymous (no OAuth) and
MIT licensed; tools are discovered dynamically.

- Endpoint: https://scrapiq.io/mcp
- Source: https://github.com/NG-PR0JECT/scrapiq-mcp-server
- Docs: https://scrapiq.io/docs
@NG-PR0JECT
NG-PR0JECT requested a review from a team as a code owner September 20, 2026 06:06
@NG-PR0JECT

Copy link
Copy Markdown
Author

Gentle ping — the two runs on this branch (.github/workflows/ci.yaml and Security Review Trigger) are both conclusion: action_required with 0 s runtime, which I read as the first-time-contributor approval gate rather than a problem with the submission. Nothing else here is waiting on me:

  • the PR adds three files under servers/scrapiq/ (readme.md, server.yaml, tools.json), and is MERGEABLE;
  • npx prettier --check passes on all three exactly as committed;
  • source.commit fb3e050 is reachable from NG-PR0JECT/scrapiq-mcp-server@main, and that repository is MIT-licensed;
  • the remote declared in server.yaml answers right now: POST https://scrapiq.io/mcp with an initialize returns protocol 2025-03-26 and serverInfo scrapiq-mcp-server 0.2.0, and tools/list returns scrapiq_extract;
  • dynamic: {tools: true} is set, so CI never has to reach the endpoint, and tools.json is [] accordingly.

Anything needed from my side for the runs to be approved, or a maintainer action to wait on? Happy to change the category or tag set if the catalogue prefers something else.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants