Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions plugins/Typesense/v1/configValidation.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
{
"steps": [
{
"displayName": "Authenticate and check collection",
"dataStream": { "name": "searchValidation" },
"required": true,
"error": "Could not query the collection. Check the Host URL, that the Search API Key is valid, and that it is authorised for the configured Collection.",
"success": "Connected successfully."
}
]
}
57 changes: 57 additions & 0 deletions plugins/Typesense/v1/dataStreams/documentSearch.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
{
"name": "documentSearch",
"displayName": "Document Search",
"description": "Documents from the configured collection matching a search query",
"tags": ["Search"],
"baseDataSourceName": "httpRequestUnscoped",
"config": {
"httpMethod": "get",
"endpointPath": "collections/{{dataSource.collection}}/documents/search",
"getArgs": [
{ "key": "q", "value": "{{q || '*'}}" },
{ "key": "query_by", "value": "{{query_by || null}}" },
{ "key": "filter_by", "value": "{{filter_by || null}}" },
{ "key": "sort_by", "value": "{{sort_by || null}}" },
{ "key": "per_page", "value": "{{per_page || 50}}" }
Comment on lines +11 to +15

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟑 Minor | ⚑ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '--- files ---\n'
git ls-files 'plugins/Typesense/v1/dataStreams/*.json'

printf '\n--- relevant snippets ---\n'
for f in plugins/Typesense/v1/dataStreams/documentSearch.json plugins/Typesense/v1/dataStreams/searchSummary.json; do
  echo "FILE: $f"
  nl -ba "$f" | sed -n '1,120p'
  echo
done

printf '\n--- search for query_by validation / docs references ---\n'
rg -n '"query_by"|query_by|filter_by|q != "\*"' plugins/Typesense/v1 -g '*.json' -g '*.md' -g '*.ts' -g '*.js'

Repository: squaredup/plugins

Length of output: 246


🏁 Script executed:

#!/bin/bash
set -euo pipefail

printf '--- file sizes ---\n'
wc -l plugins/Typesense/v1/dataStreams/documentSearch.json plugins/Typesense/v1/dataStreams/searchSummary.json

printf '\n--- surrounding content: documentSearch.json ---\n'
sed -n '1,120p' plugins/Typesense/v1/dataStreams/documentSearch.json | nl -ba

printf '\n--- surrounding content: searchSummary.json ---\n'
sed -n '1,120p' plugins/Typesense/v1/dataStreams/searchSummary.json | nl -ba

printf '\n--- query_by references ---\n'
rg -n '"query_by"|query_by|filter_by|q' plugins/Typesense/v1 -g '*.json'

Repository: squaredup/plugins

Length of output: 246


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Focus on the two stream files and nearby plugin metadata.
printf '--- plugin tree ---\n'
find plugins/Typesense -maxdepth 3 -type f | sort

printf '\n--- documentSearch.json ---\n'
cat -n plugins/Typesense/v1/dataStreams/documentSearch.json | sed -n '1,120p'

printf '\n--- searchSummary.json ---\n'
cat -n plugins/Typesense/v1/dataStreams/searchSummary.json | sed -n '1,120p'

printf '\n--- metadata/version files ---\n'
cat -n plugins/Typesense/v1/metadata.json 2>/dev/null || true

Repository: squaredup/plugins

Length of output: 246


🌐 Web query:

Typesense search API q wildcard query_by required documentation

πŸ’‘ Result:

In the Typesense Search API, both the 'q' and 'query_by' parameters are required [1][2][3]. The 'q' parameter accepts the text you wish to search for [1][4]. To perform a wildcard search that returns all documents (typically used in combination with filters), you should set 'q' to '*' [1][2]. The 'query_by' parameter is used to specify one or more field names that Typesense should query against [1][2]. You can provide multiple fields by separating them with a comma (e.g., 'field1,field2') [1][2]. The order of these fields determines relevance, where matches in earlier fields are ranked higher than those in later fields [1][2]. Only string and string array fields are supported for full-text search in this parameter [1][2].

Citations:


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Look for the same pattern in other Typesense stream definitions.
rg -n '"key": "query_by"|query_by.*required|required.*query_by|q.*\*' plugins/Typesense -g '*.json' -g '*.md' | sed -n '1,200p'

Repository: squaredup/plugins

Length of output: 2425


Require query_by for non-* searches
Both streams accept q values other than * while leaving query_by empty, which makes Typesense reject the request.

  • plugins/Typesense/v1/dataStreams/documentSearch.json
  • plugins/Typesense/v1/dataStreams/searchSummary.json
πŸ“ Affects 2 files
  • plugins/Typesense/v1/dataStreams/documentSearch.json#L11-L15 (this comment)
  • plugins/Typesense/v1/dataStreams/searchSummary.json#L11-L14
πŸ€– Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/Typesense/v1/dataStreams/documentSearch.json` around lines 11 - 15,
Require a non-empty query_by whenever q is not "*", while preserving wildcard
searches without query_by, in the request parameter mappings for
plugins/Typesense/v1/dataStreams/documentSearch.json lines 11-15 and
plugins/Typesense/v1/dataStreams/searchSummary.json lines 11-14; update both
streams consistently so invalid Typesense requests cannot be produced.

],
"postRequestScript": "documentSearch.js"
},
"matches": "none",
"ui": [
{
"type": "text",
"name": "q",
"label": "Query (q)",
"defaultValue": "*"
},
{
"type": "text",
"name": "query_by",
"label": "Query by fields",
"help": "Comma-separated fields to search; required unless q is *"
},
{
"type": "text",
"name": "filter_by",
"label": "Filter by"
},
{
"type": "text",
"name": "sort_by",
"label": "Sort by"
},
{
"type": "number",
"name": "per_page",
"label": "Max results",
"defaultValue": 50,
"help": "Typesense max is 250"
}
],
"manualConfigApply": true,
"metadata": [
{ "name": "_relevance", "displayName": "Relevance", "shape": "number" },
{ "pattern": ".*" }
],
"timeframes": false
}
32 changes: 32 additions & 0 deletions plugins/Typesense/v1/dataStreams/multiSearch.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
{
"name": "multiSearch",
"displayName": "Multi Search",
"description": "Documents from a raw Typesense multi_search request, flattened across all sub-searches",
"tags": ["Search"],
"baseDataSourceName": "httpRequestUnscoped",
"config": {
"httpMethod": "post",
"endpointPath": "multi_search",
"getArgs": [{ "key": "collection", "value": "{{dataSource.collection}}" }],
"postBody": "{{query}}",
"postRequestScript": "multiSearch.js"
},
"matches": "none",
"ui": [
{
"type": "code",
"name": "query",
"label": "Query (JSON)",
"language": "json",
"defaultValue": { "searches": [{ "q": "*" }] },
"help": "A Typesense multi_search body. Each search may set its own collection; otherwise the connection's configured collection is used."
}
],
"manualConfigApply": true,
"metadata": [
{ "name": "_search", "displayName": "Search #", "shape": "number" },
{ "name": "_relevance", "displayName": "Relevance", "shape": "number" },
{ "pattern": ".*" }
],
"timeframes": false
}
5 changes: 5 additions & 0 deletions plugins/Typesense/v1/dataStreams/scripts/documentSearch.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
// dataStreams/scripts/documentSearch.js
// Typesense search response: { found, out_of, hits: [ { document: {...}, text_match, highlight, highlights } ] }
// Documents have an arbitrary, unknown schema per collection, so flatten each hit's
// document fields to top-level columns and add the relevance score.
result = (data.hits || []).map((h) => ({ ...h.document, _relevance: h.text_match }));
6 changes: 6 additions & 0 deletions plugins/Typesense/v1/dataStreams/scripts/multiSearch.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
// dataStreams/scripts/multiSearch.js
// Typesense multi_search response: { results: [ { found, hits: [ { document:{...}, text_match } ] } ] }
// Flatten hits across all sub-searches, tagging which sub-search each row came from;
// our meta fields must win over document fields.
result = (data.results || []).flatMap((r, i) =>
(r.hits || []).map((h) => ({ ...h.document, _search: i, _relevance: h.text_match })));
Comment on lines +5 to +6

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟑 Minor | ⚑ Quick win

Use 1-based search numbering.

multiSearch.json displays _search as β€œSearch #”, but the script assigns zero-based i, so the first search appears as 0. Use i + 1, or rename the field to indicate that it is a zero-based index.

πŸ€– Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/Typesense/v1/dataStreams/scripts/multiSearch.js` around lines 5 - 6,
Update the _search assignment in the multi-search result mapping to use 1-based
numbering by assigning i + 1, so the first search is displayed as Search `#1`
while preserving the existing result mapping.

9 changes: 9 additions & 0 deletions plugins/Typesense/v1/dataStreams/scripts/searchSummary.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
// dataStreams/scripts/searchSummary.js
// Typesense search response root is an OBJECT: { found, out_of, search_time_ms, page, hits: [...] }
// Build a single summary row from the top-level counts, ignoring hits.
result = [{
found: data.found,
out_of: data.out_of,
search_time_ms: data.search_time_ms,
page: data.page
}];
44 changes: 44 additions & 0 deletions plugins/Typesense/v1/dataStreams/searchSummary.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
{
"name": "searchSummary",
"displayName": "Search Summary",
"description": "Match count, total document count and search time for a search query",
"tags": ["Search"],
"baseDataSourceName": "httpRequestUnscoped",
"config": {
"httpMethod": "get",
"endpointPath": "collections/{{dataSource.collection}}/documents/search",
"getArgs": [
{ "key": "q", "value": "{{q || '*'}}" },
{ "key": "query_by", "value": "{{query_by || null}}" },
{ "key": "filter_by", "value": "{{filter_by || null}}" },
{ "key": "per_page", "value": "0" }
],
"postRequestScript": "searchSummary.js"
},
"matches": "none",
"ui": [
{
"type": "text",
"name": "q",
"label": "Query (q)",
"defaultValue": "*"
},
{
"type": "text",
"name": "query_by",
"label": "Query by fields"
},
{
"type": "text",
"name": "filter_by",
"label": "Filter by"
}
],
"metadata": [
{ "name": "found", "displayName": "Matching Documents", "shape": "number", "role": "value" },
{ "name": "out_of", "displayName": "Total Documents", "shape": "number" },
{ "name": "search_time_ms", "displayName": "Search Time", "shape": "milliseconds" },
{ "name": "page", "displayName": "Page", "shape": "number", "visible": false }
],
"timeframes": false
}
20 changes: 20 additions & 0 deletions plugins/Typesense/v1/dataStreams/searchValidation.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"name": "searchValidation",
"displayName": "Search Validation",
"description": "Up to one document from the configured collection",
"tags": ["Search"],
"baseDataSourceName": "httpRequestUnscoped",
"config": {
"httpMethod": "get",
"endpointPath": "collections/{{dataSource.collection}}/documents/search",
"getArgs": [
{ "key": "q", "value": "*" },
{ "key": "per_page", "value": "1" }
],
"pathToData": "hits"
},
"matches": "none",
"metadata": [{ "pattern": ".*" }],
"timeframes": false,
"visibility": { "type": "hidden" }
}
3 changes: 3 additions & 0 deletions plugins/Typesense/v1/defaultContent/manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{
"items": [{ "name": "overviewDashboard", "type": "dashboard" }]
}
139 changes: 139 additions & 0 deletions plugins/Typesense/v1/defaultContent/overviewDashboard.dash.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,139 @@
{
"name": "Overview",
"schemaVersion": "1.5",
"timeframe": "last24hours",
"dashboard": {
"_type": "layout/grid",
"columns": 4,
"version": 1,
"contents": [
{
"i": "1e102028-7d08-4220-b038-9acab1d00e7a",
"x": 0,
"y": 0,
"w": 4,
"h": 1,
"moved": false,
"static": false,
"z": 0,
"config": {
"_type": "tile/text",
"title": "",
"description": "",
"visualisation": {
"config": {
"content": "Showing results from the configured Typesense collection.",
"fontSize": 14,
"align": "left",
"autoSize": true
}
}
}
},
{
"i": "cc0e7c1c-a05f-47b8-8838-0597691526c3",
"x": 0,
"y": 1,
"w": 2,
"h": 2,
"moved": false,
"static": false,
"z": 0,
"config": {
"_type": "tile/data-stream",
"title": "Total Documents",
"description": "",
"activePluginConfigIds": ["{{configId}}"],
"dataStream": {
"id": "{{dataStreams.[searchSummary]}}",
"name": "searchSummary",
"pluginConfigId": "{{configId}}",
"dataSourceConfig": {
"q": "*"
}
},
"timeframe": "none",
"visualisation": {
"type": "data-stream-scalar",
"config": {
"data-stream-scalar": {
"value": "found",
"comparisonColumn": "none",
"label": "Matching Documents"
}
}
}
}
},
{
"i": "693f6164-c6e5-4906-b77b-4c1da5222915",
"x": 2,
"y": 1,
"w": 2,
"h": 2,
"moved": false,
"static": false,
"z": 0,
"config": {
"_type": "tile/data-stream",
"title": "Search Time",
"description": "",
"activePluginConfigIds": ["{{configId}}"],
"dataStream": {
"id": "{{dataStreams.[searchSummary]}}",
"name": "searchSummary",
"pluginConfigId": "{{configId}}",
"dataSourceConfig": {
"q": "*"
}
},
"timeframe": "none",
"visualisation": {
"type": "data-stream-scalar",
"config": {
"data-stream-scalar": {
"value": "search_time_ms",
"comparisonColumn": "none",
"label": "Search Time"
}
}
}
}
},
{
"i": "8b4d3946-efa0-4180-bc02-5a20ed445cef",
"x": 0,
"y": 3,
"w": 4,
"h": 6,
"moved": false,
"static": false,
"z": 0,
"config": {
"_type": "tile/data-stream",
"title": "Documents",
"description": "",
"activePluginConfigIds": ["{{configId}}"],
"dataStream": {
"id": "{{dataStreams.[documentSearch]}}",
"name": "documentSearch",
"pluginConfigId": "{{configId}}",
"dataSourceConfig": {
"q": "*",
"per_page": 25
}
},
"timeframe": "none",
"visualisation": {
"type": "data-stream-table",
"config": {
"data-stream-table": {
"transpose": false
}
}
}
}
}
]
}
}
1 change: 1 addition & 0 deletions plugins/Typesense/v1/defaultContent/scopes.json
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
[]
61 changes: 61 additions & 0 deletions plugins/Typesense/v1/docs/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,61 @@
# Typesense

Query documents from your [Typesense](https://typesense.org) collections directly in SquaredUp. This plugin runs searches against a Typesense Cloud (or self-hosted) instance and returns matching documents as rows you can chart, table, and build dashboards from.

It is designed to work with a **search-only API key**, so it never needs admin access to your cluster.

## What this plugin does

- Runs structured searches against a single collection and returns one row per matching document.
- Runs raw `multi_search` queries from a JSON blob for federated / multi-collection / advanced queries.
- Returns search result counts (matching documents, total documents scanned, timing).

Because a search-only key cannot list collections (`collections:list` requires an admin key), this plugin does **not** import your collections or documents as objects into the SquaredUp graph. You choose the collection to query in the plugin configuration.

## Prerequisites β€” getting a search API key

1. Log in to the [Typesense Cloud dashboard](https://cloud.typesense.org) (or your self-hosted admin tooling).
2. Note your cluster's **nodes hostname** β€” it looks like `xxxxxxxxx.a1.typesense.net`. Your base URL is `https://<that-host>`.
3. Create a **search-only API key**:
- Using an admin key, call the [Create API Key](https://typesense.org/docs/latest/api/api-keys.html) endpoint with the `documents:search` action, scoped to the collection(s) you want to expose. Example:
```
curl "https://<host>/keys" \
-X POST \
-H "X-TYPESENSE-API-KEY: <ADMIN_KEY>" \
-H "Content-Type: application/json" \
-d '{"description":"SquaredUp search key","actions":["documents:search"],"collections":["<your-collection>"]}'
```
- Copy the returned `value` β€” this is your search-only key. It is shown only once.
4. Confirm the key works:
```
curl "https://<host>/collections/<your-collection>/documents/search?q=*&per_page=1" \
-H "X-TYPESENSE-API-KEY: <SEARCH_KEY>"
```
Comment on lines +21 to +33

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

πŸ“ Maintainability & Code Quality | 🟑 Minor | ⚑ Quick win

Make the setup code fences lint-clean.

The two fenced examples need bash language identifiers and blank lines before and after the fences. Markdownlint reports MD031 and MD040 for these lines.

Proposed formatting fix
    - Using an admin key, call the [Create API Key](https://typesense.org/docs/latest/api/api-keys.html) endpoint with the `documents:search` action, scoped to the collection(s) you want to expose. Example:
+     
-     ```
+     ```bash
      curl "https://<host>/keys" \
...
      -d '{"description":"SquaredUp search key","actions":["documents:search"],"collections":["<your-collection>"]}'
-     ```
+     
...
  4. Confirm the key works:
-  ```
+  
+  ```bash
    curl "https://<host>/collections/<your-collection>/documents/search?q=*&per_page=1" \
      -H "X-TYPESENSE-API-KEY: <SEARCH_KEY>"
-  ```
+  
πŸ“ Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
```
curl "https://<host>/keys" \
-X POST \
-H "X-TYPESENSE-API-KEY: <ADMIN_KEY>" \
-H "Content-Type: application/json" \
-d '{"description":"SquaredUp search key","actions":["documents:search"],"collections":["<your-collection>"]}'
```
- Copy the returned `value` β€” this is your search-only key. It is shown only once.
4. Confirm the key works:
```
curl "https://<host>/collections/<your-collection>/documents/search?q=*&per_page=1" \
-H "X-TYPESENSE-API-KEY: <SEARCH_KEY>"
```
🧰 Tools
πŸͺ› markdownlint-cli2 (0.23.0)

[warning] 21-21: Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


[warning] 21-21: Fenced code blocks should have a language specified

(MD040, fenced-code-language)


[warning] 27-27: Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


[warning] 30-30: Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)


[warning] 30-30: Fenced code blocks should have a language specified

(MD040, fenced-code-language)


[warning] 33-33: Fenced code blocks should be surrounded by blank lines

(MD031, blanks-around-fences)

πŸ€– Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@plugins/Typesense/v1/docs/README.md` around lines 21 - 33, Update the two
setup examples in the README to use bash language identifiers on their fenced
code blocks and add blank lines immediately before and after each fence.
Preserve the existing curl commands and surrounding setup text.

Source: Linters/SAST tools

You should get a JSON response containing a `found` count.

## Configuration fields

| Field | Required | What it is / where to find it |
|-------|----------|-------------------------------|
| **Host URL** | Yes | Your Typesense base URL including scheme, e.g. `https://xxxxxxxxx.a1.typesense.net`. No trailing slash. From the Typesense Cloud dashboard (nodes hostname) or your self-hosted address. |
| **Search API key** | Yes | A search-only API key (`documents:search` action). See "getting a search API key" above. Stored securely. |
| **Collection** | Yes | The name of the collection this connection queries, e.g. `products`. The key must be authorised for this collection. Add another connection for another collection. |

When you save the configuration, the plugin runs a lightweight `q=*` search against the collection to confirm the host, key, and collection are all valid.

## What gets indexed

Nothing. This is a query-only plugin β€” it imports no objects into the SquaredUp graph. Data is retrieved live each time a tile runs.

## Data streams

- **Document Search** β€” structured search against the configured collection. Tile parameters: `q` (query, default `*`), `query_by`, `filter_by`, `sort_by`, `per_page`. Returns one row per matching document, with the document's own fields as columns plus a relevance score.
- **Multi Search** β€” runs a raw `multi_search` JSON blob (e.g. `{"searches":[{"q":"*"}]}`) for federated or advanced queries. Returns one row per hit across all sub-searches.
- **Search Summary** β€” returns a single row with the number of matching documents, total documents scanned, and search time.

## Known limitations

- **Search-only scope.** No cluster health, metrics, API-key or alias management, and no collection/document import β€” these require an admin key.
- **Result size.** A single search returns at most 250 documents per page (`per_page`). Search is designed to return the most relevant results, not to bulk-export a collection.
- **Response size.** Very large result sets can exceed SquaredUp's ~6MB per-request limit β€” keep `per_page` and returned field counts reasonable, or use `filter_by` to narrow results.
- **No time range.** Typesense search has no built-in time-range parameter, so these streams return current results with no timeframe picker. To restrict by time, add a `filter_by` on a timestamp field in your documents.
Loading
Loading