From 4ff123508c0d0fbbda71f026556d0eaba9d93e4a Mon Sep 17 00:00:00 2001 From: Oleksandr Cherevach Date: Wed, 2 Sep 2026 12:23:11 +0300 Subject: [PATCH] docs(config): document file data source upload limits Add the three settings introduced by EPMCDME-13212 (codemie MR !3939) to the File Storage section of the API configuration reference: - FILE_DATASOURCE_MAX_UPLOAD_COUNT (default 10) replaces the hardcoded 10-file cap on File data source create and update - FILE_DATASOURCE_MAX_UPLOAD_TOTAL_SIZE (default 1 GB) caps the combined size of one upload request - FILE_DATASOURCE_UPLOAD_MAX_WORKERS (default 3) controls concurrent file writes to storage Also note that the configured file count is advertised via GET /v1/info (fileDatasourceMaxUploadCount) and consumed by the UI (codemie-ui MR !1645). Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_01AxXUhubD1gC1UaeABscyQw --- .../codemie/api-configuration.md | 23 +++++++++++++------ 1 file changed, 16 insertions(+), 7 deletions(-) diff --git a/docs/admin/configuration/codemie/api-configuration.md b/docs/admin/configuration/codemie/api-configuration.md index 64269122..b3172f88 100644 --- a/docs/admin/configuration/codemie/api-configuration.md +++ b/docs/admin/configuration/codemie/api-configuration.md @@ -286,13 +286,22 @@ Configure where and how CodeMie stores uploaded files, attachments, and generate ### General Storage Settings -| Parameter | Type | Default | Description | -| ------------------------------- | ------- | --------------------- | ----------------------------------------------------------------------------------------------------------- | -| `FILES_STORAGE_TYPE` | string | `"filesystem"` | Storage backend: `filesystem` (local on pod), `aws` (S3), `azure` (blob), `gcp` (bucket) | -| `FILES_STORAGE_DIR` | string | `"./codemie-storage"` | Local directory path when using `filesystem` storage type | -| `FILES_STORAGE_MAX_UPLOAD_SIZE` | integer | `104857600` | Maximum file size in bytes (100 MB default); increase for large document processing | -| `REPOS_LOCAL_DIR` | string | `"./codemie-repos"` | Directory for cloned Git repositories during code indexing | -| `IMAGE_INDEXING_MAX_SIZE_BYTES` | integer | `10485760` | Maximum image file size in bytes (10 MB) during datasource indexing; files exceeding this limit are skipped | +| Parameter | Type | Default | Description | +| --------------------------------------- | ------- | --------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `FILES_STORAGE_TYPE` | string | `"filesystem"` | Storage backend: `filesystem` (local on pod), `aws` (S3), `azure` (blob), `gcp` (bucket) | +| `FILES_STORAGE_DIR` | string | `"./codemie-storage"` | Local directory path when using `filesystem` storage type | +| `FILES_STORAGE_MAX_UPLOAD_SIZE` | integer | `104857600` | Maximum file size in bytes (100 MB default); increase for large document processing | +| `FILE_DATASOURCE_MAX_UPLOAD_COUNT` | integer | `10` | Maximum number of files in a File data source; checked on creation and, on update, against retained plus newly uploaded files combined. Must be greater than `0` | +| `FILE_DATASOURCE_MAX_UPLOAD_TOTAL_SIZE` | integer | `1073741824` | Maximum combined size in bytes of all files in one File data source create or update request (1 GB default). Does not scale automatically; raise it together with `FILES_STORAGE_MAX_UPLOAD_SIZE` or `FILE_DATASOURCE_MAX_UPLOAD_COUNT` | +| `FILE_DATASOURCE_UPLOAD_MAX_WORKERS` | integer | `3` | Concurrent workers that write uploaded File data source files to storage. Each worker holds one full file in memory, so higher values trade peak memory for upload throughput. Must be greater than `0` | +| `REPOS_LOCAL_DIR` | string | `"./codemie-repos"` | Directory for cloned Git repositories during code indexing | +| `IMAGE_INDEXING_MAX_SIZE_BYTES` | integer | `10485760` | Maximum image file size in bytes (10 MB) during datasource indexing; files exceeding this limit are skipped | + +:::info File data source upload limits +`FILE_DATASOURCE_MAX_UPLOAD_COUNT` and `FILE_DATASOURCE_MAX_UPLOAD_TOTAL_SIZE` are enforced on `POST /v1/index/knowledge_base/file` and `PUT /v1/index/knowledge_base/file`. Requests that exceed either limit are rejected with HTTP 422 (`Too many files` or `Total upload size too large`) before any file is stored. Accepted files are written to storage concurrently by `FILE_DATASOURCE_UPLOAD_MAX_WORKERS` workers, so raising the file count to hundreds of files does not slow the request down proportionally. + +The configured file count is advertised as `fileDatasourceMaxUploadCount` in `GET /v1/info`. The CodeMie UI reads it to size the upload area of the File data source form, so no UI configuration is needed; when the field is missing the UI falls back to `10`. +::: ### Cloud Storage - AWS S3