V2.31.0 updates

docs/contributing/release_process.md

@@ -58,11 +58,13 @@ describe: See the changes list at
 
 ## Documentation Update
 * Update the version number in the docusaurus.config.ts file (in the footer section).
+* Delete the i18n/en directory (needed because write-translations will not overwrite an existing file) and run `npm run write-translations` to export the new footer string.
 * Edit the documentation pages (deploy-install.md and deploy-update.md).
   * Search for \[erddap.war\] 
-  * Copy the existing information (slightly reformatted) to the list of previous installations 2.
+  * Copy the existing information (slightly reformatted) to the list of previous installations.
   * Change the current release information for erddap.war at \[erddap.war\]
-* Run the translations for the documentation site.
+* Run the translations for the documentation site. It is recommended to only translate pages that have changed since this step can be very slow.
+  * Make sure the footers are translated with the new version number.
 * Make a pull request and merge the changes.
 * Deploy the documentation site (see readme).
 

docs/server-admin/admin-tips/blocking-on-bad-requests.md

@@ -0,0 +1,96 @@
+# Blocking ERDDAP™ requests based on the content of the request, not based on the IP
+
+This content is based on a [message from Roy Mendelssohn to the ERDDAP™ users group](https://groups.google.com/g/erddap/c/XcvPkoGtchg).
+
+## The problem 
+
+If you are like us,  you are seeing a lot of bots making data requests to your ERDDAP™,  and the requests often seem like they are done by poorly coded scripts,  whether by LLMs (my guess) or by humans.  One thing about LLMs is they will do exactly what you prompt them to do,  so if you don’t tell them to have the script check return codes, and don’t  tell them what to do in different cases,  then usually the code will not do either of those things.  And if you tell it to keep on trying,  it will.  Which is what we are seeing at least.
+
+But the ones that are doing a number on our ERDDAP™ are like this:
+
+https://coastwatch.pfeg.noaa.gov/erddap/jplMURSST41.parquetWMeta
+
+These send ERDDAP™ into never-never land, due,  to the best that I can determine,  that this URL is not only requesting the entire dataset,  which is I don’t know how many terabytes,  but wants it converted to a parquet file.  Worse the requests are coming in from moving IPs,  so blocking IPs is worse than whack-a-mole,  you will never get them all blocked.
+
+So the question is how can you block based on the content of the request?  Before going further,  if you are experience crashes it may be from different causes then what we are seeing - I spent a lot of time going through logs and also I get notified when there is dangerous high memory use,  and noted when a crash followed that notification,  and also noticed what was the common denominator in these crashes,  which may not be the case for your server.  So research this before taking any action,  but this may  give you some idea how to block these requests. 
+
+Just so I am clear,  you can think of an ERDDAP™ request as:
+
+https://baseURL/erddap/method/datasetID.filetype?constraint
+
+In our case the common denominator of the crashes were griddap or tabletop requests with certain filetypes but no constraint.  So the question is how to block requests without a constraint?  Except it is not  that simple,  because there are any number of filetypes that don’t need a constraint and will be well-behaved, so we don’t want to block those.  After talking to Chris,  and no doubt I have left out something,  the filetypes that are well behaved without a constraint are:
+
+- .croissant
+- .iso19115_2
+- .iso19139_2007
+- .iso19115_3_2016
+- .ncCFHeader
+- .ncCFMAHeader
+- .das
+- .dds
+- .html
+- .graph
+- .subset
+- .ncHeader
+- .help
+- .fgdc
+- .iso19115
+- .ncoJsonHeader (this one will be in the upcoming new release).
+
+## Solution
+
+The solution "I found" works in Apache2,  I do not know nginx but I imagine there is something similar.  At first I tried mod_security,  but that got too complicated and didn’t work very well.  The solution is to use mod_rewrite.  Now I am anything but an expert on this, so while I defined what I was trying to accomplish,  the solution, as well as the explanation,  is due to Claude.ai.  ChatGPT will supply you with basically the same answer.
+
+Step 1.  Make certain that mod_rewrite is installed and enabled.  Since how to do this varies with OS,  this is something you can ask your favorite chatbot.
+
+Step 2.  In the appropriate file that configures ssl for apache2 (which again varies by OS), for example it might be something like /etc/apache2/sites-enabled/ssl.conf,  add the following under the appropriate VirtualHost definition (note if you copy this there are only 4 lines,  the third line may be wrapped,  unwrap it)
+
+```
+RewriteEngine On
+RewriteCond %{QUERY_STRING} ^$
+RewriteCond %{REQUEST_URI} ^/erddap/(griddap|tabledap)/[^/?]+\.(?!(?:croissant|iso19115_2|iso19139_2007|iso19115_3_2016|ncCFHeader|ncCFMAHeader|das|dds|html|graph|subset|ncHeader|help|fgdc|iso19115|ncoJsonHeader)$)[A-Za-z0-9_]+$
+RewriteRule ^ - [R=429,L]
+```
+
+Step 3.  Check that the configuration is valid:  `sudo apache2ctl configtest`
+
+Step 4.  Restart apache2:  `sudo systemctl restart apache2`
+
+Step 5.  Check your logs that nothing is being blocked that should not be,  and that appropriate requests without a constraint return a 429 without ever hitting your tomcat
+
+## Explanation
+
+Why does this work and what does this do - here is the explanation from Claude.ai:
+
+Line 1 — `RewriteEngine On`
+Turns on mod_rewrite processing for this scope. Without it, the RewriteCond/RewriteRule directives below are simply ignored.
+
+Line 2 — `RewriteCond %{QUERY_STRING} ^$`
+A condition that must be true before the rule below applies. `%{QUERY_STRING}` is everything after the ? in the request URL. ^$ is a regex meaning "start of string immediately followed by end of string" — i.e., an empty string. So this condition is true only when there's no query string at all — no subsetting/constraint expression on the request.
+
+Line 3 — `RewriteCond %{REQUEST_URI} ^/erddap/(griddap|tabledap)/[^/?]+\.(?!(?:...)$)[A-Za-z0-9_]+$`
+A second condition, checked against `%{REQUEST_URI}` — the literal request path as the client sent it, always the full path regardless of where in the config this rule lives (deliberately chosen over letting the RewriteRule pattern itself do the matching, because pattern-matching inside a `<Location>` block can behave ambiguously — see note below). Breaking down the regex:
+
+`^/erddap/` — must start with /erddap/
+(griddap|tabledap)/ — followed by one of the two ERDDAP™ access methods
+
+`[^/?]+` — the datasetID: one or more characters that aren't / or ?
+
+`\.` — a literal dot
+
+`(?!(?:croissant|iso19115_2|...|ncoJsonHeader)$)` — a negative lookahead: "as long as what follows is not one of these exact fileType names all the way to the end of the string." These are the fileTypes ERDDAP™ can legitimately serve with no constraint (metadata, structure, form pages, etc.) — the lookahead is what excludes them from being blocked.
+
+`[A-Za-z0-9_]+$` — the actual fileType extension (letters, digits, underscore), required to run to the end of the string.
+So this condition is true only when the path is a griddap/tabledap request for some fileType that isn't on the safe-without-constraint list.
+
+Line 4 — `RewriteRule ^ - [R=429,L]`
+The rule itself. Because both conditions above must already be true for Apache to even evaluate this line, the pattern here doesn't need to check anything else — ^ just matches "start of the string," which is always true. - means "don't rewrite the URL to anything different" (we're not redirecting anywhere, just short-circuiting the request). The flags:
+
+`R=429` — respond with an HTTP redirect-class action carrying status code 429 ("Too Many Requests") instead of serving the request.
+L — "Last": stop processing any further rewrite rules once this one fires.
+Put together: if the query string is empty, AND the request is for a griddap/tabledap fileType that isn't on the safe-unconstrained list, immediately return 429 — without ever contacting the ERDDAP/Tomcat backend.
+
+Why `%{REQUEST_URI}` instead of letting the RewriteRule pattern match the path directly (worth including as a note for colleagues, since it's the non-obvious part): inside a `<Location>` block, what a bare RewriteRule pattern actually gets matched against can behave inconsistently depending on Apache version and context. Explicitly pulling the full path via RewriteCond `%{REQUEST_URI}` sidesteps that ambiguity entirely — it's always the literal, complete request path, so the regex behaves exactly as written regardless of where the rule is nested.
+
+
+I have been using this for several days now and it appears to work very well,  it is blocking what I am trying to block and not blocking what I don’t want to block.  And our ERDDAP™ has become much more stable.

docs/server-admin/admin-tips/deploy-kubernetes.md

@@ -1,6 +1,6 @@
-# How to Deploy ERDDAP on Kubernetes
+# How to Deploy ERDDAP™ on Kubernetes
 
-Deploying ERDDAP on Kubernetes provides a scalable, resilient environment for your data server. This guide covers the essential components required to host ERDDAP using standard Kubernetes manifests, including managing persistent storage, deploying the application, configuring networking, and generating new dataset XMLs directly from within the cluster.
+Deploying ERDDAP™ on Kubernetes provides a scalable, resilient environment for your data server. This guide covers the essential components required to host ERDDAP™ using standard Kubernetes manifests, including managing persistent storage, deploying the application, configuring networking, and generating new dataset XMLs directly from within the cluster.
 
 ## Prerequisites
 Before you begin, ensure you have:
@@ -11,7 +11,7 @@ Before you begin, ensure you have:
 -----
 
 ## 1. Persistent Storage (PVC)
-ERDDAP requires persistent storage to maintain cache files, logs, and state across pod restarts. Using a `PersistentVolumeClaim` (PVC) ensures that your `bigParentDirectory` (where ERDDAP stores its generated data) is not lost if a pod goes down. This volume can also be linked to your data storage location where the raw data files will live.
+ERDDAP™ requires persistent storage to maintain cache files, logs, and state across pod restarts. Using a `PersistentVolumeClaim` (PVC) ensures that your `bigParentDirectory` (where ERDDAP™ stores its generated data) is not lost if a pod goes down. This volume can also be linked to your data storage location where the raw data files will live.
 
 Create a file named `pvc.yaml` like so:
 ```yaml
@@ -34,14 +34,14 @@ spec:
 
 ----
 
-## 2.  The ERDDAP Deployment
-The Deployment manifest manages the ERDDAP pod itself. We recommend using the official erddap/erddap Docker image with long term support.
+## 2.  The ERDDAP™ Deployment
+The Deployment manifest manages the ERDDAP™ pod itself. We recommend using the official erddap/erddap Docker image with long term support.
 
 :::info
 As of May 2026, [v2.30.0](https://github.com/erddap/erddap/pkgs/container/erddap/779906687?tag=v2.30.0) was the latest version. It is wise to re-deploy occasionally to keep up with security vulnerabilities.
 :::
 
-In this configuration, we inject key environment variables to handle timezone settings, ensure Tomcat has the correct read/write permissions for the storage volume, and tell ERDDAP how to properly route URLs when sitting behind a Kubernetes Ingress. We also mount the PVC to `/erddapData` (the default `bigParentDirectory`) to inject the datasets.xml and setup.xml into `/usr/local/tomcat/content/erddap`.
+In this configuration, we inject key environment variables to handle timezone settings, ensure Tomcat has the correct read/write permissions for the storage volume, and tell ERDDAP™ how to properly route URLs when sitting behind a Kubernetes Ingress. We also mount the PVC to `/erddapData` (the default `bigParentDirectory`) to inject the datasets.xml and setup.xml into `/usr/local/tomcat/content/erddap`.
 
 Create a file named `deployment.yaml`:
 
@@ -121,23 +121,23 @@ spec:
         persistentVolumeClaim:
           claimName: erddap-pvc
 ```
-- **TZ**: Sets the timezone for the Tomcat server and ERDDAP logs.
+- **TZ**: Sets the timezone for the Tomcat server and ERDDAP™ logs.
 
-- **TOMCAT_USER_ID & TOMCAT_GROUP_ID**: By default, the ERDDAP container runs Tomcat as a specific user. If the persistent volume mounted to /erddapData is owned by a different user/group ID on your host storage system, ERDDAP will crash due to permission denied errors. Setting these variables forces Tomcat to run with the matching IDs.
+- **TOMCAT_USER_ID & TOMCAT_GROUP_ID**: By default, the ERDDAP™ container runs Tomcat as a specific user. If the persistent volume mounted to /erddapData is owned by a different user/group ID on your host storage system, ERDDAP™ will crash due to permission denied errors. Setting these variables forces Tomcat to run with the matching IDs.
 
     :::tip
     Find your user UID on the server where the NFS mount is like this: `id -u <your-user_name>`. This will return the numeric value you need. 
     :::
 
-- **ERDDAP_baseUrl & ERDDAP_baseHttpsUrl**: When ERDDAP runs in Kubernetes behind a Service and an Ingress, Tomcat thinks it is serving traffic on localhost:8080. These variables override ERDDAP's internal URL generation so that links (like your custom logo or dataset links) correctly resolve to your public-facing domain name.
+- **ERDDAP_baseUrl & ERDDAP_baseHttpsUrl**: When ERDDAP™ runs in Kubernetes behind a Service and an Ingress, Tomcat thinks it is serving traffic on localhost:8080. These variables override ERDDAP's internal URL generation so that links (like your custom logo or dataset links) correctly resolve to your public-facing domain name.
 
 :::note 
 If you are running separate Production and QA environments, be cautious about sharing a single PVC. Modifying or deleting cached data in one environment will immediately affect the other. We manage this using deployment overlays for QA and Production and adding subfolders for each overlay. This allows us to test on QA with a QA datasets.XML before touching the production deployment.
 :::
 ---
 
 ## 3. Networking: Service and Ingress
-To expose your ERDDAP pod to the web, you need a Service to route internal cluster traffic, and an Ingress to bind it to a public DNS name.
+To expose your ERDDAP™ pod to the web, you need a Service to route internal cluster traffic, and an Ingress to bind it to a public DNS name.
 
 Create a file named `service.yaml`:
 ```yaml
@@ -211,7 +211,7 @@ erddap/
         └── kustomization.yaml
 ```
 
-Create the `kustomization.yaml` file to collect all the resources and map your custom setup and datasets XML files. These will get passed into your ERDDAP Docker image when deployed so you can style your ERDDAP page and add datasets from your GitHub repository while letting `kustomize` map them to your deployment.
+Create the `kustomization.yaml` file to collect all the resources and map your custom setup and datasets XML files. These will get passed into your ERDDAP™ Docker image when deployed so you can style your ERDDAP™ page and add datasets from your GitHub repository while letting `kustomize` map them to your deployment.
 
 #### Base (`base/kustomization.yaml`)
 The base kustomization simply bundles your core resources shared across the overlays. We keep the production `datasets.xml` and `setup.xml` in base and only update these after testing on QA.
@@ -296,7 +296,7 @@ Check the status of your deployment:
 
 ---
 ## 5. Dataset XML Generation in Kubernetes
- Adding new datasets to ERDDAP requires generating an XML block for the `datasets.xml` file. ERDDAP ships with two interactive utilities, `GenerateDatasetsXml.sh` and `DasDds.sh`, which you can run directly inside your active pod.
+ Adding new datasets to ERDDAP™ requires generating an XML block for the `datasets.xml` file. ERDDAP™ ships with two interactive utilities, `GenerateDatasetsXml.sh` and `DasDds.sh`, which you can run directly inside your active pod.
 
  ### Step 1: Generate the XML
    - Find the pod ID: `kubectl get pods`
@@ -305,14 +305,14 @@ Check the status of your deployment:
    - Copy the resulting XML output to your `datasets.xml` in your repository and to the `datasets.xml` in your volume mount. After we validate the XML, we can redeploy and the config will map the new `datasets.xml` file to your deployment.
 
 ### Step 2: Validate the new Dataset XML
-Before restarting the entire deployment, test that ERDDAP can successfully read your new XML configuration using the `DasDds.sh` script.
+Before restarting the entire deployment, test that ERDDAP™ can successfully read your new XML configuration using the `DasDds.sh` script.
   - Ensure your updated `datasets.xml` is saved to your mounted config directory.
   - Run the validation script: `kubectl exec -it <erddap-pod-id> -- bash -c "cd /usr/local/tomcat/webapps/erddap/WEB-INF && ./DasDds.sh"`
   - Enter the `datasetID` you just created in the last step.
   - If the XML is valid, the script will print the `.das` and `.dds` structure to your terminal. If there are errors, use the output to troubleshoot and correct your `datasets.xml`. Repeat steps 1 and 2 until there are no more errors.
 
   ### Step 3: Apply the Changes
-  Once validated, restart your deployment so ERDDAP can ingest the new configurations: 
+  Once validated, restart your deployment so ERDDAP™ can ingest the new configurations: 
   `kubectl rollout restart deployment/erddap-deployment`
 
   ---
@@ -330,4 +330,4 @@ Before restarting the entire deployment, test that ERDDAP can successfully read
 ---
 
 ### Notes
-This is only one way of deploying ERDDAP using Kubernetes, and is the way we have deployed the [ERDDAP](https://erddap.riddc.brown.edu/erddap/index.html) associated with the [Rhode Island Data Discovery Center](https://riddc.brown.edu/). We use the manifest approach with `kustomize` so it's easier to understand all the connections and we still get the benefits of using overlays and testing on QA. Helm Charts is another viable approach, but would use a completely different configuration approach. 
+This is only one way of deploying ERDDAP™ using Kubernetes, and is the way we have deployed the [ERDDAP](https://erddap.riddc.brown.edu/erddap/index.html) associated with the [Rhode Island Data Discovery Center](https://riddc.brown.edu/). We use the manifest approach with `kustomize` so it's easier to understand all the connections and we still get the benefits of using overlays and testing on QA. Helm Charts is another viable approach, but would use a completely different configuration approach. 

docs/server-admin/admin-tips/duckdb.md

@@ -1,4 +1,4 @@
-This content is based on a [message from Roy Mendelssohn to the ERDDAP users group](https://groups.google.com/g/erddap/c/6Hl024ZGkes/m/DS5WzsydAQAJ).
+This content is based on a [message from Roy Mendelssohn to the ERDDAP™ users group](https://groups.google.com/g/erddap/c/6Hl024ZGkes/m/DS5WzsydAQAJ).
 
 ERDDAP™ tries to be agnostic about what data formats people use for their data, instead trying to work with the data formats of most use to the communities we mainly serve. As more and more work is in the cloud, and there are a plethora of data formats that people use in the cloud, it would be nice if ERDDAP™ could support a lot of these formats. Alas, ERDDAP™ development and maintenance is already understaffed, and what would be desirable is to make use of the work of others to achieve this goal, without having to modify ERDDAP™.
 

docs/server-admin/admin-tips/optimizing_netcdf.md

@@ -1,4 +1,4 @@
-This content is based on a [message from Roy Mendelssohn to the ERDDAP users group](https://groups.google.com/g/erddap/c/JWoS_y3cygg/m/zCpcNTxNAAAJ).
+This content is based on a [message from Roy Mendelssohn to the ERDDAP™ users group](https://groups.google.com/g/erddap/c/JWoS_y3cygg/m/zCpcNTxNAAAJ).
 
 1. Optimizing netcdf files for the cloud
 ————————————————-