Document GCS, Azure, IBM, and Oracle patch storage backends

docs/.custom_wordlist.txt

@@ -19,6 +19,7 @@ Canonical('s)?
 Charmcraft
 charmstore
 cjk
+COS
 crypto
 cryptographically
 CVEs
@@ -29,20 +30,24 @@ Downloader
 dvipng
 enablement
 Enablement
+Entra
 eTags
 filepath
 firewalled
 fonts
 freefont
 Furo
 gcp
+GCE
+GCS
 github
 GitHub
 gkeop
 glibc
 GPG
 gyre
 hostnames
+HMAC
 hotfix(es)?
 html
 https
@@ -76,6 +81,7 @@ MyST
 npm
 Numpy
 oem
+OCI
 Open Graph
 openapi
 otf
@@ -94,6 +100,7 @@ pymarkdown
 pymarkdownlnt
 QEMU
 Read the Docs
+rclone
 readthedocs
 redeclare
 rediraffe
@@ -106,6 +113,7 @@ Rockcraft
 rollout
 rollouts
 rst
+rsync
 rtd
 SCons
 sequenceDiagram

docs/Makefile

@@ -120,6 +120,7 @@ lint-md: pymarkdownlnt-install
 			--recurse \
 			--exclude=$(DEV_DIR)/** \
 			--exclude=$(DOCS_VENVDIR)/** \
+			--exclude=$(DOCS_BUILDDIR)/** \
 			$(DOCS_SOURCEDIR); \
 	status=$$?; \
 	if [ $$status -eq 1 ]; then \

docs/client/how-to-guides/installation/disable-livepatch-during-startup.md

@@ -52,8 +52,8 @@ The process of reporting, blocklisting, and releasing a replacement Livepatch ta
 1. If the current Livepatch mode in the `/var/local/canonical_livepatch_mode` file is `stop` and the Livepatch daemon is not running, no configuration changes can be made.
 2. Set the Livepatch mode to `no-apply` using `sudo bash -c 'echo -n no-apply > /var/local/canonical_livepatch_mode'`.
 3. Run `sudo snap restart canonical-livepatch`. This enables the Livepatch daemon. The daemon refreshes patch information but never applies the patch to the kernel in this mode.
-4. Once the daemon is enabled, set the `cutoff-date` configuration option to a date before the faulty patch was released. For example, `sudo canonical-livepatch config cutoff-date=2024-01-01T01:00:00Z`. The release date for the problematic LSN can be found on the [Ubuntu Security Notices page](https://ubuntu.com/security/notices?details=lsn).
+4. Once the daemon is enabled, set the `cutoff-date` configuration option to a date before the faulty patch was released. For example, `sudo canonical-livepatch config cutoff-date=2024-01-01T01:00:00Z`. The release date for the problematic LSN can be found on the [Ubuntu Security Notices page](https://ubuntu.com/security/notices).
 5. Set the Livepatch mode to `normal` or delete the Livepatch mode file.
 6. Run `sudo snap restart canonical-livepatch`. This enables the full operation of the Livepatch daemon. The daemon retrieves the most recent stable patch version due to the `cutoff-date` config option and applies it to the running kernel.
 
-This process improves kernel security until a new and stable Livepatch is released for the kernel, at which point the `cutoff-date` configuration option can be disabled or changed to retrieve the most recent Livepatch.
+This process improves kernel security until a new and stable Livepatch is released for the kernel, at which point the `cutoff-date` configuration option can be disabled or changed to retrieve the most recent Livepatch.

docs/client/reference/patches/patch-security.md

@@ -31,7 +31,7 @@ Patch files are distributed as tarballs containing metadata and a Linux kernel m
 All kernel modules are signed by Canonical to verify their authenticity, using asymmetric encryption:
 
 - Signature algorithm: SHA-512 with RSA
-- Canonical's [public key](https://git.launchpad.net/~ubuntu-kernel/ubuntu/+source/linux/+git/jammy/plain/debian/certs/canonical-livepatch-all.pem)
+- Canonical's public key
 
 Kernel modules are authenticated before they are installed. This ensures the patch was produced by Canonical and protects the client from installing maliciously crafted patches. The public keys used for signature verification are embedded in the Livepatch Client and do not require external access.
 
@@ -57,4 +57,4 @@ The Livepatch Client can also be configured to enforce TLS for all patch downloa
 
 Go packages used for TLS configuration:
 
-- [`crypto/tls`](https://pkg.go.dev/crypto/tls)
+- [`crypto/tls`](https://pkg.go.dev/crypto/tls)

docs/server/how-to-guides/security/harden-your-deployment.md

@@ -180,4 +180,4 @@ To reset the charm to its default configuration:
 juju config canonical-livepatch-server-k8s --reset <key>
 ```
 
-To fully reset, remove the application and redeploy.
+To fully reset, remove the application and redeploy.

docs/server/reference/patch-storage/index.md

@@ -13,12 +13,16 @@ Livepatch Server supports several different drivers for storing patch files down
 
 1. Local filesystem
 2. Swift
-3. S3 (and compatible implementations, e.g. minio)
-4. Postgresql
+3. S3 (and compatible implementations, e.g. MinIO)
+4. PostgreSQL
+5. Google Cloud Storage (GCS)
+6. Azure Blob Storage
+7. IBM Cloud Object Storage
+8. Oracle Cloud Infrastructure (OCI) Object Storage
 
 The filesystem patch store is easiest to deploy and suits most configurations. However, if there is a need to scale out the Livepatch Server such as have multiple Livepatch Servers running to handle the load, the filesystem patch store should not be used.
 
-In case there is a need to scale out Livepatch on-prem, use the s3, postgresql or swift patch stores. Any patch store should have enough space for storing live kernel patches - currently at least 45GB for all patches, see [this guide](/server/reference/patch-management/patch-sync-filters.md) to filter patches sent to your on-prem instance to specific kernel variants/architectures and lower this requirement.
+In case there is a need to scale out Livepatch on-prem, use one of the object storage backends (s3, gcs, azure, ibm, oracle), postgresql, or swift patch stores. Any patch store should have enough space for storing live kernel patches - currently at least 45GB for all patches, see [this guide](/server/reference/patch-management/patch-sync-filters.md) to filter patches sent to your on-prem instance to specific kernel variants/architectures and lower this requirement.
 
 See the [patch storage](/server/reference/platform/configuration.md) config for all available parameters.
 
@@ -29,4 +33,9 @@ See the [patch storage](/server/reference/platform/configuration.md) config for
 :hidden:
 
 Use S3 for patch storage <use-s3-for-patch-storage.md>
+Use GCS for patch storage <use-gcs-for-patch-storage.md>
+Use Azure for patch storage <use-azure-for-patch-storage.md>
+Use IBM COS for patch storage <use-ibm-for-patch-storage.md>
+Use OCI Object Storage for patch storage <use-oracle-for-patch-storage.md>
+Migrate patches between storage backends <migrating-patch-storage.md>
 ```

docs/server/reference/patch-storage/migrating-patch-storage.md

@@ -0,0 +1,42 @@
+---
+myst:
+  html_meta:
+    description: "Migrate patches between storage backends - learn about this topic in Livepatch on-prem."
+---
+
+(server-reference-migrating-patch-storage)=
+
+# Migrating patches between storage backends
+
+This guide covers migrating between the file-based patch storage backends (filesystem, S3, GCS, Azure, IBM, Oracle, Swift). Each of these stores patches as a flat set of files, each keyed by its filename, so the same set of patch files works unmodified on any of them: migrating between backends is simply a matter of copying the patch files across, then updating [`patch-storage.type`](/server/reference/platform/configuration.md) and its associated options to point at the new backend.
+
+The Postgres backend (`patch-storage.postgres-connection-string`) is not covered here, since it stores patches as rows in a database rather than as files, and cannot be migrated with `rsync` in the same way.
+
+The recommended tool for copying patches is `rsync`. If you need checksum-based verification, add `--checksum` (note that this can be slower). Disable patch synchronisation in the config first (so no new patches arrive, and avoid running any manual syncs), then run `rsync` once to copy the existing patches across.
+
+## Filesystem to filesystem
+
+If either the source or destination (or both) is the filesystem backend, `rsync` can be used directly, locally or over SSH:
+
+```bash
+rsync -avz --progress /var/snap/canonical-livepatch-server/common/patches/ user@remote-host:/path/to/new/patches/
+```
+
+## Migrating to or from object storage
+
+For the object storage backends (S3, GCS, Azure, IBM, Oracle, Swift), mount the bucket or container as a local directory with [rclone](https://rclone.org/), which supports all of these backends (including any S3-compatible endpoint, such as IBM COS or MinIO), then `rsync` into or out of the mount as if it were a normal directory.
+
+1. Disable [`patch-sync`](/server/reference/patch-management/patch-sync-filters.md) in the config so no new patches are synced to the on-prem server, and avoid running any manual syncs, while the migration is in progress.
+2. Install rclone and configure a remote for the bucket/container, following [rclone's documentation](https://rclone.org/docs/) for the relevant backend (`s3`, `google cloud storage`, `azureblob`, `swift`, or `oracle-object-storage`).
+3. Mount the remote:
+   ```bash
+   mkdir -p /mnt/livepatch-patches
+   rclone mount <remote-name>:<bucket-or-container> /mnt/livepatch-patches --daemon
+   ```
+4. Run `rsync` to copy the existing patches across:
+   ```bash
+   rsync -avz --progress /var/snap/canonical-livepatch-server/common/patches/ /mnt/livepatch-patches/
+   ```
+5. Verify the file counts match between source and destination, then update the [patch storage config](/server/reference/platform/configuration.md) to the new backend and re-enable patch-sync.
+
+The same approach applies in reverse (object storage to filesystem, or between two object storage backends) by mounting both sides and running `rsync` between the two mount points.

docs/server/reference/patch-storage/use-azure-for-patch-storage.md

@@ -0,0 +1,19 @@
+---
+myst:
+  html_meta:
+    description: "Use Azure for patch storage - learn about this topic in Livepatch on-prem."
+---
+
+(server-reference-livepatch-on-prem-with-azure-patch-storage)=
+
+# Livepatch on-prem with Azure Blob Storage patch storage
+
+To configure this, follow these steps:
+
+- Create a Blob Storage container in the preferred region (best if the region is the same as the deployment's). Care needs to be taken to make the container not publicly writable, as this would pose a significant security risk.
+- Choose an authentication method: a storage account name/key pair, a connection string, an Entra ID (Azure AD) service principal, or a managed identity bound to the VM.
+- Configure the relevant Azure [config options](/server/reference/platform/configuration.md).
+
+Once this is configured, Livepatch will store and retrieve patch files from the Azure Blob Storage container.
+
+`patch-storage.azure-account-name` is not required when `patch-storage.azure-connection-string` is set, since the connection string already carries the account name and key. When account key, connection string, and service principal credentials are all omitted, a managed identity bound to the VM is used instead (the system-assigned identity by default, or a user-assigned identity selected via `patch-storage.azure-managed-identity-client-id`).

docs/server/reference/patch-storage/use-gcs-for-patch-storage.md

@@ -0,0 +1,19 @@
+---
+myst:
+  html_meta:
+    description: "Use GCS for patch storage - learn about this topic in Livepatch on-prem."
+---
+
+(server-reference-livepatch-on-prem-with-gcs-patch-storage)=
+
+# Livepatch on-prem with Google Cloud Storage patch storage
+
+To configure this, follow these steps:
+
+- Create a GCS bucket in the preferred region (best if the region is the same as the deployment's). Care needs to be taken to make the bucket not publicly writable, as this would pose a significant security risk.
+- Create a service account with permissions to read and write objects in that bucket, or rely on the credentials already bound to the VM (e.g. the GCE metadata service, or workload identity) if it is granted the required permissions.
+- Configure the relevant GCS [config options](/server/reference/platform/configuration.md).
+
+Once this is configured, Livepatch will store and retrieve patch files from the GCS bucket.
+
+`patch-storage.gcs-credentials-file` and `patch-storage.gcs-credentials-json` are both optional; when both are omitted, Application Default Credentials are used instead. `patch-storage.gcs-impersonate-service-account` is also optional; when set, the resolved credentials are used to impersonate that service account instead of being used directly.

docs/server/reference/patch-storage/use-ibm-for-patch-storage.md

@@ -0,0 +1,19 @@
+---
+myst:
+  html_meta:
+    description: "Use IBM Cloud Object Storage for patch storage - learn about this topic in Livepatch on-prem."
+---
+
+(server-reference-livepatch-on-prem-with-ibm-patch-storage)=
+
+# Livepatch on-prem with IBM Cloud Object Storage patch storage
+
+To configure this, follow these steps:
+
+- Create an IBM Cloud Object Storage (COS) bucket in the preferred region (best if the region is the same as the deployment's). Care needs to be taken to make the bucket not publicly writable, as this would pose a significant security risk.
+- Choose an authentication method: an IAM API key with a service instance ID, or the ambient VPC Instance Metadata Service if the VM is granted the required trusted profile.
+- Configure the relevant IBM [config options](/server/reference/platform/configuration.md).
+
+Once this is configured, Livepatch will store and retrieve patch files from the COS bucket.
+
+Credentials are resolved in the following order of preference: an IAM API key (`patch-storage.ibm-api-key`/`patch-storage.ibm-service-instance-id`), then, if not configured, the ambient VPC Instance Metadata Service (optionally selecting a trusted profile via `patch-storage.ibm-trusted-profile-id`).

docs/server/reference/patch-storage/use-oracle-for-patch-storage.md

@@ -0,0 +1,19 @@
+---
+myst:
+  html_meta:
+    description: "Use OCI Object Storage for patch storage - learn about this topic in Livepatch on-prem."
+---
+
+(server-reference-livepatch-on-prem-with-oracle-patch-storage)=
+
+# Livepatch on-prem with Oracle Cloud Infrastructure Object Storage patch storage
+
+To configure this, follow these steps:
+
+- Create an Object Storage bucket in the preferred region (best if the region is the same as the deployment's). Care needs to be taken to make the bucket not publicly writable, as this would pose a significant security risk.
+- Choose an authentication method: an OCI config file/profile, or instance principal authentication if the compute instance is granted the required IAM policies.
+- Configure the relevant OCI [config options](/server/reference/platform/configuration.md).
+
+Once this is configured, Livepatch will store and retrieve patch files from the OCI Object Storage bucket.
+
+`patch-storage.oracle-config-file` (and `patch-storage.oracle-profile`) are optional; when omitted, instance principal authentication is used instead, bound to the OCI compute instance the server runs on.

docs/server/reference/platform/configuration.md

@@ -143,7 +143,7 @@ The following values configure how the server manages patch storage. See the [pa
 
 | Name | Description | Values |
 | ---- | ----------- | ------ |
-| `patch-storage.type` | Storage backend type for on-premises patch synchronisation. | `oneof: filesystem, swift, postgres, s3` |
+| `patch-storage.type` | Storage backend type for on-premises patch synchronisation. | `oneof: filesystem, swift, postgres, s3, gcs, azure, ibm, oracle` |
 | `patch-storage.filesystem-path` | Directory path for filesystem storage. | `string` |
 | `patch-storage.swift-username` | Swift account username. | `string` |
 | `patch-storage.swift-api-key` | Swift API key. | `string` |
@@ -154,11 +154,36 @@ The following values configure how the server manages patch storage. See the [pa
 | `patch-storage.swift-region` | Swift region. | `string` |
 | `patch-storage.postgres-connection-string` | PostgreSQL connection string. Can be left blank in charmed deployments to use Juju relations. | `string` |
 | `patch-storage.s3-bucket` | S3 bucket for storing patches. | `string` |
-| `patch-storage.s3-endpoint` | S3 endpoint URL. | `string` |
+| `patch-storage.s3-endpoint` | S3 endpoint URL (only needed for non-AWS S3-compatible services, e.g. MinIO). | `string` |
 | `patch-storage.s3-region` | AWS region for S3. | `string` |
 | `patch-storage.s3-secure` | Whether to use secure transfers. | `bool` |
-| `patch-storage.s3-access-key` | AWS access key. | `string` |
-| `patch-storage.s3-secret-key` | AWS secret key. | `string` |
+| `patch-storage.s3-use-path-style` | Whether to use path-style bucket addressing. Defaults to `false` (virtual-hosted-style); typically needs to be `true` for non-AWS S3-compatible services (e.g. MinIO, Ceph). | `bool` |
+| `patch-storage.s3-access-key` | AWS access key (optional; defaults to the VM's ambient credentials, e.g. an EC2 instance role). | `string` |
+| `patch-storage.s3-secret-key` | AWS secret key (optional; see `patch-storage.s3-access-key`). | `string` |
+| `patch-storage.s3-assume-role-arn` | IAM role ARN to assume via STS, using the resolved credentials above (optional). | `string` |
+| `patch-storage.gcs-bucket` | Google Cloud Storage bucket for storing patches. | `string` |
+| `patch-storage.gcs-credentials-file` | Path to a service account JSON key file (optional; defaults to Application Default Credentials). | `string` |
+| `patch-storage.gcs-credentials-json` | Inline service account JSON key (optional; see `patch-storage.gcs-credentials-file`). | `string` |
+| `patch-storage.gcs-impersonate-service-account` | Email of a service account to impersonate, using the resolved credentials above (optional). | `string` |
+| `patch-storage.azure-container` | Azure Blob Storage container for storing patches. | `string` |
+| `patch-storage.azure-account-name` | Azure storage account name (not required when `patch-storage.azure-connection-string` is set, which already carries the account name). | `string` |
+| `patch-storage.azure-account-key` | Azure storage account key (optional; defaults to a managed identity bound to the VM). | `string` |
+| `patch-storage.azure-connection-string` | Azure storage connection string (optional alternative to account name/key). | `string` |
+| `patch-storage.azure-tenant-id` | Entra ID tenant ID, for explicit service principal auth (optional; requires `patch-storage.azure-client-id` and `patch-storage.azure-client-secret`). | `string` |
+| `patch-storage.azure-client-id` | Entra ID application (client) ID, for explicit service principal auth (optional). | `string` |
+| `patch-storage.azure-client-secret` | Entra ID client secret, for explicit service principal auth (optional). | `string` |
+| `patch-storage.azure-managed-identity-client-id` | Client ID of a user-assigned managed identity to use instead of the VM's system-assigned identity (optional). | `string` |
+| `patch-storage.ibm-bucket` | IBM Cloud Object Storage bucket for storing patches. | `string` |
+| `patch-storage.ibm-endpoint` | IBM Cloud Object Storage endpoint. | `string` |
+| `patch-storage.ibm-region` | IBM Cloud Object Storage region. | `string` |
+| `patch-storage.ibm-api-key` | IBM Cloud IAM API key (optional; defaults to the ambient VPC Instance Metadata Service). | `string` |
+| `patch-storage.ibm-service-instance-id` | IBM COS resource instance ID, required when using `patch-storage.ibm-api-key`. | `string` |
+| `patch-storage.ibm-trusted-profile-id` | Trusted profile ID to assume via the ambient VPC Instance Metadata Service (optional; defaults to the instance's default trusted profile). | `string` |
+| `patch-storage.oracle-bucket` | OCI Object Storage bucket for storing patches. | `string` |
+| `patch-storage.oracle-namespace` | OCI Object Storage namespace. | `string` |
+| `patch-storage.oracle-region` | OCI region. | `string` |
+| `patch-storage.oracle-config-file` | Path to an OCI config file (optional; defaults to instance principal authentication). | `string` |
+| `patch-storage.oracle-profile` | Profile to use within `patch-storage.oracle-config-file` (optional). | `string` |
 
 ### Patch cache
 
@@ -232,4 +257,4 @@ The cloud delay feature is deprecated. Use the `patch-delay` and `cutoff-date` c
 | Name | Description | Values |
 | ---- | ----------- | ------ |
 | `cloud-delay.enabled` | Whether to delay patch releases to clients based on their cloud, region, or availability zone. | `bool` |
-| `cloud-delay.default-delay-hours` | Default delay in hours for clouds, regions, or availability zones without a specified delay. | `int` |
+| `cloud-delay.default-delay-hours` | Default delay in hours for clouds, regions, or availability zones without a specified delay. | `int` |