diff --git a/README.md b/README.md index 0721020..ce7c119 100644 --- a/README.md +++ b/README.md @@ -1,163 +1,333 @@ # ravendb-embedded -`ravendb-embedded` starts a real RavenDB server as a child process managed by your Python program. -You `pip install` it, start the server, and talk to it with the normal `ravendb` client. There is -no separate server to install, configure, or keep running: the server process's lifetime follows -your Python process. +`ravendb-embedded` starts a real RavenDB server as a child process of your Python application. +The server starts with your code, stops with it, and is accessed through the standard `ravendb` +client. -Reach for it when you want: +Use it for local development, integration tests, or applications that should manage their own +RavenDB process. -- **Local development** without setting up a standalone RavenDB. -- **Integration tests** against a real server instead of a mock (see also `ravendb-test-driver`). -- **Small or self-contained apps** that ship the database alongside the code. +## Install + +```bash +pip install ravendb-embedded +``` + +Python 3.10+ is required. + +## Quick start + +The default mode uses the RavenDB binaries included in the wheel and a compatible system .NET +runtime: ```python -from ravendb_embedded import EmbeddedServer +from ravendb_embedded import EmbeddedServer, ServerOptions + +options = ServerOptions() with EmbeddedServer() as server: - server.start_server() - with server.get_document_store("Embedded") as store: + server.start_server(options) + with server.get_document_store("MyDatabase") as store: with store.open_session() as session: session.store({"name": "Ayende"}, "people/1") session.save_changes() ``` -## Installation +## Choose how RavenDB runs -```bash -pip install ravendb-embedded -``` +| Mode | System .NET required? | Server lifecycle | Best for | +|------|-----------------------|------------------|----------| +| [Bundled server](#bundled-server-default) | Yes | Managed by the package | The simplest local setup | +| [On-demand self-contained server](#on-demand-self-contained-server) | No | Downloaded, cached, and managed by the package | Portable developer and CI environments | +| [Self-contained server you provide](#self-contained-server-you-provide) | No | Managed by the package | Offline or centrally controlled server artifacts | -The install includes a copy of the RavenDB server binaries. Python 3.10+ is required. +Only the bundled-server mode requires .NET to be installed on the machine running Python. -## The .NET requirement (read this) +### Bundled server (default) -The bundled server is a .NET application, so a matching **.NET runtime** must be on the machine. -The required version tracks the bundled server: +The wheel includes a framework-dependent RavenDB server. Its runtime requirement follows the +RavenDB version: | `ravendb-embedded` version | Required runtime | |----------------------------|------------------| | 7.2.x | .NET 10 | | 7.1.x | .NET 8 | -Check what is installed with `dotnet --list-runtimes` (look for `Microsoft.NETCore.App`). Because -the requirement follows the bundled server, it can change on a minor upgrade, so re-check it when -you bump versions. - -If the machine cannot or should not have .NET, use the self-contained path under -[Run without installing .NET](#run-without-installing-net) below. +Run `dotnet --list-runtimes` and look for `Microsoft.NETCore.App`. Re-check this requirement when +upgrading to a new RavenDB minor version. -## Usage +Runnable walkthrough: [Lab 01 — embedded zero-config](labs/01-embedded-zero-config.md). -The three sections below are the ways people actually use this package. Pick the one that matches -your environment; each links to a runnable walkthrough in `labs/`. +### On-demand self-contained server -### Run it (the default, needs .NET) - -Start the server and get a document store. This is the zero-config path and uses the system .NET -described above. Pass a `ServerOptions` when you want to control where data lives, the bind URL, -and so on. +For the same managed experience without installing .NET, let the package download a +self-contained RavenDB build: ```python from ravendb_embedded import EmbeddedServer, ServerOptions options = ServerOptions() -options.data_directory = "MYPATH/RavenDBDataDir" # optional; defaults to a local RavenDB folder +options.with_auto_downloaded_server() +options.data_directory = "./data/RavenDB" +options.logs_path = "./logs/RavenDB" with EmbeddedServer() as server: server.start_server(options) - with server.get_document_store("MyDb") as store: - ... # ordinary ravendb client code + with server.get_document_store("MyDatabase") as store: + ... ``` -Runnable walkthrough: [`labs/01-embedded-zero-config.md`](labs/01-embedded-zero-config.md). +The operating system and architecture are detected at runtime, so the same Python configuration +is portable across: -### Run without installing .NET +- Windows x64 and x86 +- Linux x64 and ARM64 +- macOS x64 and ARM64 -On locked-down hosts or minimal CI images where you do not want a system .NET, bring a -**self-contained** RavenDB build (it bundles its own runtime). Point the server at the extracted -`Server` folder: the driver detects the bundled runtime and launches the server's native apphost -directly, never calling `dotnet`. +Unsupported targets, including Windows ARM64, fail with an explicit error instead of downloading +an incompatible build. + +The first run downloads a self-contained server (100 MB+) and caches it under +`~/.cache/ravendb-embedded`. Later runs reuse that cache. Pass `cache_root` to +`with_auto_downloaded_server()` when you want a different location: + +```python +options.with_auto_downloaded_server(cache_root="./.ravendb-cache") +``` + +By default, the download follows the installed package's RavenDB version line. The cache is not +refreshed automatically; remove that version's cache directory when you intentionally want to +resolve a newer server build from the same line. + +The wheel does not contain a self-contained build for every platform; it downloads only the build +needed by the current machine. Self-contained mode removes the system .NET requirement, but normal +RavenDB operating-system dependencies still apply. Minimal Linux images may need their +distribution's ICU package. + +Runnable walkthrough: [Lab 03 — on-demand server](labs/03-on-demand-server.md). + +### Self-contained server you provide + +Download and extract the Server package for the target platform, then point the package at its +`Server` directory: ```python from ravendb_embedded import EmbeddedServer, ServerOptions options = ServerOptions() -options.with_external_server("/path/to/extracted/Server") # a self-contained build +options.with_external_server("/path/to/extracted/Server") with EmbeddedServer() as server: server.start_server(options) - with server.get_document_store("MyDb") as store: + with server.get_document_store("MyDatabase") as store: ... ``` -Download the Server package for your platform from the [RavenDB downloads page](https://ravendb.net/downloads); -the server files live in the archive's `Server/` folder. Runnable walkthrough: -[`labs/02-embedded-external-server.md`](labs/02-embedded-external-server.md). +The native apphost is run directly, so `dotnet` is not called. Server packages are available on +the [RavenDB downloads page](https://ravendb.net/downloads). + +Runnable walkthrough: [Lab 02 — external self-contained server](labs/02-embedded-external-server.md). + +### Advanced server sources -Or skip the manual download and let the package fetch and cache one for you on first use: +`with_external_server()` also accepts a ZIP archive containing RavenDB server files. Applications +that ship the archive as a Python package resource can use +`ExtractFromPkgResourceServerProvider(package, resource_name)`. For other sources, implement +`ProvideRavenDBServer.provide(target_directory)` and assign the provider to `options.provider`. + +## Configuration + +Create a `ServerOptions` instance before starting the server: + +Construct options with `ServerOptions()`. The misleading `ServerOptions.INSTANCE()` constructor +alias is deprecated and will be removed in a future release. + +- `data_directory`: where database data is stored; retains the existing + `/RavenDB` default. Set it explicitly for applications that need a writable + or application-owned location. +- `logs_path`: where RavenDB writes its logs; defaults to `/Logs` and follows a + changed `data_directory` until you set `logs_path` explicitly. +- `accept_eula`: controls the RavenDB EULA command-line setting and defaults to `True`. +- `server_url`: address to bind; the default uses localhost and a free port. +- `dot_net_path`: path to `dotnet` for bundled mode when it is not on `PATH`. +- `command_line_args`: additional + [RavenDB server arguments](https://ravendb.net/docs/article-page/latest/csharp/server/configuration/command-line-arguments). +- `framework_version`: defaults to `""`, preserving the dotnet host's normal runtime selection and + roll-forward behavior. Set it to `auto` to read the server runtime configuration and select a + compatible installed patch, or set an exact version for advanced setups. +- `graceful_shutdown_timeout`: how long to wait before terminating the child process. +- `process_kill_timeout`: how long to wait for the process after terminating or killing it. +- `max_server_startup_time_duration`: maximum server startup time. + +Startup failures raise `ServerStartupError`. When the configured startup duration expires, the +more specific `ServerStartupTimeoutError` is raised. A failed start does not dispose the +`EmbeddedServer`; correct the configuration and call `start_server()` on the same instance again. ```python +from datetime import timedelta + +from ravendb_embedded import EmbeddedServer, ServerOptions, ServerStartupTimeoutError + options = ServerOptions() -options.with_auto_downloaded_server() # downloads + caches a self-contained server, no .NET needed -``` +options.max_server_startup_time_duration = timedelta(seconds=15) -The package detects the host operating system and architecture at runtime, so the same Python -configuration is portable across supported Windows, Linux, and macOS machines. +with EmbeddedServer() as server: + try: + server.start_server(options) + except ServerStartupTimeoutError: + options.max_server_startup_time_duration = timedelta(minutes=1) + server.start_server(options) +``` -Walkthrough: [`labs/03-on-demand-server.md`](labs/03-on-demand-server.md). -Self-contained builds remove the system .NET requirement, but the normal RavenDB OS dependencies -still apply. In particular, minimal Linux images may need their distribution's ICU package. +### License and EULA -### Don't manage a server at all (tests) +`accept_eula` remains enabled by default. Applications can configure it through +`options.accept_eula`. See the +[RavenDB EULA](https://ravendb.net/legal/ravendb/commercial-license-eula) for the applicable terms. -For test suites that should not touch .NET or embedded startup, `ravendb-test-driver` can attach -to a RavenDB you run yourself (Docker, testcontainers, a shared CI service) while still giving -each test its own database. See the -[`ravendb-python-testdriver`](https://github.com/ravendb/ravendb-python-testdriver) repository. +Configure a license through `options.licensing`: -## Configuration +```python +options.licensing.license = '{"Id": "..."}' # Inline license JSON +# Or: +options.licensing.license_path = "/path/to/license.json" + +options.licensing.disable_auto_update = True +options.licensing.disable_auto_update_from_api = True +options.licensing.disable_license_support_check = True +options.licensing.throw_on_invalid_or_missing_license = True +``` -### `ServerOptions` +The fail-fast option is useful in CI and production environments where falling back to an +unlicensed server would hide a configuration problem. -Create `ServerOptions()` and set attributes: +### HTTPS and client certificates -- `data_directory`: where database data is stored (defaults to a local `RavenDB` folder). Set a - stable path for data that outlives the process, see - [`labs/05-embedded-persistent.md`](labs/05-embedded-persistent.md). -- `server_url`: the URL to bind (defaults to localhost on a free port). -- `dot_net_path`: path to `dotnet` when it is not on `PATH` (ignored on the self-contained path). -- `command_line_args`: extra [server command-line arguments](https://ravendb.net/docs/article-page/latest/csharp/server/configuration/command-line-arguments). -- `framework_version`: pin an exact .NET version (advanced; leave empty to autodetect the installed runtime). +Use `ServerOptions.secured()` with a server PFX to start RavenDB over HTTPS: -### Security +```python +options = ServerOptions() +options.secured("server.pfx") +``` -Secure the server with `ServerOptions.secured()`: +This configures the embedded server only. External clients must authenticate with a certificate +trusted by RavenDB. To let `EmbeddedServer` create authenticated `DocumentStore` instances, supply +a client PEM as well: ```python options = ServerOptions() options.secured( - server_pfx_certificate_path, # server certificate (.pfx), required - client_pem_certificate_path, # client certificate (.pem) - server_pfx_certificate_password=None, - ca_certificate_path=None, + "server.pfx", + "client.pem", + server_pfx_certificate_password="", + ca_certificate_path="ca.crt", +) +``` + +The returned `DocumentStore` receives the client certificate and custom CA automatically. +The client PEM must contain both its certificate and matching unencrypted private key. If the +certificate declares Extended Key Usage, it must include TLS Client Authentication. + +The server PFX is not reused as a client certificate: a server-only certificate may +not have Client Authentication EKU, and its PFX may not contain a suitable client identity. +`ca_certificate_path` is optional for certificates trusted by the operating system; provide a CA +bundle for private or self-signed server certificates. A supplied client PEM and CA bundle are +validated before the RavenDB process starts. + +When another process supplies the server certificate, use `secured_with_certificate_exec()`: + +```python +options.secured_with_certificate_exec( + certificate_exec="python", + certificate_arguments='"load_certificate.py" "server.pfx"', + client_pem_certificate_path="client.pem", + ca_certificate_path="ca.crt", +) +``` + +The certificate command must write the PFX bytes to standard output. + +Runnable walkthrough: [Lab 04 — secured embedded server](labs/04-embedded-secured.md). + +### Server lifecycle + +Use `get_server_process_id()` to inspect the child process. `stop_server()` stops RavenDB without +disposing the `EmbeddedServer`, and `restart_server()` starts it again with the same options. +Calling `close()` stops the process and disposes all document stores. + +The public API is synchronous, matching the RavenDB Python client's execution model. Server start, +shutdown, and restart calls block until the requested lifecycle transition completes. Async +applications can run these blocking lifecycle calls with `asyncio.to_thread()` instead of +maintaining a second embedded API with different behavior. + +`EmbeddedServer` is intentionally not a singleton. An application may run independent server +instances in the same Python process; give each instance a different `data_directory` and +`logs_path`. Prefer `with EmbeddedServer() as server:` so the context manager always closes that +instance's process and document stores. + +```python +first_options = ServerOptions() +first_options.data_directory = "./servers/first" + +second_options = ServerOptions() +second_options.data_directory = "./servers/second" + +with EmbeddedServer() as first, EmbeddedServer() as second: + first.start_server(first_options) + second.start_server(second_options) + assert first.get_server_process_id() != second.get_server_process_id() +``` + +Register a process-exit callback when the application needs to observe server failures: + +```python +server.add_server_process_exited( + lambda event: print(event.process_id, event.exit_code, event.expected) ) ``` -Runnable example (HTTPS + client-certificate auth): -[`labs/04-embedded-secured.md`](labs/04-embedded-secured.md). +Callbacks run on a background thread. `event.expected` is `True` for exits caused by +`stop_server()`, `restart_server()`, or `close()`, and `False` when the child exits unexpectedly. + +Runnable walkthrough: [Lab 06 — lifecycle and process monitoring](labs/06-embedded-lifecycle.md). + +### Persistent data + +The default data directory remains `RavenDB` inside the installed package directory. Set +`data_directory` explicitly when the package directory may be read-only, the application owns its +storage layout, or several embedded servers run in one process. Use a temporary directory for +disposable tests. + +Runnable walkthrough: [Lab 05 — persistent data](labs/05-embedded-persistent.md). -### Working with data +### Document stores -`get_document_store(database_name)` returns a `DocumentStore` you use like any RavenDB client. For -finer control, build a `DatabaseOptions` (via `DatabaseOptions.from_database_name`) and call -`get_document_store_from_options`; set `skip_creating_database=True` to not auto-create the -database. +`get_document_store(database_name)` returns a standard RavenDB `DocumentStore` and creates the +database when needed. For finer control, create `DatabaseOptions` with +`DatabaseOptions.from_database_name()` and call `get_document_store_from_options()`. -Call `open_studio_in_browser()` to open RavenDB Studio in your default browser. +Set `skip_creating_database=True` on `DatabaseOptions` when the database is managed elsewhere. +Call `open_studio_in_browser()` to open RavenDB Studio. ## Labs -The `labs/` folder holds runnable, self-checking guides, one per usage case above. The scripts live -in this repository rather than in the installed wheel, so run them from a checkout. Start at -[`labs/README.md`](labs/README.md). +The repository contains runnable, self-checking examples: + +| Lab | Scenario | Needs system .NET? | +|-----|----------|--------------------| +| [01](labs/01-embedded-zero-config.md) | Bundled zero-config server | Yes | +| [02](labs/02-embedded-external-server.md) | Self-contained server you provide | No | +| [03](labs/03-on-demand-server.md) | Automatic platform detection, download, and cache | No | +| [04](labs/04-embedded-secured.md) | HTTPS and client-certificate authentication | Yes | +| [05](labs/05-embedded-persistent.md) | Data that survives server restarts | Yes | +| [06](labs/06-embedded-lifecycle.md) | Stop, restart, PID, and process-exit monitoring | Yes | + +The scripts live in the repository rather than the installed wheel. Clone or download the +repository, install the package, and run them from the repository root. See the +[complete labs guide](labs/README.md). + +## Links + +- [PyPI](https://pypi.org/project/ravendb-embedded/) +- [Source](https://github.com/ravendb/ravendb-python-embedded) +- [RavenDB Python client documentation](https://ravendb.net/docs/article-page/latest/python) diff --git a/labs/01-embedded-zero-config.md b/labs/01-embedded-zero-config.md index 852a317..cbc0b16 100644 --- a/labs/01-embedded-zero-config.md +++ b/labs/01-embedded-zero-config.md @@ -17,10 +17,12 @@ The complete, runnable example is [`01_embedded_zero_config.py`](01_embedded_zer The core is just: ```python -from ravendb_embedded import EmbeddedServer +from ravendb_embedded import EmbeddedServer, ServerOptions + +options = ServerOptions() with EmbeddedServer() as server: - server.start_server() + server.start_server(options) with server.get_document_store("Lab") as store: with store.open_session() as session: session.store({"name": "Ayende"}, "people/1") @@ -33,10 +35,10 @@ with EmbeddedServer() as server: server (`Raven.Server.dll`) that ships inside the wheel. 2. `RavenServerRunner` launches it as **`dotnet Raven.Server.dll ...`**, using the `dotnet` found on your `PATH` (`ServerOptions.dot_net_path`). -3. `ServerOptions.framework_version` is empty by default, so no `--fx-version` is passed and - standard .NET host resolution applies. The host does **not** roll forward across major - versions, which is why a net10.0 server hard-fails on a machine that only has .NET 8 - (error: "You must install ... version 10.0.x"). +3. `ServerOptions.framework_version` defaults to `""`, leaving runtime selection and roll-forward + to the dotnet host. Set it to `auto` when you want the package to read + `Raven.Server.runtimeconfig.json` and select a compatible installed patch. A net10.0 server + still requires .NET 10. 4. The one exception: point the server at a **self-contained** build (an apphost `Raven.Server` with the runtime bundled) and it runs **without `dotnet`**. That is Lab 02. diff --git a/labs/04-embedded-secured.md b/labs/04-embedded-secured.md index b478edd..e1adc1e 100644 --- a/labs/04-embedded-secured.md +++ b/labs/04-embedded-secured.md @@ -30,10 +30,13 @@ with EmbeddedServer() as server: ## Certificates -`secured()` needs a server `.pfx` and a client `.pem`; a password and a CA certificate are -optional. In production you bring your own (for example from RavenDB's setup wizard or your CA). -The lab generates a throwaway self-signed pair so it can run unattended; that generator is demo -code, not something to ship. A RavenDB server certificate needs the right extensions +The server `.pfx` is required. The client `.pem` is needed when `EmbeddedServer` creates +authenticated document stores; a password and a CA certificate are optional. To configure only +the HTTPS server and use clients managed elsewhere, call `options.secured(server_pfx_path)`. + +In production you bring your own certificates (for example from RavenDB's setup wizard or your +CA). The lab generates a throwaway self-signed pair so it can run unattended; that generator is +demo code, not something to ship. A RavenDB server certificate needs the right extensions (digitalSignature key usage, serverAuth/clientAuth EKU, and SANs for how the server is reached), which the lab's generator sets. diff --git a/labs/04_embedded_secured.py b/labs/04_embedded_secured.py index 4e64295..127a0ee 100644 --- a/labs/04_embedded_secured.py +++ b/labs/04_embedded_secured.py @@ -65,6 +65,8 @@ def _demo_certificates(directory): .add_extension( x509.ExtendedKeyUsage([ExtendedKeyUsageOID.SERVER_AUTH, ExtendedKeyUsageOID.CLIENT_AUTH]), critical=False ) + .add_extension(x509.SubjectKeyIdentifier.from_public_key(key.public_key()), critical=False) + .add_extension(x509.AuthorityKeyIdentifier.from_issuer_public_key(key.public_key()), critical=False) .sign(key, hashes.SHA256()) ) cert_pem = cert.public_bytes(serialization.Encoding.PEM) diff --git a/labs/05-embedded-persistent.md b/labs/05-embedded-persistent.md index 9a1f869..5bf6e7a 100644 --- a/labs/05-embedded-persistent.md +++ b/labs/05-embedded-persistent.md @@ -37,8 +37,9 @@ with EmbeddedServer() as server: ## Notes -- The default `data_directory` is a folder under the package location, which is fine for tests - but not what you want for real data. Set it explicitly to a path you control. +- The default `data_directory` remains `RavenDB` inside the installed package directory. Set it + explicitly when the package directory may be read-only or the application needs a fixed, + application-owned location. - `get_document_store("Lab")` reuses the existing database on the second run; it only creates one when it is missing. diff --git a/labs/06-embedded-lifecycle.md b/labs/06-embedded-lifecycle.md new file mode 100644 index 0000000..a72fe1d --- /dev/null +++ b/labs/06-embedded-lifecycle.md @@ -0,0 +1,56 @@ +# Lab 06: Server lifecycle and process monitoring + +**For:** applications that need to stop or restart their managed RavenDB process and observe +whether a process exit was expected or unexpected. + +## Run it + +```bash +pip install ravendb-embedded +python labs/06_embedded_lifecycle.py +``` + +The complete, self-checking example is +[`06_embedded_lifecycle.py`](06_embedded_lifecycle.py). It exercises: + +- `get_server_process_id()` +- `stop_server()` +- `restart_server()` +- `add_server_process_exited()` +- document access after restart + +The core lifecycle flow is: + +```python +from threading import Event + +from ravendb_embedded import EmbeddedServer, ServerOptions + +options = ServerOptions() + +exit_observed = Event() +exit_events = [] + +with EmbeddedServer() as server: + server.add_server_process_exited( + lambda event: (exit_events.append(event), exit_observed.set()) + ) + server.start_server(options) + first_process_id = server.get_server_process_id() + + server.stop_server() + assert exit_observed.wait(10) + assert exit_events[-1].process_id == first_process_id + assert exit_events[-1].expected + + server.restart_server() + assert server.get_server_process_id() != first_process_id +``` + +`event.expected` is `True` for `stop_server()`, `restart_server()`, and context-manager cleanup. +It is `False` when RavenDB exits on its own. + +## Takeaway + +`EmbeddedServer` owns the RavenDB child process but exposes enough lifecycle information for an +application to coordinate restarts and distinguish planned shutdowns from server failures. diff --git a/labs/06_embedded_lifecycle.py b/labs/06_embedded_lifecycle.py new file mode 100644 index 0000000..d54b7ea --- /dev/null +++ b/labs/06_embedded_lifecycle.py @@ -0,0 +1,53 @@ +"""Lab 06: Stop, restart, process IDs, and process-exit notifications.""" + +import tempfile +from pathlib import Path +from threading import Event + +from ravendb_embedded import EmbeddedServer, ServerOptions + + +def main() -> None: + with tempfile.TemporaryDirectory() as work: + options = ServerOptions() + options.data_directory = str(Path(work, "data")) + options.logs_path = str(Path(work, "logs")) + + exit_observed = Event() + exit_events = [] + + def on_server_exit(event) -> None: + exit_events.append(event) + exit_observed.set() + + with EmbeddedServer() as server: + server.add_server_process_exited(on_server_exit) + server.start_server(options) + first_process_id = server.get_server_process_id() + print(f"Started RavenDB process {first_process_id}") + + server.stop_server() + if not exit_observed.wait(10): + raise RuntimeError("The server exit callback was not invoked.") + + stopped = exit_events[-1] + assert stopped.process_id == first_process_id + assert stopped.expected + print(f"Stopped RavenDB process {stopped.process_id}") + + exit_observed.clear() + server.restart_server() + second_process_id = server.get_server_process_id() + assert second_process_id != first_process_id + print(f"Restarted RavenDB as process {second_process_id}") + + with server.get_document_store("LifecycleLab") as store: + with store.open_session() as session: + session.store({"status": "running"}, "status/1") + session.save_changes() + + print("PASS: stop, exit notification, restart, and document access succeeded.") + + +if __name__ == "__main__": + main() diff --git a/labs/README.md b/labs/README.md index 1331b90..de61f60 100644 --- a/labs/README.md +++ b/labs/README.md @@ -14,6 +14,7 @@ the released library and server binaries used by the scripts. | [03](03-on-demand-server.md) | On-demand cached self-contained download (no manual steps) | No | | [04](04-embedded-secured.md) | Secured embedded server (HTTPS + client certificate) | Yes | | [05](05-embedded-persistent.md) | Persistent data directory (data survives restarts) | Yes | +| [06](06-embedded-lifecycle.md) | Stop, restart, PID, and process-exit monitoring | Yes | RavenDB version to .NET mapping: **7.1.x needs .NET 8, 7.2.x needs .NET 10.** The bundled server decides this, so it can change on a minor bump; check the lab for your version. diff --git a/ravendb_embedded/__init__.py b/ravendb_embedded/__init__.py index faac0e1..a4416b4 100644 --- a/ravendb_embedded/__init__.py +++ b/ravendb_embedded/__init__.py @@ -1,9 +1,16 @@ -from ravendb_embedded.embedded_server import EmbeddedServer -from ravendb_embedded.options import DatabaseOptions, ServerOptions, SecurityOptions +from ravendb_embedded.embedded_server import ( + EmbeddedServer, + ServerProcessExitedEvent, + ServerStartupError, + ServerStartupTimeoutError, +) +from ravendb_embedded.options import DatabaseOptions, LicensingOptions, ServerOptions, SecurityOptions from ravendb_embedded.on_demand import ensure_server from ravendb_embedded.provide import ( CopyServerFromNugetProvider, CopyServerProvider, ExternalServerProvider, + ExtractFromPkgResourceServerProvider, ExtractFromZipServerProvider, + ProvideRavenDBServer, ) diff --git a/ravendb_embedded/embedded_server.py b/ravendb_embedded/embedded_server.py index 9d89ce8..3b18a91 100644 --- a/ravendb_embedded/embedded_server.py +++ b/ravendb_embedded/embedded_server.py @@ -6,32 +6,53 @@ import shutil import subprocess import time +from dataclasses import dataclass from datetime import timedelta -from typing import Optional, List, Callable, Tuple, Generic, TypeVar, Dict, IO -from threading import Thread +from typing import Optional, List, Callable, Tuple, Generic, TypeVar +from threading import Event, Lock, RLock, Thread from queue import Queue import webbrowser from ravendb import DocumentStore, CreateDatabaseOperation -from ravendb.exceptions.raven_exceptions import RavenException -from ravendb.tools.utils import Stopwatch +from ravendb.exceptions.raven_exceptions import ConcurrencyException from ravendb_embedded.options import ServerOptions, DatabaseOptions from ravendb_embedded.raven_server_runner import RavenServerRunner _T = TypeVar("_T") -class EmbeddedServer: - END_OF_STREAM_MARKER = "$$END_OF_STREAM$$" +class ServerStartupError(RuntimeError): + pass + + +class ServerStartupTimeoutError(ServerStartupError): + pass + + +@dataclass(frozen=True) +class ServerProcessExitedEvent: + process_id: int + exit_code: int + expected: bool + - # singleton +class EmbeddedServer: def __init__(self): self.server_task: Optional[Lazy[Tuple[str, subprocess.Popen]]] = None + self._server_options: Optional[ServerOptions] = None + self._lifecycle_lock = RLock() + self._exit_handler: Optional[Callable[[], None]] = None + self._process_exit_lock = RLock() + self._process_exit_callbacks: List[Callable[[ServerProcessExitedEvent], None]] = [] + self._watched_processes = set() + self._expected_process_exits = set() self.document_stores = {} + self._document_stores_lock = RLock() self.client_pem_certificate_path: Optional[str] = None self.trust_store_path: Optional[str] = None self._graceful_shutdown_timeout: Optional[timedelta] = None - self.logger = logging.Logger(self.__class__.__name__, logging.DEBUG) + self._process_kill_timeout: Optional[timedelta] = None + self.logger = logging.getLogger(f"ravendb_embedded.{self.__class__.__name__}") def __enter__(self): return self @@ -43,23 +64,44 @@ def _log_debug(self, message: str) -> None: if not self.logger.disabled and self.logger.isEnabledFor(logging.DEBUG): self.logger.log(logging.DEBUG, message) + def add_server_process_exited(self, callback: Callable[[ServerProcessExitedEvent], None]) -> None: + if not callable(callback): + raise ValueError("callback must be callable") + with self._process_exit_lock: + if callback not in self._process_exit_callbacks: + self._process_exit_callbacks.append(callback) + + def remove_server_process_exited(self, callback: Callable[[ServerProcessExitedEvent], None]) -> None: + with self._process_exit_lock: + if callback in self._process_exit_callbacks: + self._process_exit_callbacks.remove(callback) + def start_server(self, options_param: ServerOptions = None) -> None: options = options_param or ServerOptions() - self._graceful_shutdown_timeout = options.graceful_shutdown_timeout + with self._lifecycle_lock: + if self.server_task is not None: + raise RuntimeError("The server was already started") - start_server = Lazy(lambda: self._run_server(options)) + self._server_options = options + self._graceful_shutdown_timeout = options.graceful_shutdown_timeout + self._process_kill_timeout = options.process_kill_timeout - if self.server_task and self.server_task.created and self.server_task != start_server: - raise RuntimeError("The server was already started") + start_server = Lazy(lambda: self._run_server(options)) + self.server_task = start_server - self.server_task = start_server + if options.security is not None: + self.client_pem_certificate_path = options.security.client_pem_certificate_path + self.trust_store_path = options.security.ca_certificate_path - if options.security is not None: - self.client_pem_certificate_path = options.security.client_pem_certificate_path - self.trust_store_path = options.security.ca_certificate_path - - start_server.get_value() + try: + start_server.get_value() + except Exception: + if self.server_task is start_server: + self.server_task = None + self._server_options = None + self._unregister_exit_handler() + raise def get_document_store(self, database: str) -> DocumentStore: return self.get_document_store_from_options(DatabaseOptions.from_database_name(database)) @@ -73,7 +115,11 @@ def _initialize_document_store(self, database_name, options): store.trust_store_path = self.trust_store_path store.conventions = options.conventions - store.add_after_close(lambda: self.document_stores.pop(database_name)) + def remove_store(): + with self._document_stores_lock: + self.document_stores.pop(database_name, None) + + store.add_after_close(remove_store) store.initialize() @@ -84,60 +130,175 @@ def _initialize_document_store(self, database_name, options): def get_document_store_from_options(self, options: DatabaseOptions) -> DocumentStore: database_name = options.database_record.database_name - if not database_name or database_name.isspace(): raise ValueError("DatabaseName cannot be null or whitespace") - self._log_debug(f"Creating document store for '{database_name}'.") + with self._lifecycle_lock: + if self.server_task is None: + raise RuntimeError("Please run start_server() before trying to use the server.") + + self._log_debug(f"Creating document store for '{database_name}'.") - lazy = Lazy(lambda: self._initialize_document_store(database_name, options)) + with self._document_stores_lock: + lazy = self.document_stores.get(database_name) + if lazy is None: + lazy = Lazy(lambda: self._initialize_document_store(database_name, options)) + self.document_stores[database_name] = lazy - return self.document_stores.setdefault(database_name, lazy).get_value() + return lazy.get_value() def _try_create_database(self, options: DatabaseOptions, store: DocumentStore) -> None: try: store.maintenance.server.send(CreateDatabaseOperation(options.database_record)) - except Exception as e: - # Expected behavior when the database already exists - if "conflict" in e.args[0] or "already exists" in e.args[0]: - self._log_debug(f"{options.database_record.database_name} already exists.") - else: - raise e + except ConcurrencyException: + self._log_debug(f"{options.database_record.database_name} already exists.") def get_server_uri(self) -> str: - server = self.server_task - if server is None: - raise RuntimeError("Please run start_server() before trying to use the server.") + with self._lifecycle_lock: + server = self.server_task + if server is None: + raise RuntimeError("Please run start_server() before trying to use the server.") + + return server.get_value()[0] + + def get_server_process_id(self) -> int: + with self._lifecycle_lock: + server = self._require_started_server("get_server_process_id") + return server.get_value()[1].pid + + def stop_server(self) -> None: + with self._lifecycle_lock: + server = self._require_started_server("stop_server") + self._unregister_exit_handler() + self._shutdown_server_process(server.get_value()[1]) + + def restart_server(self) -> None: + with self._lifecycle_lock: + existing_server = self._require_started_server("restart_server") + options = self._server_options + + self._unregister_exit_handler() + try: + self._shutdown_server_process(existing_server.get_value()[1]) + except Exception: + pass + + if self.server_task is not existing_server: + raise RuntimeError("The server changed while restarting it") - return server.get_value()[0] + restarted_server = Lazy(lambda: self._run_server(options)) + self.server_task = restarted_server + try: + restarted_server.get_value() + except Exception: + if self.server_task is restarted_server: + self.server_task = None + self._unregister_exit_handler() + raise + + def _require_started_server(self, operation: str) -> Lazy[Tuple[str, subprocess.Popen]]: + server = self.server_task + if self._server_options is None or server is None or not server.created: + raise RuntimeError(f"Cannot call {operation}() before calling start_server()") + return server def _shutdown_server_process(self, process: subprocess.Popen) -> None: - if not process or process.poll() is not None: + if not process: return - with process: - if process.poll() is not None: # Check if the process has already terminated + try: + if process.poll() is not None: return - try: - self._log_debug("Try shutdown server gracefully.") - with process.stdin: - process.stdin.write("shutdown no-confirmation\n".encode("utf-8")) + self._mark_process_exit_expected(process) + graceful_timeout = (self._graceful_shutdown_timeout or timedelta(seconds=30)).total_seconds() + kill_timeout = (self._process_kill_timeout or timedelta(seconds=5)).total_seconds() - if process.wait(self._graceful_shutdown_timeout.total_seconds()) == 0: - return + try: + self._log_debug("Trying to shut down the server gracefully.") + if process.stdin is None: + raise RuntimeError("The server process stdin is not available.") - except Exception as e: + process.stdin.write(b"shutdown no-confirmation\n") + process.stdin.flush() + process.wait(timeout=graceful_timeout) + return + except Exception as error: self._log_debug( - f"Failed to gracefully shutdown server in {self._graceful_shutdown_timeout}. Error: {e}" + f"Failed to gracefully shut down the server in {self._graceful_shutdown_timeout}. Error: {error}" ) + if process.poll() is not None: + return + + try: + self._log_debug("Terminating the server process.") + process.terminate() + process.wait(timeout=kill_timeout) + return + except subprocess.TimeoutExpired: + self._log_debug(f"The server did not terminate in {self._process_kill_timeout}; killing it.") + except Exception as error: + self._log_debug(f"Failed to terminate the server process. Error: {error}") + + if process.poll() is not None: + return + + try: + process.kill() + process.wait(timeout=kill_timeout) + except Exception as error: + self._log_debug(f"Failed to kill the server process in {self._process_kill_timeout}. Error: {error}") + finally: + if process.poll() is not None: + self._close_process_streams(process) + + @staticmethod + def _close_process_streams(process: subprocess.Popen) -> None: + for stream in (process.stdin, process.stdout, process.stderr): + if stream is None or stream.closed: + continue try: - self._log_debug("Killing global server") - process.terminate() # Use terminate() instead of kill() - process.wait() - except Exception as e: - self._log_debug(f"Failed to terminate server process. Error: {e}") + stream.close() + except Exception: + pass + + def _register_exit_handler(self, process: subprocess.Popen) -> None: + self._unregister_exit_handler() + self._exit_handler = lambda: self._shutdown_server_process(process) + atexit.register(self._exit_handler) + + def _watch_server_process(self, process: subprocess.Popen) -> None: + with self._process_exit_lock: + self._watched_processes.add(process.pid) + + def watch(): + exit_code = process.wait() + with self._process_exit_lock: + expected = process.pid in self._expected_process_exits + self._expected_process_exits.discard(process.pid) + self._watched_processes.discard(process.pid) + callbacks = list(self._process_exit_callbacks) + + event = ServerProcessExitedEvent(process.pid, exit_code, expected) + for callback in callbacks: + try: + callback(event) + except Exception as error: + self._log_debug(f"Server process exit callback failed. Error: {error}") + + Thread(target=watch, daemon=True, name=f"ravendb-process-{process.pid}").start() + + def _mark_process_exit_expected(self, process: subprocess.Popen) -> None: + with self._process_exit_lock: + if process.pid in self._watched_processes: + self._expected_process_exits.add(process.pid) + + def _unregister_exit_handler(self) -> None: + if self._exit_handler is None: + return + atexit.unregister(self._exit_handler) + self._exit_handler = None def _run_server(self, options: ServerOptions) -> Tuple[str, subprocess.Popen]: try: @@ -156,28 +317,95 @@ def _run_server(self, options: ServerOptions) -> Tuple[str, subprocess.Popen]: self._log_debug("Starting global server") - atexit.register(lambda: self._shutdown_server_process(process)) + server_url, output_string, error_string, timed_out = self._wait_for_server_start(process, options) + if server_url is None: + self._shutdown_server_process(process) + message = self.build_startup_exception_message( + output_string, + error_string, + process, + timed_out=timed_out, + startup_timeout=options.max_server_startup_time_duration, + ) + if timed_out: + raise ServerStartupTimeoutError(message) + raise ServerStartupError(message) - url_ref: Dict[str, Optional[str]] = {"value": None} - startup_duration = Stopwatch.create_started() + self._register_exit_handler(process) + self._watch_server_process(process) + return server_url, process - output_string = self.read_output( - process.stdout, - startup_duration, - options, - lambda line, builder: self.online(line, builder, url_ref, process, startup_duration, options), - ) + def _wait_for_server_start( + self, + process: subprocess.Popen, + options: ServerOptions, + ) -> Tuple[Optional[str], str, str, bool]: + output_events: Queue[Tuple[str, Optional[str]]] = Queue() + startup_complete = Event() - if url_ref["value"] is None: - error_string = self.read_output(process.stderr, Stopwatch.create_started(), options, None) - self._shutdown_server_process(process) - raise RuntimeError(self.build_startup_exception_message(output_string, error_string, process)) + def read_stream(stream_name, stream): + try: + for raw_line in iter(stream.readline, b""): + line = raw_line.decode("utf-8", errors="replace").rstrip("\r\n") + if not startup_complete.is_set(): + output_events.put((stream_name, line)) + except Exception as error: + if not startup_complete.is_set(): + output_events.put((stream_name, f"Unable to read process output: {error}")) + finally: + if not startup_complete.is_set(): + output_events.put((stream_name, None)) + + Thread(target=read_stream, args=("stdout", process.stdout), daemon=True).start() + Thread(target=read_stream, args=("stderr", process.stderr), daemon=True).start() + + stdout_lines = [] + stderr_lines = [] + ended_streams = set() + deadline = time.monotonic() + options.max_server_startup_time_duration.total_seconds() + prefix = "Server available on: " - return url_ref["value"], process + try: + while True: + remaining = deadline - time.monotonic() + if remaining <= 0: + return None, self._join_output(stdout_lines), self._join_output(stderr_lines), True + + try: + stream_name, line = output_events.get(timeout=min(0.1, remaining)) + except queue.Empty: + continue + + if line is None: + ended_streams.add(stream_name) + if len(ended_streams) == 2: + return None, self._join_output(stdout_lines), self._join_output(stderr_lines), False + continue + + lines = stdout_lines if stream_name == "stdout" else stderr_lines + lines.append(line) + if stream_name == "stdout" and line.startswith(prefix): + return line[len(prefix) :], self._join_output(stdout_lines), self._join_output(stderr_lines), False + finally: + startup_complete.set() + + @staticmethod + def _join_output(lines: List[str]) -> str: + return os.linesep.join(lines) + (os.linesep if lines else "") @staticmethod - def build_startup_exception_message(output_string: str, error_string: str, process: subprocess.Popen) -> str: - sb = ["Unable to start the RavenDB Server", os.linesep] + def build_startup_exception_message( + output_string: str, + error_string: str, + process: subprocess.Popen, + timed_out: bool = False, + startup_timeout: timedelta = None, + ) -> str: + if timed_out: + heading = f"Server failed to start in {startup_timeout.total_seconds()} seconds." + else: + heading = "Unable to start the RavenDB Server" + sb = [heading, os.linesep] if process.args: sb.append("Command:") @@ -200,104 +428,41 @@ def build_startup_exception_message(output_string: str, error_string: str, proce sb.append("Check your ServerOptions and host dependencies, or run the command manually to see detailed error.") return "".join(sb) - def online( - self, - line: str, - builder: List[str], - url_ref: Dict[str, Optional[str]], - process: subprocess.Popen, - startup_duration: Stopwatch, - options: ServerOptions, - ): - if line is None: - error_string = self.read_output(process.stderr, Stopwatch.create_started(), options, None) - self._shutdown_server_process(process) - raise RuntimeError(self.build_startup_exception_message("".join(builder), error_string, process)) - - prefix = "Server available on: " - if line.startswith(prefix): - url_ref["value"] = line[len(prefix) :] - return True - - return False - - def read_output( - self, - output: IO, - startup_duration: Stopwatch, - options: ServerOptions, - online: Optional[Callable[[str, List[str]], bool]], - ): - def read_output_line() -> Optional[str]: - while True: - try: - line_ = output_queue.get_nowait() - return line_ - except queue.Empty: - if options.max_server_startup_time_duration - startup_duration.elapsed() <= timedelta(seconds=0): - return None - time.sleep(1) - - def output_reader(): - try: - for line_ in iter(output.readline, b""): - output_queue.put(line_.decode("utf-8").strip()) - output_queue.put(self.END_OF_STREAM_MARKER) - except Exception as e: - raise RavenException("Unable to read server output") from e - - output_queue: Queue[str] = Queue() - output_thread = Thread(target=output_reader, daemon=True) - output_thread.start() - - sb = [] - - while True: - line = read_output_line() - - if options.max_server_startup_time_duration < startup_duration.elapsed(): - return "".join(sb) - - if line is None: - break - - if line == self.END_OF_STREAM_MARKER: - break - - sb.append(line) - sb.append(os.linesep) - - should_stop = False - if online is not None: - should_stop = online(line, sb) - - if should_stop: - break - - return "".join(sb) - def open_studio_in_browser(self): - server_url = self.get_server_uri() + studio_url = self._get_studio_url(self.get_server_uri()) try: - webbrowser.open(server_url) + webbrowser.open(studio_url) except Exception as e: raise RuntimeError(e) + @staticmethod + def _get_studio_url(server_url: str) -> str: + return f"{server_url.rstrip('/')}/studio/index.html?disableAnalytics=true" + def close(self): - lazy = self.server_task - if lazy is None or not lazy.created: - return + with self._lifecycle_lock: + lazy = self.server_task + if lazy is None or not lazy.created: + return + + self.server_task = None + self._unregister_exit_handler() + process = lazy.get_value()[1] + self._shutdown_server_process(process) + + with self._document_stores_lock: + stores = list(self.document_stores.values()) - process = lazy.get_value()[1] - self._shutdown_server_process(process) + for value in stores: + if value.created: + value.get_value().close() - for key, value in self.document_stores.items(): - if value.created: - value.get_value().close() + self.document_stores.clear() - self.document_stores.clear() - self.server_task = None + self._server_options = None + self.client_pem_certificate_path = None + self.trust_store_path = None class Lazy(Generic[_T]): @@ -305,13 +470,17 @@ def __init__(self, func: Callable[[], _T]): self.func = func self._value = None self._created = False + self._lock = Lock() def get_value(self) -> _T: if not self._created: - self._value = self.func() - self._created = True + with self._lock: + if not self._created: + self._value = self.func() + self._created = True return self._value @property def created(self): - return self._created + with self._lock: + return self._created diff --git a/ravendb_embedded/options.py b/ravendb_embedded/options.py index 535b81a..33ebf45 100644 --- a/ravendb_embedded/options.py +++ b/ravendb_embedded/options.py @@ -1,6 +1,7 @@ from __future__ import annotations import os +import warnings from datetime import timedelta from pathlib import Path from typing import Optional @@ -38,14 +39,26 @@ def __init__(self): self.certificate_arguments: Optional[str] = None +class LicensingOptions: + def __init__(self): + self.license: Optional[str] = None + self.license_path: Optional[str] = None + self.disable_auto_update: bool = False + self.disable_auto_update_from_api: bool = False + self.disable_license_support_check: bool = True + self.throw_on_invalid_or_missing_license: bool = False + + class ServerOptions: BASE_MODULE_DIRECTORY = str(Path(__file__).parent) DEFAULT_SERVER_LOCATION = os.path.join(BASE_MODULE_DIRECTORY, CopyServerFromNugetProvider.SERVER_FILES) + _DEFAULT_DATA_DIRECTORY = BASE_MODULE_DIRECTORY + "/RavenDB" + _DEFAULT_LOGS_PATH = BASE_MODULE_DIRECTORY + "/RavenDB/Logs" def __init__(self): self.framework_version: Optional[str] = "" - self.logs_path: str = self.BASE_MODULE_DIRECTORY + "/RavenDB/Logs" - self.data_directory: str = self.BASE_MODULE_DIRECTORY + "/RavenDB" + self._data_directory: str = self._DEFAULT_DATA_DIRECTORY + self._logs_path: Optional[str] = None self.provider: ProvideRavenDBServer = CopyServerFromNugetProvider() self.target_server_location: str = self.DEFAULT_SERVER_LOCATION self.dot_net_path: str = "dotnet" @@ -53,12 +66,39 @@ def __init__(self): self.accept_eula: bool = True self.server_url: Optional[str] = None self.graceful_shutdown_timeout: timedelta = timedelta(seconds=30) + self.process_kill_timeout: timedelta = timedelta(seconds=5) self.max_server_startup_time_duration: timedelta = timedelta(minutes=1) self.command_line_args: list[str] = list() + self.licensing: LicensingOptions = LicensingOptions() self.security: Optional[SecurityOptions] = None + @property + def data_directory(self) -> str: + return self._data_directory + + @data_directory.setter + def data_directory(self, value: str) -> None: + self._data_directory = value + + @property + def logs_path(self) -> str: + if self._logs_path is not None: + return self._logs_path + if self._data_directory == self._DEFAULT_DATA_DIRECTORY: + return self._DEFAULT_LOGS_PATH + return str(Path(self._data_directory) / "Logs") + + @logs_path.setter + def logs_path(self, value: Optional[str]) -> None: + self._logs_path = value + @classmethod def INSTANCE(cls): + warnings.warn( + "ServerOptions.INSTANCE() is deprecated; construct ServerOptions() directly.", + DeprecationWarning, + stacklevel=2, + ) return cls() @classmethod @@ -93,6 +133,29 @@ def secured( return self + def secured_with_certificate_exec( + self, + certificate_exec: str, + certificate_arguments: str, + client_pem_certificate_path: str, + ca_certificate_path: str = None, + ) -> "ServerOptions": + if certificate_exec is None: + raise ValueError("certificate_exec cannot be None") + if certificate_arguments is None: + raise ValueError("certificate_arguments cannot be None") + if client_pem_certificate_path is None: + raise ValueError("client_pem_certificate_path cannot be None") + if self.security is not None: + raise RuntimeError("The security has already been set up for this ServerOptions object") + + self.security = SecurityOptions() + self.security.certificate_exec = certificate_exec + self.security.certificate_arguments = certificate_arguments + self.security.client_pem_certificate_path = client_pem_certificate_path + self.security.ca_certificate_path = ca_certificate_path + return self + def with_external_server(self, server_location: str) -> None: self.provider = ExternalServerProvider(server_location) # A directory is already a runnable server: run it in place, so we neither copy it nor diff --git a/ravendb_embedded/provide.py b/ravendb_embedded/provide.py index ecffbb3..3378aea 100644 --- a/ravendb_embedded/provide.py +++ b/ravendb_embedded/provide.py @@ -56,21 +56,38 @@ def provide(self, target_directory): @staticmethod def unzip(source: Union[str, bytes], out: str) -> None: + if isinstance(source, bytes): + source = BytesIO(source) + + destination = Path(out).resolve() + + def is_inside_destination(name: str) -> bool: + resolved = (destination / name).resolve() + return resolved == destination or destination in resolved.parents + with zipfile.ZipFile(source, "r") as zipped: + for name in zipped.namelist(): + if not is_inside_destination(name): + raise RuntimeError(f"Refusing to extract unsafe archive path: {name}") zipped.extractall(out) class ExtractFromPkgResourceServerProvider(ProvideRavenDBServer): - def provide(self, target_directory): - resource_name = "ravendb_server.zip" + def __init__( + self, + package: str = "ravendb_embedded", + resource_name: str = "ravendb_server.zip", + ): + self.package = package + self.resource_name = resource_name - resource_data = pkgutil.get_data(self.__class__.__module__, resource_name) + def provide(self, target_directory): + resource_data = pkgutil.get_data(self.package, self.resource_name) if resource_data is None: - raise RuntimeError(f"Unable to find resource: {resource_name}") + raise RuntimeError(f"Unable to find resource '{self.resource_name}' in package '{self.package}'.") - with BytesIO(resource_data) as bytes_buffer: - ExtractFromZipServerProvider.unzip(bytes_buffer.read(), target_directory) + ExtractFromZipServerProvider.unzip(resource_data, target_directory) class ExternalServerProvider(ProvideRavenDBServer): diff --git a/ravendb_embedded/raven_server_runner.py b/ravendb_embedded/raven_server_runner.py index 30923df..6bd608e 100644 --- a/ravendb_embedded/raven_server_runner.py +++ b/ravendb_embedded/raven_server_runner.py @@ -1,9 +1,11 @@ import os +import re import subprocess from cryptography import x509 from cryptography.hazmat.backends import default_backend -from cryptography.hazmat.primitives import hashes +from cryptography.hazmat.primitives import hashes, serialization +from cryptography.x509.oid import ExtendedKeyUsageOID, ExtensionOID from ravendb.exceptions.raven_exceptions import RavenException from ravendb_embedded.options import ServerOptions @@ -13,8 +15,20 @@ class RavenServerRunner: + _CERTIFICATE_PATTERN = re.compile( + rb"-----BEGIN CERTIFICATE-----.*?-----END CERTIFICATE-----", + re.DOTALL, + ) + _PRIVATE_KEY_PATTERN = re.compile( + rb"-----BEGIN (?:RSA |EC |DSA |ENCRYPTED )?PRIVATE KEY-----.*?" + rb"-----END (?:RSA |EC |DSA |ENCRYPTED )?PRIVATE KEY-----", + re.DOTALL, + ) + @staticmethod def run(options: ServerOptions) -> subprocess.Popen: + client_certificate = RavenServerRunner._validate_security_options(options) + if not options.target_server_location.strip(): raise ValueError("target_server_location cannot be None or whitespace") @@ -53,13 +67,31 @@ def run(options: ServerOptions) -> subprocess.Popen: # Args are passed to Popen as a list (no shell), so they need no manual escaping. command_line_args = [ - f"--Embedded.ParentProcessId={RavenServerRunner.get_process_id('0')}", + f"--Embedded.ParentProcessId={RavenServerRunner.get_process_id()}", f"--License.Eula.Accepted={'true' if options.accept_eula else 'false'}", + f"--License.DisableAutoUpdate={'true' if options.licensing.disable_auto_update else 'false'}", + ( + "--License.DisableAutoUpdateFromApi=" + f"{'true' if options.licensing.disable_auto_update_from_api else 'false'}" + ), + ( + "--License.DisableLicenseSupportCheck=" + f"{'true' if options.licensing.disable_license_support_check else 'false'}" + ), + ( + "--License.ThrowOnInvalidOrMissingLicense=" + f"{'true' if options.licensing.throw_on_invalid_or_missing_license else 'false'}" + ), "--Setup.Mode=None", f"--DataDir={options.data_directory}", f"--Logs.Path={options.logs_path}", ] + if options.licensing.license is not None: + command_line_args.append(f"--License={options.licensing.license}") + if options.licensing.license_path is not None: + command_line_args.append(f"--License.Path={options.licensing.license_path}") + if options.security: options.server_url = options.server_url or "https://127.0.0.1:0" @@ -73,17 +105,12 @@ def run(options: ServerOptions) -> subprocess.Popen: elif options.security.certificate_exec: command_line_args.extend( [ - f"--Security.Certificate.Exec={options.security.certificate_exec}", - f"--Security.Certificate.Exec.Arguments={options.security.certificate_arguments}", + f"--Security.Certificate.Load.Exec={options.security.certificate_exec}", + f"--Security.Certificate.Load.Exec.Arguments={options.security.certificate_arguments}", ] ) - if options.security.client_pem_certificate_path: - with open(options.security.client_pem_certificate_path, "rb") as cert_file: - cert_data = cert_file.read() - - cert = x509.load_pem_x509_certificate(cert_data, default_backend()) - thumbprint = cert.fingerprint(hashes.SHA256()).hex() - + if client_certificate: + thumbprint = client_certificate.fingerprint(hashes.SHA1()).hex().upper() command_line_args.append(f"--Security.WellKnownCertificates.Admin={thumbprint}") else: options.server_url = options.server_url or "http://127.0.0.1:0" @@ -95,19 +122,98 @@ def run(options: ServerOptions) -> subprocess.Popen: if not is_sfa: command_line_args.insert(0, options.dot_net_path) - if options.framework_version: - framework_version = RuntimeFrameworkVersionMatcher.match(options) + requested_framework = options.framework_version + if requested_framework == RuntimeFrameworkVersionMatcher.AUTO: + requested_framework = RuntimeFrameworkVersionMatcher.required_framework_version(server_file_path) + + if requested_framework: + framework_version = RuntimeFrameworkVersionMatcher.match(options, requested_framework) command_line_args.insert(1, framework_version) command_line_args.insert(1, "--fx-version") - process_builder = subprocess.Popen(command_line_args, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + try: + process_builder = subprocess.Popen( + command_line_args, + stdin=subprocess.PIPE, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE, + ) + except FileNotFoundError as error: + if is_sfa: + raise + raise RuntimeError( + f"Unable to execute the .NET host '{options.dot_net_path}'. Install the required .NET runtime, " + "set ServerOptions.dot_net_path, or use with_auto_downloaded_server() to run without system .NET." + ) from error process = process_builder return process @staticmethod - def get_process_id(fallback: str) -> str: + def _validate_security_options(options: ServerOptions) -> x509.Certificate | None: + security = options.security + if security is None: + return None + + if security.ca_certificate_path: + ca_path = security.ca_certificate_path + try: + with open(ca_path, "rb") as ca_file: + ca_data = ca_file.read() + except OSError as error: + raise ValueError(f"Unable to read the CA certificate bundle '{ca_path}': {error}") from error + + ca_certificates = RavenServerRunner._CERTIFICATE_PATTERN.findall(ca_data) + if not ca_certificates: + raise ValueError(f"CA certificate bundle '{ca_path}' does not contain an X.509 certificate.") + try: + for ca_certificate in ca_certificates: + x509.load_pem_x509_certificate(ca_certificate, default_backend()) + except ValueError as error: + raise ValueError(f"CA certificate bundle '{ca_path}' contains an invalid certificate.") from error + + if not security.client_pem_certificate_path: + return None + + client_path = security.client_pem_certificate_path try: - return str(os.getpid()) - except Exception: - return fallback + with open(client_path, "rb") as client_file: + client_data = client_file.read() + except OSError as error: + raise ValueError(f"Unable to read the client PEM certificate '{client_path}': {error}") from error + + certificate_match = RavenServerRunner._CERTIFICATE_PATTERN.search(client_data) + if certificate_match is None: + raise ValueError(f"Client PEM '{client_path}' does not contain an X.509 certificate.") + try: + certificate = x509.load_pem_x509_certificate(certificate_match.group(), default_backend()) + except ValueError as error: + raise ValueError(f"Client PEM '{client_path}' contains an invalid X.509 certificate.") from error + + key_match = RavenServerRunner._PRIVATE_KEY_PATTERN.search(client_data) + if key_match is None: + raise ValueError(f"Client PEM '{client_path}' does not contain a private key.") + try: + private_key = serialization.load_pem_private_key(key_match.group(), password=None) + except (TypeError, ValueError) as error: + raise ValueError(f"Client PEM '{client_path}' must contain a valid, unencrypted private key.") from error + + public_format = serialization.PublicFormat.SubjectPublicKeyInfo + certificate_key = certificate.public_key().public_bytes(serialization.Encoding.DER, public_format) + private_key_public = private_key.public_key().public_bytes(serialization.Encoding.DER, public_format) + if certificate_key != private_key_public: + raise ValueError(f"The certificate and private key in client PEM '{client_path}' do not match.") + + try: + extended_key_usage = certificate.extensions.get_extension_for_oid(ExtensionOID.EXTENDED_KEY_USAGE).value + except x509.ExtensionNotFound: + pass + else: + if ExtendedKeyUsageOID.CLIENT_AUTH not in extended_key_usage: + raise ValueError(f"Client certificate '{client_path}' does not allow TLS client authentication.") + + return certificate + + @staticmethod + def get_process_id() -> str: + return str(os.getpid()) diff --git a/ravendb_embedded/runtime_framework_version_matcher.py b/ravendb_embedded/runtime_framework_version_matcher.py index d9fb98d..e2e0175 100644 --- a/ravendb_embedded/runtime_framework_version_matcher.py +++ b/ravendb_embedded/runtime_framework_version_matcher.py @@ -1,6 +1,8 @@ from __future__ import annotations +import json import subprocess from enum import Enum +from pathlib import Path from typing import Optional, Tuple, List from ravendb_embedded.options import ServerOptions @@ -12,15 +14,17 @@ class MatchingType(Enum): class RuntimeFrameworkVersionMatcher: + AUTO = "auto" WILDCARD = "x" GREATER_OR_EQUAL = "+" @classmethod - def match(cls, options: ServerOptions) -> str: - if not cls.needs_match(options): - return options.framework_version + def match(cls, options: ServerOptions, framework_version: str = None) -> str: + framework_version = options.framework_version if framework_version is None else framework_version + if not cls.needs_match(options, framework_version): + return framework_version - runtime = RuntimeFrameworkVersion(options.framework_version) + runtime = RuntimeFrameworkVersion(framework_version) runtimes = cls.get_framework_versions(options) return cls.match_runtime(runtime, runtimes) @@ -33,29 +37,45 @@ def match_runtime(runtime: RuntimeFrameworkVersion, runtimes: List[RuntimeFramew if runtime.match(version): return str(version) - available_runtimes = "- ".join([str(r) for r in sorted_runtimes]) + available_runtimes = "\n- ".join([str(r) for r in sorted_runtimes]) raise RuntimeError( f"Could not find a matching runtime for '{runtime}'. Available runtimes:\n- {available_runtimes}" ) @classmethod - def needs_match(cls, options: ServerOptions) -> bool: - if not options or not options.framework_version: + def needs_match(cls, options: ServerOptions, framework_version: str = None) -> bool: + framework_version = options.framework_version if framework_version is None else framework_version + if not options or not framework_version: return False - framework_version = options.framework_version.lower() + framework_version = framework_version.lower() if cls.WILDCARD not in framework_version and cls.GREATER_OR_EQUAL not in framework_version: return False return True + @classmethod + def required_framework_version(cls, server_file_path: str) -> str: + runtime_config = Path(server_file_path).with_suffix(".runtimeconfig.json") + try: + config = json.loads(runtime_config.read_text(encoding="utf-8")) + frameworks = config["runtimeOptions"]["frameworks"] + version = next(item["version"] for item in frameworks if item["name"] == "Microsoft.NETCore.App") + except (OSError, ValueError, KeyError, StopIteration, TypeError) as error: + raise RuntimeError( + f"Unable to determine the required .NET runtime from '{runtime_config}'. " + "Set ServerOptions.framework_version explicitly to override automatic detection." + ) from error + return f"{version}{cls.GREATER_OR_EQUAL}" + @staticmethod def get_framework_versions(options: ServerOptions): if not options.dot_net_path: raise RuntimeError("Dotnet path is not provided.") - process_command = [options.dot_net_path, "--info"] + process_command = [options.dot_net_path, "--list-runtimes"] runtimes = [] + process = None try: with subprocess.Popen( @@ -64,29 +84,24 @@ def get_framework_versions(options: ServerOptions): stderr=subprocess.PIPE, text=True, ) as process: - inside_runtimes = False - runtime_lines = [] - for line in process.stdout: - line = line.strip() - if line.startswith(".NET runtimes installed:") or line.startswith(".NET Core runtimes installed:"): - inside_runtimes = True + values = line.strip().split() + if not values or values[0] != "Microsoft.NETCore.App": continue - if inside_runtimes and line.startswith("Microsoft.NETCore.App"): - runtime_lines.append(line) - - for runtime_line in runtime_lines: - values = runtime_line.split(" ") if len(values) < 2: raise RuntimeError( - f"Invalid runtime line. Expected 'Microsoft.NETCore.App x.x.x', but was '{runtime_line}'" + f"Invalid runtime line. Expected 'Microsoft.NETCore.App x.x.x', but was '{line.strip()}'" ) runtimes.append(RuntimeFrameworkVersion(values[1])) except Exception as e: - raise RuntimeError("Unable to execute dotnet to retrieve list of installed runtimes") from e + raise RuntimeError( + f"Unable to execute '{options.dot_net_path}' to retrieve installed .NET runtimes. " + "Install the required .NET runtime, set ServerOptions.dot_net_path, " + "or use with_auto_downloaded_server() to run without system .NET." + ) from e finally: - if process: + if process is not None and process.poll() is None: process.kill() return runtimes diff --git a/setup.py b/setup.py index f366edb..9bcd14d 100644 --- a/setup.py +++ b/setup.py @@ -46,7 +46,7 @@ def run(self): include_package_data=True, long_description=open("README.md").read(), long_description_content_type="text/markdown", - version="7.2.5.post1", + version="7.2.5.post2", description="RavenDB Embedded library to run RavenDB in an embedded way", author="RavenDB", author_email="support@ravendb.net", diff --git a/tests/certificates.py b/tests/certificates.py index ef5c3d0..db86c03 100644 --- a/tests/certificates.py +++ b/tests/certificates.py @@ -50,6 +50,8 @@ def generate_self_signed_certificates(directory): x509.ExtendedKeyUsage([ExtendedKeyUsageOID.SERVER_AUTH, ExtendedKeyUsageOID.CLIENT_AUTH]), critical=False, ) + .add_extension(x509.SubjectKeyIdentifier.from_public_key(key.public_key()), critical=False) + .add_extension(x509.AuthorityKeyIdentifier.from_issuer_public_key(key.public_key()), critical=False) .sign(key, hashes.SHA256()) ) @@ -67,3 +69,101 @@ def generate_self_signed_certificates(directory): client_pem.write_bytes(key_pem + certificate_pem) ca_crt.write_bytes(certificate_pem) return str(server_pfx), str(client_pem), str(ca_crt) + + +def generate_separate_server_and_client_certificates(directory): + ca_key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + ca_name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "Embedded Test CA")]) + now = datetime.datetime.now(datetime.timezone.utc) + ca_certificate = ( + x509.CertificateBuilder() + .subject_name(ca_name) + .issuer_name(ca_name) + .public_key(ca_key.public_key()) + .serial_number(x509.random_serial_number()) + .not_valid_before(now - datetime.timedelta(days=1)) + .not_valid_after(now + datetime.timedelta(days=3650)) + .add_extension(x509.BasicConstraints(ca=True, path_length=None), critical=True) + .add_extension( + x509.KeyUsage( + digital_signature=True, + key_encipherment=False, + content_commitment=False, + data_encipherment=False, + key_agreement=False, + key_cert_sign=True, + crl_sign=True, + encipher_only=False, + decipher_only=False, + ), + critical=True, + ) + .add_extension(x509.SubjectKeyIdentifier.from_public_key(ca_key.public_key()), critical=False) + .add_extension(x509.AuthorityKeyIdentifier.from_issuer_public_key(ca_key.public_key()), critical=False) + .sign(ca_key, hashes.SHA256()) + ) + + def issue_certificate(common_name, extended_key_usage, subject_alternative_name=None): + key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + builder = ( + x509.CertificateBuilder() + .subject_name(x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, common_name)])) + .issuer_name(ca_name) + .public_key(key.public_key()) + .serial_number(x509.random_serial_number()) + .not_valid_before(now - datetime.timedelta(days=1)) + .not_valid_after(now + datetime.timedelta(days=3650)) + .add_extension( + x509.KeyUsage( + digital_signature=True, + key_encipherment=True, + content_commitment=False, + data_encipherment=False, + key_agreement=False, + key_cert_sign=False, + crl_sign=False, + encipher_only=False, + decipher_only=False, + ), + critical=True, + ) + .add_extension(x509.ExtendedKeyUsage(extended_key_usage), critical=False) + .add_extension(x509.SubjectKeyIdentifier.from_public_key(key.public_key()), critical=False) + .add_extension(x509.AuthorityKeyIdentifier.from_issuer_public_key(ca_key.public_key()), critical=False) + ) + if subject_alternative_name: + builder = builder.add_extension(subject_alternative_name, critical=False) + return key, builder.sign(ca_key, hashes.SHA256()) + + server_key, server_certificate = issue_certificate( + "localhost", + [ExtendedKeyUsageOID.SERVER_AUTH, ExtendedKeyUsageOID.CLIENT_AUTH], + x509.SubjectAlternativeName([x509.DNSName("localhost"), x509.IPAddress(ipaddress.ip_address("127.0.0.1"))]), + ) + client_key, client_certificate = issue_certificate( + "embedded-admin", + [ExtendedKeyUsageOID.CLIENT_AUTH], + ) + + server_pfx = Path(directory, "server.pfx") + client_pem = Path(directory, "client.pem") + ca_crt = Path(directory, "ca.crt") + server_pfx.write_bytes( + pkcs12.serialize_key_and_certificates( + b"localhost", + server_key, + server_certificate, + [ca_certificate], + serialization.NoEncryption(), + ) + ) + client_pem.write_bytes( + client_key.private_bytes( + serialization.Encoding.PEM, + serialization.PrivateFormat.TraditionalOpenSSL, + serialization.NoEncryption(), + ) + + client_certificate.public_bytes(serialization.Encoding.PEM) + ) + ca_crt.write_bytes(ca_certificate.public_bytes(serialization.Encoding.PEM)) + return str(server_pfx), str(client_pem), str(ca_crt) diff --git a/tests/test_basic.py b/tests/test_basic.py index 526eb08..5a18d0c 100644 --- a/tests/test_basic.py +++ b/tests/test_basic.py @@ -1,6 +1,8 @@ import shutil import tempfile +from concurrent.futures import ThreadPoolExecutor from pathlib import Path +from threading import Event, Thread from unittest import TestCase from ravendb_embedded import EmbeddedServer, ServerOptions, CopyServerFromNugetProvider, DatabaseOptions @@ -8,13 +10,121 @@ class BasicTest(TestCase): + def test_backward_compatible_defaults_are_runnable(self): + with tempfile.TemporaryDirectory() as temp_dir: + options = ServerOptions() + options.data_directory = str(Path(temp_dir, "RavenDB")) + + self.assertTrue(options.accept_eula) + self.assertEqual("", options.framework_version) + + with EmbeddedServer() as embedded: + embedded.start_server(options) + with embedded.get_document_store("Defaults") as store: + with store.open_session() as session: + session.store({"works": True}, "defaults/1") + session.save_changes() + + self.assertTrue(Path(temp_dir, "RavenDB").is_dir()) + self.assertTrue(Path(temp_dir, "RavenDB", "Logs").is_dir()) + + def test_close_waits_for_document_store_initialization(self): + initialization_started = Event() + continue_initialization = Event() + close_finished = Event() + results = {} + errors = [] + + class PausingEmbeddedServer(EmbeddedServer): + def _initialize_document_store(self, database_name, options): + initialization_started.set() + continue_initialization.wait() + return super()._initialize_document_store(database_name, options) + + with tempfile.TemporaryDirectory() as temp_dir: + embedded = PausingEmbeddedServer() + server_options = ServerOptions() + server_options.accept_eula = True + server_options.data_directory = str(Path(temp_dir, "RavenDB")) + server_options.logs_path = str(Path(temp_dir, "Logs")) + server_options.provider = CopyServerFromNugetProvider() + embedded.start_server(server_options) + + def get_store(): + try: + results["store"] = embedded.get_document_store("ConcurrentClose") + except Exception as error: + errors.append(error) + + def close_server(): + try: + embedded.close() + except Exception as error: + errors.append(error) + finally: + close_finished.set() + + get_thread = Thread(target=get_store, daemon=True) + close_thread = Thread(target=close_server, daemon=True) + get_thread.start() + self.assertTrue(initialization_started.wait(10)) + close_thread.start() + + self.assertFalse(close_finished.wait(0.2)) + continue_initialization.set() + get_thread.join(30) + close_thread.join(30) + + self.assertFalse(get_thread.is_alive()) + self.assertFalse(close_thread.is_alive()) + self.assertEqual([], errors) + self.assertTrue(results["store"].disposed) + + def test_concurrent_calls_share_one_document_store(self): + with tempfile.TemporaryDirectory() as temp_dir: + with EmbeddedServer() as embedded: + server_options = ServerOptions() + server_options.accept_eula = True + server_options.data_directory = str(Path(temp_dir, "RavenDB")) + server_options.logs_path = str(Path(temp_dir, "Logs")) + server_options.provider = CopyServerFromNugetProvider() + embedded.start_server(server_options) + + with ThreadPoolExecutor(max_workers=8) as executor: + stores = list(executor.map(lambda _: embedded.get_document_store("Shared"), range(8))) + + self.assertEqual(1, len({id(store) for store in stores})) + stores[0].close() + + def test_close_disposes_open_document_stores(self): + with tempfile.TemporaryDirectory() as temp_dir: + embedded = EmbeddedServer() + server_options = ServerOptions() + server_options.accept_eula = True + server_options.data_directory = str(Path(temp_dir, "RavenDB")) + server_options.logs_path = str(Path(temp_dir, "Logs")) + server_options.provider = CopyServerFromNugetProvider() + embedded.start_server(server_options) + + first = embedded.get_document_store("First") + second = embedded.get_document_store("Second") + + embedded.close() + + self.assertTrue(first.disposed) + self.assertTrue(second.disposed) + self.assertEqual({}, embedded.document_stores) + self.assertIsNone(embedded.server_task) + def test_embedded(self): temp_dir = tempfile.mkdtemp() try: with EmbeddedServer() as embedded: server_options = ServerOptions() + server_options.accept_eula = True server_options.data_directory = str(Path(temp_dir, "RavenDB")) server_options.logs_path = str(Path(temp_dir, "Logs")) + server_options.framework_version = "auto" server_options.provider = CopyServerFromNugetProvider() server_options.command_line_args = ["--Features.Availability=Experimental"] embedded.start_server(server_options) @@ -35,6 +145,7 @@ def test_embedded(self): with EmbeddedServer() as embedded: server_options = ServerOptions() + server_options.accept_eula = True server_options.data_directory = str(Path(temp_dir, "RavenDB")) server_options.provider = CopyServerFromNugetProvider() embedded.start_server(server_options) diff --git a/tests/test_custom_provider.py b/tests/test_custom_provider.py index bae95a7..169ccd6 100644 --- a/tests/test_custom_provider.py +++ b/tests/test_custom_provider.py @@ -1,17 +1,63 @@ +import io +import importlib import shutil +import sys import tempfile +import zipfile from pathlib import Path from unittest import TestCase from ravendb_embedded.embedded_server import EmbeddedServer from ravendb_embedded.options import ServerOptions, DatabaseOptions -from ravendb_embedded.provide import CopyServerFromNugetProvider +from ravendb_embedded.provide import ( + CopyServerFromNugetProvider, + ExtractFromPkgResourceServerProvider, + ExtractFromZipServerProvider, +) from tests import Person class TestCustomProvider(TestCase): + def test_can_extract_server_from_a_package_resource(self): + with tempfile.TemporaryDirectory() as temp_directory: + package_directory = Path(temp_directory, "example_server_package") + package_directory.mkdir() + Path(package_directory, "__init__.py").touch() + resource = Path(package_directory, "server.zip") + with zipfile.ZipFile(resource, "w") as zipped: + zipped.writestr("Server/Raven.Server.dll", "server") + + sys.path.insert(0, temp_directory) + importlib.invalidate_caches() + try: + destination = Path(temp_directory, "extracted") + ExtractFromPkgResourceServerProvider( + "example_server_package", + "server.zip", + ).provide(str(destination)) + self.assertTrue(Path(destination, "Server", "Raven.Server.dll").is_file()) + finally: + sys.modules.pop("example_server_package", None) + sys.path.remove(temp_directory) + importlib.invalidate_caches() + + def test_external_zip_rejects_path_traversal(self): + archive = io.BytesIO() + with zipfile.ZipFile(archive, "w") as zipped: + zipped.writestr("../escaped.txt", "unsafe") + + with tempfile.TemporaryDirectory() as temp_directory: + destination = Path(temp_directory, "server") + destination.mkdir() + + with self.assertRaisesRegex(RuntimeError, "unsafe archive path"): + ExtractFromZipServerProvider.unzip(archive.getvalue(), str(destination)) + + self.assertFalse(Path(temp_directory, "escaped.txt").exists()) + @staticmethod def configure_server_options(temp_dir: str, server_options: ServerOptions) -> ServerOptions: + server_options.accept_eula = True server_options.target_server_location = str(Path(temp_dir, "RavenDBServer")) server_options.data_directory = str(Path(temp_dir, "RavenDB")) server_options.logs_path = str(Path(temp_dir, "Logs")) @@ -55,10 +101,15 @@ def test_can_use_directory_as_external_server_source(self): self.assertIsNone(loaded_person) def test_can_use_default_nuget_provider(self): - with EmbeddedServer() as embedded: - database_options = DatabaseOptions.from_database_name("Test") - embedded.start_server() - with embedded.get_document_store_from_options(database_options) as store: - with store.open_session() as session: - loaded_person = session.load("no-such-person", Person) - self.assertIsNone(loaded_person) + with tempfile.TemporaryDirectory() as temp_directory: + with EmbeddedServer() as embedded: + options = ServerOptions() + options.accept_eula = True + options.data_directory = str(Path(temp_directory, "RavenDB")) + options.logs_path = str(Path(temp_directory, "Logs")) + database_options = DatabaseOptions.from_database_name("Test") + embedded.start_server(options) + with embedded.get_document_store_from_options(database_options) as store: + with store.open_session() as session: + loaded_person = session.load("no-such-person", Person) + self.assertIsNone(loaded_person) diff --git a/tests/test_diagnostics.py b/tests/test_diagnostics.py new file mode 100644 index 0000000..7143019 --- /dev/null +++ b/tests/test_diagnostics.py @@ -0,0 +1,21 @@ +import logging +from unittest import TestCase + +from ravendb_embedded import EmbeddedServer + + +class TestDiagnostics(TestCase): + def test_studio_url_disables_analytics(self): + self.assertEqual( + "http://127.0.0.1:8080/studio/index.html?disableAnalytics=true", + EmbeddedServer._get_studio_url("http://127.0.0.1:8080/"), + ) + + def test_embedded_logging_uses_python_logging_hierarchy(self): + server = EmbeddedServer() + + with self.assertLogs("ravendb_embedded.EmbeddedServer", logging.DEBUG) as captured: + server._log_debug("embedded diagnostic") + + self.assertEqual("ravendb_embedded.EmbeddedServer", server.logger.name) + self.assertIn("embedded diagnostic", captured.output[0]) diff --git a/tests/test_licensing.py b/tests/test_licensing.py new file mode 100644 index 0000000..4a7db0d --- /dev/null +++ b/tests/test_licensing.py @@ -0,0 +1,52 @@ +import json +import sys +import tempfile +from pathlib import Path +from unittest import TestCase + +from ravendb_embedded import EmbeddedServer, ServerOptions + + +class TestLicensing(TestCase): + def test_eula_acceptance_remains_enabled_by_default(self): + self.assertTrue(ServerOptions().accept_eula) + + def test_server_receives_all_licensing_options(self): + with tempfile.TemporaryDirectory() as directory: + server_directory = Path(directory, "Server") + server_directory.mkdir() + arguments_file = Path(directory, "arguments.json") + Path(server_directory, "Raven.Server.dll").write_text( + "import json\n" + "import pathlib\n" + "import sys\n" + f"pathlib.Path({str(arguments_file)!r}).write_text(json.dumps(sys.argv[1:]), encoding='utf-8')\n" + "print('Server available on: http://127.0.0.1:12345', flush=True)\n" + "sys.stdin.readline()\n", + encoding="utf-8", + ) + + options = ServerOptions() + options.with_external_server(str(server_directory)) + options.dot_net_path = sys.executable + options.framework_version = "" + options.data_directory = str(Path(directory, "data")) + options.logs_path = str(Path(directory, "logs")) + options.licensing.license = '{"Id":"test-license"}' + options.licensing.license_path = str(Path(directory, "license.json")) + options.licensing.disable_auto_update = True + options.licensing.disable_auto_update_from_api = True + options.licensing.disable_license_support_check = False + options.licensing.throw_on_invalid_or_missing_license = True + + with EmbeddedServer() as server: + server.start_server(options) + + arguments = json.loads(arguments_file.read_text(encoding="utf-8")) + self.assertIn("--License.Eula.Accepted=true", arguments) + self.assertIn("--License.DisableAutoUpdate=true", arguments) + self.assertIn("--License.DisableAutoUpdateFromApi=true", arguments) + self.assertIn("--License.DisableLicenseSupportCheck=false", arguments) + self.assertIn("--License.ThrowOnInvalidOrMissingLicense=true", arguments) + self.assertIn('--License={"Id":"test-license"}', arguments) + self.assertIn(f"--License.Path={options.licensing.license_path}", arguments) diff --git a/tests/test_lifecycle.py b/tests/test_lifecycle.py new file mode 100644 index 0000000..b03f971 --- /dev/null +++ b/tests/test_lifecycle.py @@ -0,0 +1,49 @@ +import tempfile +from pathlib import Path +from unittest import TestCase + +from ravendb_embedded import CopyServerFromNugetProvider, EmbeddedServer, ServerOptions + + +class TestLifecycle(TestCase): + def test_lifecycle_methods_require_a_started_server(self): + server = EmbeddedServer() + + for operation in ( + server.get_server_process_id, + server.stop_server, + server.restart_server, + ): + with self.subTest(operation=operation.__name__): + with self.assertRaisesRegex( + RuntimeError, + rf"Cannot call {operation.__name__}\(\) before calling start_server\(\)", + ): + operation() + + def test_stop_and_restart_real_server(self): + with tempfile.TemporaryDirectory() as directory: + options = ServerOptions() + options.accept_eula = True + options.data_directory = str(Path(directory, "RavenDB")) + options.logs_path = str(Path(directory, "Logs")) + options.provider = CopyServerFromNugetProvider() + + with EmbeddedServer() as server: + server.start_server(options) + first_process = server.server_task.get_value()[1] + first_pid = server.get_server_process_id() + + server.stop_server() + + self.assertEqual(first_pid, first_process.pid) + self.assertIsNotNone(first_process.poll()) + + server.restart_server() + second_pid = server.get_server_process_id() + + self.assertNotEqual(first_pid, second_pid) + with server.get_document_store("Lifecycle") as store: + with store.open_session() as session: + session.store({"Status": "restarted"}, "lifecycle/1") + session.save_changes() diff --git a/tests/test_multiple_servers.py b/tests/test_multiple_servers.py new file mode 100644 index 0000000..1a31b4c --- /dev/null +++ b/tests/test_multiple_servers.py @@ -0,0 +1,42 @@ +import tempfile +from pathlib import Path +from unittest import TestCase + +from ravendb_embedded import EmbeddedServer, ServerOptions + + +class TestMultipleServers(TestCase): + def test_independent_servers_can_run_in_the_same_process(self): + with tempfile.TemporaryDirectory() as directory: + first_options = self._options(directory, "first") + second_options = self._options(directory, "second") + + with EmbeddedServer() as first_server, EmbeddedServer() as second_server: + first_server.start_server(first_options) + second_server.start_server(second_options) + + self.assertNotEqual(first_server.get_server_process_id(), second_server.get_server_process_id()) + self.assertNotEqual(first_server.get_server_uri(), second_server.get_server_uri()) + + with first_server.get_document_store("SharedName") as first_store: + with first_store.open_session() as session: + session.store({"server": "first"}, "markers/1") + session.save_changes() + + with second_server.get_document_store("SharedName") as second_store: + with second_store.open_session() as session: + self.assertIsNone(session.load("markers/1", dict)) + session.store({"server": "second"}, "markers/1") + session.save_changes() + + with first_server.get_document_store("SharedName") as first_store: + with first_store.open_session() as session: + self.assertEqual("first", session.load("markers/1", dict)["server"]) + + @staticmethod + def _options(root: str, name: str) -> ServerOptions: + options = ServerOptions() + options.accept_eula = True + options.data_directory = str(Path(root, name, "data")) + options.logs_path = str(Path(root, name, "logs")) + return options diff --git a/tests/test_on_demand.py b/tests/test_on_demand.py index ec26ff1..5de0157 100644 --- a/tests/test_on_demand.py +++ b/tests/test_on_demand.py @@ -1,16 +1,88 @@ import io import tarfile import tempfile +import threading +import time import unittest import zipfile +from concurrent.futures import ThreadPoolExecutor +from contextlib import contextmanager +from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer from pathlib import Path +from ravendb_embedded import on_demand from ravendb_embedded.on_demand import _extract_safely, _platform_download, _platform_download_for, ensure_server +def _server_archive(extension): + archive = io.BytesIO() + payload = b"real cache payload" * 4096 + path = "RavenDB/Server/Raven.Server.dll" + + if extension == "zip": + with zipfile.ZipFile(archive, "w") as bundle: + bundle.writestr(path, payload) + else: + with tarfile.open(fileobj=archive, mode="w:bz2") as bundle: + member = tarfile.TarInfo(path) + member.size = len(payload) + bundle.addfile(member, io.BytesIO(payload)) + + return archive.getvalue() + + +@contextmanager +def _serve_archive(payload, interrupt=False, pause=False): + class ArchiveHandler(BaseHTTPRequestHandler): + request_count = 0 + request_count_lock = threading.Lock() + + def do_GET(self): + with self.request_count_lock: + type(self).request_count += 1 + + self.send_response(200) + self.send_header("Content-Type", "application/octet-stream") + self.send_header("Content-Length", str(len(payload))) + self.end_headers() + + midpoint = max(1, len(payload) // 2) + self.wfile.write(payload[:midpoint]) + self.wfile.flush() + if interrupt: + self.close_connection = True + return + if pause: + time.sleep(0.1) + self.wfile.write(payload[midpoint:]) + + def log_message(self, _format, *_args): + pass + + server = ThreadingHTTPServer(("127.0.0.1", 0), ArchiveHandler) + server_thread = threading.Thread(target=server.serve_forever, daemon=True) + server_thread.start() + try: + yield f"http://127.0.0.1:{server.server_port}", ArchiveHandler + finally: + server.shutdown() + server.server_close() + server_thread.join() + + +@contextmanager +def _download_from(base_url): + original = on_demand._DOWNLOAD_BASE + on_demand._DOWNLOAD_BASE = base_url + try: + yield + finally: + on_demand._DOWNLOAD_BASE = original + + class TestOnDemand(unittest.TestCase): def test_cache_hit_skips_download(self): - # A populated cache must be reused without any network call ("by design"). + # A populated cache must be reused without any network call. with tempfile.TemporaryDirectory() as cache_root: label, _ = _platform_download() server_dir = Path(cache_root, "7.2", label.replace(" ", "_"), "Server") @@ -71,3 +143,49 @@ def test_extract_rejects_path_traversal(self): _extract_safely(bad_zip, dest, "zip") self.assertFalse((Path(work) / "escaped.txt").exists()) + + def test_interrupted_download_does_not_poison_cache(self): + label, extension = _platform_download() + payload = _server_archive(extension) + + with tempfile.TemporaryDirectory() as cache_root: + target = Path(cache_root, "interrupted", label.replace(" ", "_")) + + with _serve_archive(payload, interrupt=True) as (base_url, _): + with _download_from(base_url): + with self.assertRaises((tarfile.ReadError, zipfile.BadZipFile)): + ensure_server(version="interrupted", cache_root=cache_root) + + self.assertFalse(target.exists()) + self.assertEqual([], list(target.parent.glob("download-*"))) + + with _serve_archive(payload) as (base_url, _): + with _download_from(base_url): + resolved = Path(ensure_server(version="interrupted", cache_root=cache_root)) + + self.assertEqual(target / "RavenDB" / "Server", resolved) + self.assertTrue((resolved / "Raven.Server.dll").is_file()) + + def test_concurrent_first_downloads_leave_one_complete_cache(self): + label, extension = _platform_download() + payload = _server_archive(extension) + workers = 4 + start = threading.Barrier(workers) + + with tempfile.TemporaryDirectory() as cache_root: + with _serve_archive(payload, pause=True) as (base_url, handler): + with _download_from(base_url): + + def resolve(): + start.wait() + return ensure_server(version="concurrent", cache_root=cache_root) + + with ThreadPoolExecutor(max_workers=workers) as executor: + resolved = list(executor.map(lambda _: resolve(), range(workers))) + + target = Path(cache_root, "concurrent", label.replace(" ", "_")) + expected = target / "RavenDB" / "Server" + self.assertEqual([str(expected)] * workers, resolved) + self.assertTrue((expected / "Raven.Server.dll").is_file()) + self.assertGreaterEqual(handler.request_count, 2) + self.assertEqual([], list(target.parent.glob("download-*"))) diff --git a/tests/test_options.py b/tests/test_options.py new file mode 100644 index 0000000..9ea7c7a --- /dev/null +++ b/tests/test_options.py @@ -0,0 +1,30 @@ +from pathlib import Path +from unittest import TestCase + +from ravendb_embedded import ServerOptions + + +class TestServerOptions(TestCase): + def test_default_data_and_logs_remain_under_the_package_directory(self): + options = ServerOptions() + + self.assertEqual(ServerOptions.BASE_MODULE_DIRECTORY + "/RavenDB", options.data_directory) + self.assertEqual(ServerOptions.BASE_MODULE_DIRECTORY + "/RavenDB/Logs", options.logs_path) + + def test_default_logs_follow_an_explicit_data_directory(self): + options = ServerOptions() + options.data_directory = "custom-data" + self.assertEqual(str(Path("custom-data", "Logs")), options.logs_path) + + options.logs_path = "custom-logs" + options.data_directory = "other-data" + self.assertEqual("custom-logs", options.logs_path) + + def test_instance_is_a_deprecated_constructor_alias(self): + with self.assertWarnsRegex( + DeprecationWarning, + r"construct ServerOptions\(\) directly", + ): + options = ServerOptions.INSTANCE() + + self.assertIsInstance(options, ServerOptions) diff --git a/tests/test_process_exit.py b/tests/test_process_exit.py new file mode 100644 index 0000000..74080ac --- /dev/null +++ b/tests/test_process_exit.py @@ -0,0 +1,76 @@ +import sys +import tempfile +from pathlib import Path +from threading import Event +from unittest import TestCase + +from ravendb_embedded import EmbeddedServer, ServerOptions + + +class TestProcessExit(TestCase): + @staticmethod + def server_options(directory, script): + server_directory = Path(directory, "Server") + server_directory.mkdir() + Path(server_directory, "Raven.Server.dll").write_text(script, encoding="utf-8") + + options = ServerOptions() + options.accept_eula = True + options.with_external_server(str(server_directory)) + options.dot_net_path = sys.executable + options.framework_version = "" + options.data_directory = str(Path(directory, "data")) + options.logs_path = str(Path(directory, "logs")) + return options + + def test_reports_unexpected_process_exit(self): + with tempfile.TemporaryDirectory() as directory: + options = self.server_options( + directory, + "import sys\n" + "print('Server available on: http://127.0.0.1:12345', flush=True)\n" + "raise SystemExit(23)\n", + ) + callback_finished = Event() + events = [] + + with EmbeddedServer() as server: + + def on_exit(event): + events.append(event) + callback_finished.set() + + server.add_server_process_exited(on_exit) + server.start_server(options) + process_id = server.get_server_process_id() + + self.assertTrue(callback_finished.wait(10)) + + self.assertEqual(1, len(events)) + self.assertEqual(process_id, events[0].process_id) + self.assertEqual(23, events[0].exit_code) + self.assertFalse(events[0].expected) + + def test_reports_expected_process_exit(self): + with tempfile.TemporaryDirectory() as directory: + options = self.server_options( + directory, + "import sys\n" + "print('Server available on: http://127.0.0.1:12345', flush=True)\n" + "sys.stdin.readline()\n", + ) + callback_finished = Event() + events = [] + + with EmbeddedServer() as server: + server.add_server_process_exited(lambda event: (events.append(event), callback_finished.set())) + server.start_server(options) + process_id = server.get_server_process_id() + server.stop_server() + + self.assertTrue(callback_finished.wait(10)) + + self.assertEqual(1, len(events)) + self.assertEqual(process_id, events[0].process_id) + self.assertEqual(0, events[0].exit_code) + self.assertTrue(events[0].expected) diff --git a/tests/test_raven_server_runner.py b/tests/test_raven_server_runner.py new file mode 100644 index 0000000..0b84881 --- /dev/null +++ b/tests/test_raven_server_runner.py @@ -0,0 +1,9 @@ +import os +from unittest import TestCase + +from ravendb_embedded.raven_server_runner import RavenServerRunner + + +class TestRavenServerRunner(TestCase): + def test_parent_process_id_is_the_current_python_process(self): + self.assertEqual(str(os.getpid()), RavenServerRunner.get_process_id()) diff --git a/tests/test_runtime_framework_version_matcher.py b/tests/test_runtime_framework_version_matcher.py index d0f0a58..82066df 100644 --- a/tests/test_runtime_framework_version_matcher.py +++ b/tests/test_runtime_framework_version_matcher.py @@ -1,3 +1,5 @@ +import tempfile +from pathlib import Path from unittest import TestCase from ravendb_embedded.options import ServerOptions @@ -9,8 +11,7 @@ class TestRuntimeFrameworkVersionMatcher(TestCase): def test_match_1(self): - # Default framework version is unenforced (empty); match() returns it unchanged. - self.assertFalse(ServerOptions.INSTANCE().framework_version) + self.assertEqual("", ServerOptions().framework_version) options = ServerOptions() @@ -78,6 +79,7 @@ def test_match_4(self): "Could not find a matching runtime for '3.1.4+'. Available runtimes:", str(context.exception), ) + self.assertIn("\n- 5.0.4\n- 5.0.3\n- 5.0.0-rc.2.20475.17", str(context.exception)) with self.assertRaises(RuntimeError) as context: RuntimeFrameworkVersion("6.0.0+-preview.6.21352.12") @@ -100,6 +102,39 @@ def test_match_4(self): str(context.exception), ) + def test_missing_dotnet_preserves_execution_error(self): + with tempfile.TemporaryDirectory() as directory: + options = ServerOptions() + options.dot_net_path = str(Path(directory, "missing-dotnet")) + + with self.assertRaises(RuntimeError) as context: + RuntimeFrameworkVersionMatcher.get_framework_versions(options) + + self.assertEqual( + f"Unable to execute '{options.dot_net_path}' to retrieve installed .NET runtimes. " + "Install the required .NET runtime, set ServerOptions.dot_net_path, " + "or use with_auto_downloaded_server() to run without system .NET.", + str(context.exception), + ) + self.assertIsInstance(context.exception.__cause__, OSError) + + def test_reads_required_runtime_from_server_config(self): + with tempfile.TemporaryDirectory() as directory: + server = Path(directory, "Raven.Server.dll") + server.touch() + Path(directory, "Raven.Server.runtimeconfig.json").write_text( + '{"runtimeOptions":{"frameworks":[' + '{"name":"Microsoft.NETCore.App","version":"10.0.9"},' + '{"name":"Microsoft.AspNetCore.App","version":"10.0.9"}' + "]}}", + encoding="utf-8", + ) + + self.assertEqual( + "10.0.9+", + RuntimeFrameworkVersionMatcher.required_framework_version(str(server)), + ) + def get_runtimes(self): result = [ RuntimeFrameworkVersion("2.1.3"), diff --git a/tests/test_secured_basic.py b/tests/test_secured_basic.py index cf8bcf7..8d31270 100644 --- a/tests/test_secured_basic.py +++ b/tests/test_secured_basic.py @@ -1,4 +1,5 @@ import shutil +import sys import tempfile from pathlib import Path from unittest import TestCase @@ -7,16 +8,31 @@ from ravendb_embedded.options import ServerOptions, DatabaseOptions from ravendb_embedded.provide import CopyServerFromNugetProvider from tests import Person -from tests.certificates import generate_self_signed_certificates +from tests.certificates import generate_self_signed_certificates, generate_separate_server_and_client_certificates class TestSecuredBasic(TestCase): + def test_secured_embedded_server_starts_without_a_managed_client_certificate(self): + with tempfile.TemporaryDirectory() as temp_dir: + server_pfx, _, _ = generate_self_signed_certificates(temp_dir) + with EmbeddedServer() as embedded: + server_options = ServerOptions() + server_options.secured(server_pfx) + server_options.data_directory = str(Path(temp_dir, "RavenDB")) + server_options.logs_path = str(Path(temp_dir, "Logs")) + server_options.provider = CopyServerFromNugetProvider() + + embedded.start_server(server_options) + + self.assertTrue(embedded.get_server_uri().startswith("https://")) + def test_secured_embedded(self): temp_dir = tempfile.mkdtemp() try: server_pfx, client_pem, ca_crt = generate_self_signed_certificates(temp_dir) with EmbeddedServer() as embedded: server_options = ServerOptions() + server_options.accept_eula = True server_options.secured(server_pfx, client_pem, ca_certificate_path=ca_crt) server_options.data_directory = str(Path(temp_dir, "RavenDB")) server_options.logs_path = str(Path(temp_dir, "Logs")) @@ -34,3 +50,52 @@ def test_secured_embedded(self): session.save_changes() finally: shutil.rmtree(temp_dir) + + def test_secured_embedded_with_separate_admin_certificate(self): + with tempfile.TemporaryDirectory() as temp_dir: + server_pfx, client_pem, ca_crt = generate_separate_server_and_client_certificates(temp_dir) + with EmbeddedServer() as embedded: + server_options = ServerOptions() + server_options.accept_eula = True + server_options.secured(server_pfx, client_pem, ca_certificate_path=ca_crt) + server_options.data_directory = str(Path(temp_dir, "RavenDB")) + server_options.logs_path = str(Path(temp_dir, "Logs")) + server_options.provider = CopyServerFromNugetProvider() + embedded.start_server(server_options) + + with embedded.get_document_store("Test") as store: + with store.open_session() as session: + person = Person() + person.name = "Separate admin certificate" + session.store(person, "people/1") + session.save_changes() + + def test_secured_embedded_with_certificate_exec(self): + with tempfile.TemporaryDirectory() as temp_dir: + server_pfx, client_pem, ca_crt = generate_separate_server_and_client_certificates(temp_dir) + certificate_loader = Path(temp_dir, "load_certificate.py") + certificate_loader.write_text( + "import pathlib\n" "import sys\n" "sys.stdout.buffer.write(pathlib.Path(sys.argv[1]).read_bytes())\n", + encoding="utf-8", + ) + + with EmbeddedServer() as embedded: + server_options = ServerOptions() + server_options.accept_eula = True + server_options.secured_with_certificate_exec( + sys.executable, + f'"{certificate_loader}" "{server_pfx}"', + client_pem, + ca_certificate_path=ca_crt, + ) + server_options.data_directory = str(Path(temp_dir, "RavenDB")) + server_options.logs_path = str(Path(temp_dir, "Logs")) + server_options.provider = CopyServerFromNugetProvider() + embedded.start_server(server_options) + + with embedded.get_document_store("Test") as store: + with store.open_session() as session: + person = Person() + person.name = "Certificate loader" + session.store(person, "people/1") + session.save_changes() diff --git a/tests/test_security_validation.py b/tests/test_security_validation.py new file mode 100644 index 0000000..4255f0d --- /dev/null +++ b/tests/test_security_validation.py @@ -0,0 +1,134 @@ +import datetime +import tempfile +from pathlib import Path +from unittest import TestCase + +from cryptography import x509 +from cryptography.hazmat.primitives import hashes, serialization +from cryptography.hazmat.primitives.asymmetric import rsa +from cryptography.hazmat.primitives.serialization import pkcs12 +from cryptography.x509.oid import ExtendedKeyUsageOID, ExtensionOID, NameOID + +from ravendb_embedded import ServerOptions +from ravendb_embedded.raven_server_runner import RavenServerRunner +from tests.certificates import generate_separate_server_and_client_certificates + + +class TestSecurityValidation(TestCase): + def test_generated_certificate_chain_has_matching_key_identifiers(self): + with tempfile.TemporaryDirectory() as directory: + server_pfx, client_pem, _ = generate_separate_server_and_client_certificates(directory) + _, server_certificate, ca_certificates = pkcs12.load_key_and_certificates( + Path(server_pfx).read_bytes(), None + ) + ca_certificate = ca_certificates[0] + client_pem_bytes = Path(client_pem).read_bytes() + client_certificate = x509.load_pem_x509_certificate( + client_pem_bytes[client_pem_bytes.index(b"-----BEGIN CERTIFICATE-----") :] + ) + + ca_key_identifier = ca_certificate.extensions.get_extension_for_oid( + ExtensionOID.SUBJECT_KEY_IDENTIFIER + ).value.digest + for certificate in (server_certificate, client_certificate): + authority_key_identifier = certificate.extensions.get_extension_for_oid( + ExtensionOID.AUTHORITY_KEY_IDENTIFIER + ).value.key_identifier + self.assertEqual(authority_key_identifier, ca_key_identifier) + + def test_secured_allows_server_only_configuration(self): + options = ServerOptions().secured("server.pfx") + + self.assertEqual(options.security.server_pfx_certificate_path, "server.pfx") + self.assertIsNone(options.security.client_pem_certificate_path) + + def test_client_pem_must_contain_a_private_key(self): + with tempfile.TemporaryDirectory() as directory: + _, certificate = self._create_certificate([ExtendedKeyUsageOID.CLIENT_AUTH]) + client_pem = Path(directory, "client.pem") + client_pem.write_bytes(certificate.public_bytes(serialization.Encoding.PEM)) + + options = self._secured_options(client_pem) + with self.assertRaisesRegex(ValueError, "does not contain a private key"): + RavenServerRunner.run(options) + + def test_client_certificate_and_private_key_must_match(self): + with tempfile.TemporaryDirectory() as directory: + first_key, _ = self._create_certificate([ExtendedKeyUsageOID.CLIENT_AUTH]) + _, second_certificate = self._create_certificate([ExtendedKeyUsageOID.CLIENT_AUTH]) + client_pem = Path(directory, "client.pem") + client_pem.write_bytes( + self._private_key_pem(first_key) + second_certificate.public_bytes(serialization.Encoding.PEM) + ) + + options = self._secured_options(client_pem) + with self.assertRaisesRegex(ValueError, "do not match"): + RavenServerRunner.run(options) + + def test_declared_eku_must_allow_client_authentication(self): + with tempfile.TemporaryDirectory() as directory: + key, certificate = self._create_certificate([ExtendedKeyUsageOID.SERVER_AUTH]) + client_pem = Path(directory, "client.pem") + client_pem.write_bytes(self._private_key_pem(key) + certificate.public_bytes(serialization.Encoding.PEM)) + + options = self._secured_options(client_pem) + with self.assertRaisesRegex(ValueError, "does not allow TLS client authentication"): + RavenServerRunner.run(options) + + def test_ca_path_must_contain_a_certificate(self): + with tempfile.TemporaryDirectory() as directory: + key, certificate = self._create_certificate([ExtendedKeyUsageOID.CLIENT_AUTH]) + client_pem = Path(directory, "client.pem") + client_pem.write_bytes(self._private_key_pem(key) + certificate.public_bytes(serialization.Encoding.PEM)) + ca_path = Path(directory, "ca.crt") + ca_path.write_text("not a certificate", encoding="utf-8") + + options = self._secured_options(client_pem, ca_path) + with self.assertRaisesRegex(ValueError, "does not contain an X.509 certificate"): + RavenServerRunner.run(options) + + def test_ca_is_validated_without_a_managed_client_certificate(self): + with tempfile.TemporaryDirectory() as directory: + ca_path = Path(directory, "ca.crt") + ca_path.write_text("not a certificate", encoding="utf-8") + options = ServerOptions().secured("server.pfx", ca_certificate_path=str(ca_path)) + + with self.assertRaisesRegex(ValueError, "does not contain an X.509 certificate"): + RavenServerRunner.run(options) + + @staticmethod + def _secured_options(client_pem: Path, ca_path: Path = None) -> ServerOptions: + options = ServerOptions() + options.accept_eula = True + options.secured( + "server.pfx", + str(client_pem), + ca_certificate_path=str(ca_path) if ca_path else None, + ) + return options + + @staticmethod + def _private_key_pem(key) -> bytes: + return key.private_bytes( + serialization.Encoding.PEM, + serialization.PrivateFormat.TraditionalOpenSSL, + serialization.NoEncryption(), + ) + + @staticmethod + def _create_certificate(extended_key_usage): + key = rsa.generate_private_key(public_exponent=65537, key_size=2048) + name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, "embedded-client")]) + now = datetime.datetime.now(datetime.timezone.utc) + certificate = ( + x509.CertificateBuilder() + .subject_name(name) + .issuer_name(name) + .public_key(key.public_key()) + .serial_number(x509.random_serial_number()) + .not_valid_before(now - datetime.timedelta(days=1)) + .not_valid_after(now + datetime.timedelta(days=1)) + .add_extension(x509.ExtendedKeyUsage(extended_key_usage), critical=False) + .sign(key, hashes.SHA256()) + ) + return key, certificate diff --git a/tests/test_shutdown.py b/tests/test_shutdown.py new file mode 100644 index 0000000..5f5fafb --- /dev/null +++ b/tests/test_shutdown.py @@ -0,0 +1,85 @@ +import sys +import tempfile +from concurrent.futures import ThreadPoolExecutor +from pathlib import Path +from threading import Barrier +from unittest import TestCase + +from ravendb_embedded import EmbeddedServer, ServerOptions + + +class TestShutdown(TestCase): + def test_concurrent_start_launches_one_server(self): + with tempfile.TemporaryDirectory() as directory: + server_directory = Path(directory, "Server") + server_directory.mkdir() + start_marker = Path(directory, "starts.txt") + Path(server_directory, "Raven.Server.dll").write_text( + "import pathlib\n" + "import sys\n" + f"with pathlib.Path({str(start_marker)!r}).open('a', encoding='utf-8') as marker:\n" + " marker.write('started\\n')\n" + "print('Server available on: http://127.0.0.1:12345', flush=True)\n" + "sys.stdin.readline()\n", + encoding="utf-8", + ) + + options = ServerOptions() + options.accept_eula = True + options.with_external_server(str(server_directory)) + options.dot_net_path = sys.executable + options.framework_version = "" + options.data_directory = str(Path(directory, "data")) + options.logs_path = str(Path(directory, "logs")) + server = EmbeddedServer() + barrier = Barrier(8) + + def start(): + barrier.wait() + try: + server.start_server(options) + return "started" + except RuntimeError as error: + return str(error) + + with ThreadPoolExecutor(max_workers=8) as executor: + results = list(executor.map(lambda _: start(), range(8))) + + server.close() + + self.assertEqual(1, results.count("started")) + self.assertEqual(7, results.count("The server was already started")) + self.assertEqual(["started"], start_marker.read_text(encoding="utf-8").splitlines()) + + def test_close_sends_graceful_shutdown_command(self): + with tempfile.TemporaryDirectory() as directory: + server_directory = Path(directory, "Server") + server_directory.mkdir() + shutdown_marker = Path(directory, "graceful-shutdown.txt") + Path(server_directory, "Raven.Server.dll").write_text( + "import pathlib\n" + "import sys\n" + "print('Server available on: http://127.0.0.1:12345', flush=True)\n" + "command = sys.stdin.readline().strip()\n" + f"pathlib.Path({str(shutdown_marker)!r}).write_text(command, encoding='utf-8')\n", + encoding="utf-8", + ) + + options = ServerOptions() + options.accept_eula = True + options.with_external_server(str(server_directory)) + options.dot_net_path = sys.executable + options.framework_version = "" + options.data_directory = str(Path(directory, "data")) + options.logs_path = str(Path(directory, "logs")) + + server = EmbeddedServer() + server.start_server(options) + process = server.server_task.get_value()[1] + server.close() + + self.assertEqual("shutdown no-confirmation", shutdown_marker.read_text(encoding="utf-8")) + self.assertTrue(process.stdin.closed) + self.assertTrue(process.stdout.closed) + self.assertTrue(process.stderr.closed) + self.assertIsNone(server._exit_handler) diff --git a/tests/test_startup_errors.py b/tests/test_startup_errors.py new file mode 100644 index 0000000..c740a62 --- /dev/null +++ b/tests/test_startup_errors.py @@ -0,0 +1,134 @@ +import sys +import tempfile +from datetime import timedelta +from pathlib import Path +from unittest import TestCase + +from ravendb_embedded import EmbeddedServer, ServerOptions, ServerStartupError, ServerStartupTimeoutError + + +class TestStartupErrors(TestCase): + def test_failed_start_can_be_retried_on_the_same_server(self): + with tempfile.TemporaryDirectory() as directory: + server_directory = Path(directory, "Server") + server_directory.mkdir() + first_attempt = Path(directory, "first-attempt") + Path(server_directory, "Raven.Server.dll").write_text( + "import pathlib\n" + "import sys\n" + f"marker = pathlib.Path({str(first_attempt)!r})\n" + "if not marker.exists():\n" + " marker.touch()\n" + " print('intentional first-start failure', file=sys.stderr, flush=True)\n" + " raise SystemExit(1)\n" + "print('Server available on: http://127.0.0.1:12345', flush=True)\n" + "sys.stdin.readline()\n", + encoding="utf-8", + ) + + options = ServerOptions() + options.accept_eula = True + options.with_external_server(str(server_directory)) + options.dot_net_path = sys.executable + options.framework_version = "" + options.data_directory = str(Path(directory, "data")) + options.logs_path = str(Path(directory, "logs")) + + with EmbeddedServer() as server: + with self.assertRaises(ServerStartupError): + server.start_server(options) + + server.start_server(options) + self.assertEqual("http://127.0.0.1:12345", server.get_server_uri()) + + def test_startup_timeout_has_a_distinct_exception(self): + with tempfile.TemporaryDirectory() as directory: + server_directory = Path(directory, "Server") + server_directory.mkdir() + Path(server_directory, "Raven.Server.dll").write_text( + "import time\n" "time.sleep(60)\n", + encoding="utf-8", + ) + + options = ServerOptions() + options.accept_eula = True + options.with_external_server(str(server_directory)) + options.dot_net_path = sys.executable + options.framework_version = "" + options.data_directory = str(Path(directory, "data")) + options.logs_path = str(Path(directory, "logs")) + options.max_server_startup_time_duration = timedelta(milliseconds=200) + options.graceful_shutdown_timeout = timedelta(milliseconds=100) + options.process_kill_timeout = timedelta(seconds=1) + + with EmbeddedServer() as server: + with self.assertRaises(ServerStartupTimeoutError) as context: + server.start_server(options) + + self.assertIn("Server failed to start in 0.2 seconds.", str(context.exception)) + + def test_stderr_is_drained_before_and_after_server_is_online(self): + with tempfile.TemporaryDirectory() as directory: + server_directory = Path(directory, "Server") + server_directory.mkdir() + shutdown_marker = Path(directory, "shutdown.txt") + Path(server_directory, "Raven.Server.dll").write_text( + "import pathlib\n" + "import sys\n" + "payload = 'x' * (1024 * 1024)\n" + "sys.stderr.write(payload)\n" + "sys.stderr.flush()\n" + "print('Server available on: http://127.0.0.1:12345', flush=True)\n" + "sys.stderr.write(payload)\n" + "sys.stderr.flush()\n" + "command = sys.stdin.readline().strip()\n" + f"pathlib.Path({str(shutdown_marker)!r}).write_text(command, encoding='utf-8')\n", + encoding="utf-8", + ) + + options = ServerOptions() + options.accept_eula = True + options.with_external_server(str(server_directory)) + options.dot_net_path = sys.executable + options.framework_version = "" + options.data_directory = str(Path(directory, "data")) + options.logs_path = str(Path(directory, "logs")) + options.max_server_startup_time_duration = timedelta(seconds=10) + + with EmbeddedServer() as server: + server.start_server(options) + + self.assertEqual("shutdown no-confirmation", shutdown_marker.read_text(encoding="utf-8")) + + def test_failed_server_process_includes_stderr(self): + with tempfile.TemporaryDirectory() as directory: + server_directory = Path(directory, "Server") + server_directory.mkdir() + Path(server_directory, "Raven.Server.dll").write_text( + "import sys\n" + "sys.stdout.write('intentional stdout before failure\\n')\n" + "sys.stdout.flush()\n" + "sys.stderr.write('intentional startup failure from child process\\n')\n" + "raise SystemExit(23)\n", + encoding="utf-8", + ) + + options = ServerOptions() + options.accept_eula = True + options.with_external_server(str(server_directory)) + options.dot_net_path = sys.executable + options.framework_version = "" + options.data_directory = str(Path(directory, "data")) + options.logs_path = str(Path(directory, "logs")) + options.max_server_startup_time_duration = timedelta(seconds=10) + + with EmbeddedServer() as server: + with self.assertRaises(ServerStartupError) as context: + server.start_server(options) + + message = str(context.exception) + self.assertIn("Unable to start the RavenDB Server", message) + self.assertIn("Error:", message) + self.assertIn("intentional startup failure from child process", message) + self.assertIn("Output:", message) + self.assertIn("intentional stdout before failure", message)