Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,7 +99,8 @@ phased delivery plan; new work should move the code toward it, not entrench the
- Collector drivers subclass `CollectorDriver` (`worker/src/devicevault_worker/drivers/base.py`)
with a pydantic `Params` model, and raise `TransientError` (retried) or `PermanentError`.
SSH CLI platforms subclass `drivers.ssh.NetmikoDriver` (a Netmiko `device_type`, the
commands, and regexes for volatile lines); see `drivers/cisco.py`. Built-ins are listed in
commands, and regexes for volatile lines); see `drivers/cisco.py`. A driver that connects over SSH must
require `target.host_key` when it is set and report the key it saw in `log.host_key` (pinned on first use). Built-ins are listed in
`drivers/__init__.py`; third-party drivers register under the
`devicevault.collectors` entry point. Storage drivers subclass `storage.drivers.base.StorageDriver` (see `storage/drivers/local.py`, `s3.py`), stream
content (`store` reads a binary file, `read` writes one; backups can be hundreds of MB) and register under
Expand Down
4 changes: 4 additions & 0 deletions deploy/orchestrator/orchestrator.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,10 @@ DV_DEBUG=false
DV_ALLOWED_HOSTS=devicevault.example.com
# Full origins, required when the UI is served over HTTPS by a front proxy
DV_CSRF_TRUSTED_ORIGINS=https://devicevault.example.com
# Address (or network, comma separated) of the TLS proxy or load balancer in front of the
# web service. Without it every user appears to come from the proxy: one client's failed
# sign-ins lock everyone out, and API tokens' allowed networks check the proxy's address.
DV_TRUSTED_PROXIES=
DV_TIMEZONE=Australia/Sydney
DV_LOG_LEVEL=INFO

Expand Down
2 changes: 2 additions & 0 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,8 @@ Message formats are in `protocol/src/devicevault_protocol/registration.py`.

**Trust on first use:** the worker pins the orchestrator key from the first *approved* assignment that verifies against it, so the trust decision coincides with the admin's approval. After that, assignments signed with any other key are rejected, and so is every task (`device.collect`, `storage.store`, `storage.read`) not signed by the pinned key. Tasks rejected this way fail immediately and are not retried. The Workers page shows the orchestrator's fingerprint and flags a worker that pinned a different one. Changing `DV_SIGNING_KEY` means resetting every worker's state.

**Device host keys** follow the same model. A device's SSH host key is pinned at its first collected backup: the collector reports the key the device presented in its signed `collected` event, and the orchestrator stores it on the device if it has none, accepting it only from a collector of the device's own collection group. Every later `job.collect` task carries the pinned key in its signed payload, and the collector refuses any other key before authenticating (a permanent failure, reason `auth`). An administrator of the device's group forgets the key when the device is replaced or re-keyed. Collectors keep no known-hosts state of their own.

**Limitations:**
- Broker credentials are shared per role, so revocation is enforced cryptographically (signatures and keys), not at the broker. Per-worker broker users can be added later.
- Celery remote control stays enabled because Flower needs it, so anyone holding broker credentials can still send Celery control commands (for example, to stop a worker consuming). They cannot forge assignments or tasks.
Expand Down
31 changes: 30 additions & 1 deletion docs/DEPLOYMENT.md
Original file line number Diff line number Diff line change
Expand Up @@ -79,14 +79,25 @@ not exist, the bootstrap admin from `DV_ADMIN_USERNAME` / `DV_ADMIN_PASSWORD` as
a member of it. Put a TLS-terminating proxy or load balancer in front of `web`,
and set `DV_ALLOWED_HOSTS` and `DV_CSRF_TRUSTED_ORIGINS` to match.

Set `DV_TRUSTED_PROXIES` to that proxy's address (or its network, comma separated
for several). The `web` service records the address that connected to it, which
is then the proxy rather than the user; with the proxy listed, the orchestrator
takes the address the proxy reports instead. Without it every user appears to come
from the proxy: sign-in lockout by address then locks everyone out together, an API
token's allowed networks are compared with the proxy, and the audit log shows only
the proxy. The proxy must append to (or replace) `X-Forwarded-For`, as nginx,
HAProxy (`option forwardfor`), Traefik and cloud load balancers do. Never list an
address that clients could connect from themselves.

### Users and access

Sign-in uses a server-side session: an HttpOnly cookie (`Secure` unless
`DV_DEBUG`, so serve the UI over HTTPS), stored in the database so every
orchestrator node shares it. Sessions end after `DV_SESSION_IDLE_MIN` without
use and `DV_SESSION_MAX_HOURS` after sign-in. Repeated failed sign-ins lock out a
username or client address for `DV_LOGIN_LOCKOUT_MIN`; the client address is the
last `X-Forwarded-For` hop, which the `web` service adds.
last `X-Forwarded-For` hop that is not one of `DV_TRUSTED_PROXIES` (with none set,
the hop the `web` service adds).

**API tokens** let scripts and monitoring call the API without a session: send
`Authorization: Bearer dvt_...`. A token acts as its owner and never beyond their
Expand Down Expand Up @@ -199,6 +210,23 @@ log line matches. To automate approval: `manage.py approve_worker <name>
- **Revoke** stops a worker consuming. **Forget** removes it; if it is still
running it registers again as pending.

### Device SSH host keys

Collectors pin each device's SSH host key on first use. A device's first backup
over SSH accepts the key the device presents; the orchestrator records it on the
device (audited as `device.host_key_pin`, and noted in that backup's log) and sends
it with every later backup. From then on a collector connects only if the device
presents exactly that key, before any username or password is sent, so a host
answering in the device's place cannot collect its credentials. A backup refused
this way fails at once with the reason `auth` and is not retried.

The device's page shows the pinned key's type and `SHA256:` fingerprint; compare it
with the device's own (`show crypto key mypubkey` and the like) when the first
backup is not made over a network you trust. When a device is replaced or its keys
are regenerated, an administrator of its device group chooses **Forget host key**
there (audited as `device.host_key_forget`), and the next backup pins the new key.
Only a collector of the device's own collection group can pin a key.

### Storage targets

Under **Admin Settings > Storage Targets**, create a target per place backups go:
Expand Down Expand Up @@ -722,6 +750,7 @@ secrets). The file takes precedence.
| `DV_DEBUG` | `false` | Development mode; never in production |
| `DV_ALLOWED_HOSTS` | none (`*` in debug) | Hostnames the UI is served on |
| `DV_CSRF_TRUSTED_ORIGINS` | none | Full origins (`https://host`) behind HTTPS |
| `DV_TRUSTED_PROXIES` | none | Addresses or networks of the proxies in front of `web`; the client address is the last `X-Forwarded-For` hop that is not one of them |
| `DV_TIMEZONE` | `UTC` | Display timezone (storage is always UTC) |
| `DV_LOG_LEVEL` | `INFO` | |
| `DV_DB_ENGINE` | `sqlite` | `sqlite` or `postgres` |
Expand Down
2 changes: 2 additions & 0 deletions frontend/src/api/endpoints.ts
Original file line number Diff line number Diff line change
Expand Up @@ -164,6 +164,8 @@ export const devicesApi = {
deactivate: async (id: Id, reason: string) =>
(await http.post<DeviceDetail>(`/devices/${id}/deactivate/`, { reason })).data,
reactivate: async (id: Id) => (await http.post<DeviceDetail>(`/devices/${id}/reactivate/`)).data,
/** Drop the pinned SSH host key (a replaced or re-keyed device); the next backup pins the new one. */
forgetHostKey: async (id: Id) => (await http.post<DeviceDetail>(`/devices/${id}/forget_host_key/`)).data,
/** Global administrators: unpins and deletes every backup of an inactive device. `confirm` is its name. */
purge: async (id: Id, confirm: string) =>
(await http.post<DeviceDetail>(`/devices/${id}/purge/`, { confirm })).data,
Expand Down
5 changes: 5 additions & 0 deletions frontend/src/api/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -225,6 +225,11 @@ interface DeviceBase {
storage_target: string | null
credential: Id | null
schedule: Id | null
/** The SSH host key pinned at the device's first backup, which collectors then require; '' while none is. */
ssh_host_key_type: string
/** `SHA256:...`, as the device and `ssh-keygen -l` show it. */
ssh_host_key_fingerprint: string
ssh_host_key_pinned_at: IsoDateTime | null
/** What the signed-in user holds on this device's group. */
user_permissions: GrantFlag[]
inactive_at: IsoDateTime | null
Expand Down
58 changes: 58 additions & 0 deletions frontend/src/pages/EditDevicePage.vue
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,33 @@
{{ device.last_backup_time ? formatDateTime(device.last_backup_time) : 'Never' }}
</q-badge>
</div>
<div v-if="device.backup_method" class="col-12">
<div class="text-subtitle2">SSH host key</div>
<template v-if="device.ssh_host_key_fingerprint">
<div class="host-key">{{ device.ssh_host_key_type }} {{ device.ssh_host_key_fingerprint }}</div>
<div class="text-caption text-grey-7">
Pinned
{{ device.ssh_host_key_pinned_at ? formatDateTime(device.ssh_host_key_pinned_at) : '' }}
at the first backup. Backups fail if the device presents any other key.
</div>
<q-btn
v-if="isAdmin"
flat
dense
no-caps
color="negative"
icon="key_off"
label="Forget host key"
class="q-mt-xs"
:loading="forgetting"
@click="forgetHostKey"
/>
</template>
<div v-else class="text-caption text-grey-7">
None pinned yet. The key the device presents at its first backup over SSH is pinned and required from
then on.
</div>
</div>
</q-card-section>
</q-card>

Expand Down Expand Up @@ -211,6 +238,7 @@
</template>

<script setup lang="ts">
import { useQuasar } from 'quasar'
import { computed, onMounted, ref } from 'vue'
import { useRoute, useRouter } from 'vue-router'

Expand Down Expand Up @@ -252,6 +280,7 @@ const route = useRoute()
const router = useRouter()
const auth = useAuthStore()
const notify = useNotify()
const $q = useQuasar()

const deviceId = computed(() => (route.name === 'device-edit' ? Number(route.params.id) : null))
const isEdit = computed(() => deviceId.value !== null)
Expand Down Expand Up @@ -360,6 +389,31 @@ function toForm(detail: DeviceDetail): DeviceWrite {
}
}

const forgetting = ref(false)

function forgetHostKey(): void {
const current = device.value
if (!current) return
$q.dialog({
title: 'Forget the SSH host key?',
message:
`The next backup of ${current.name} will trust and pin whatever key the device at ` +
`${current.ip_address} presents. Do this only if the device was replaced or re-keyed.`,
cancel: true,
ok: { label: 'Forget host key', color: 'negative', flat: true },
}).onOk(async () => {
forgetting.value = true
try {
device.value = await devicesApi.forgetHostKey(current.id)
notify.success('Host key forgotten')
} catch (error) {
notify.error(error, 'Failed to forget the host key')
} finally {
forgetting.value = false
}
})
}

async function save(): Promise<void> {
saving.value = true
try {
Expand Down Expand Up @@ -419,4 +473,8 @@ onMounted(async () => {
.device-form > * {
max-width: 60ch;
}
.host-key {
font-family: monospace;
overflow-wrap: anywhere;
}
</style>
45 changes: 45 additions & 0 deletions orchestrator/accounts/tests/test_auth.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,51 @@ def test_the_client_address_comes_from_the_proxy_hop(alice):
assert AuditLog.objects.get(action='auth.login').ip == '203.0.113.5'


def _address(settings, forwarded, proxies=()):
import ipaddress

from django.test import RequestFactory

from audit.log import client_ip

settings.DEVICEVAULT_TRUSTED_PROXIES = tuple(ipaddress.ip_network(proxy) for proxy in proxies)
return client_ip(RequestFactory().get('/', REMOTE_ADDR='172.18.0.5', HTTP_X_FORWARDED_FOR=forwarded))


@pytest.mark.parametrize(
('forwarded', 'proxies', 'expected'),
[
# A TLS proxy in front of nginx: the client is the hop the proxy added, not the proxy.
('198.51.100.7, 10.0.0.2', ['10.0.0.0/24'], '198.51.100.7'),
# What the client sent itself (left of the proxy's hop) is never believed.
('192.0.2.99, 198.51.100.7, 10.0.0.2', ['10.0.0.0/24'], '198.51.100.7'),
# Two proxies in a row, IPv4 and IPv6.
('198.51.100.7, 2001:db8::1, 10.0.0.2', ['10.0.0.2/32', '2001:db8::/64'], '198.51.100.7'),
('198.51.100.7, ::ffff:10.0.0.2', ['10.0.0.0/24'], '198.51.100.7'),
# A request from the proxy itself (a health check).
('10.0.0.2', ['10.0.0.0/24'], '10.0.0.2'),
# A client pretending to be a proxy gains nothing: its own hop is still what the proxy wrote.
('10.0.0.9, 198.51.100.7, 10.0.0.2', ['10.0.0.0/24'], '198.51.100.7'),
('junk, 10.0.0.2', ['10.0.0.0/24'], None),
('198.51.100.7, 10.0.0.2', [], '10.0.0.2'),
],
)
def test_trusted_proxies_are_skipped_to_find_the_client(settings, forwarded, proxies, expected):
assert _address(settings, forwarded, proxies) == expected


def test_behind_a_trusted_proxy_one_client_cannot_lock_everyone_out(alice, settings):
import ipaddress

settings.DEVICEVAULT_LOGIN_MAX_FAILURES = 2
settings.DEVICEVAULT_TRUSTED_PROXIES = (ipaddress.ip_network('10.0.0.2/32'),)
for n in range(3):
sign_in(APIClient(HTTP_X_FORWARDED_FOR='198.51.100.7, 10.0.0.2'), username=f'guess-{n}', password='nope')

assert sign_in(APIClient(HTTP_X_FORWARDED_FOR='198.51.100.7, 10.0.0.2')).status_code == 429
assert sign_in(APIClient(HTTP_X_FORWARDED_FOR='203.0.113.5, 10.0.0.2')).status_code == 200


def test_disabled_users_cannot_sign_in_and_lose_their_session(alice, admin_client):
client = APIClient()
sign_in(client)
Expand Down
6 changes: 6 additions & 0 deletions orchestrator/api/serializers.py
Original file line number Diff line number Diff line change
Expand Up @@ -275,6 +275,9 @@ class Meta:
'storage_target',
'schedule',
'credential',
'ssh_host_key_type',
'ssh_host_key_fingerprint',
'ssh_host_key_pinned_at',
'user_permissions',
*LIFECYCLE_FIELDS,
]
Expand Down Expand Up @@ -353,6 +356,9 @@ class Meta:
'storage_target',
'schedule',
'credential',
'ssh_host_key_type',
'ssh_host_key_fingerprint',
'ssh_host_key_pinned_at',
'user_permissions',
*LIFECYCLE_FIELDS,
]
Expand Down
40 changes: 40 additions & 0 deletions orchestrator/api/tests/test_devices.py
Original file line number Diff line number Diff line change
Expand Up @@ -42,3 +42,43 @@ def test_version_needs_no_authentication(db):

assert resp.status_code == 200
assert resp.json() == {'version': version()}


HOST_KEY = 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJ/5z9AzdT8Dm9DM5cwhVsGr+7YDGjr3iqkJqscQWMlL'


def test_a_device_shows_its_pinned_host_key_as_a_fingerprint(device, admin_client):
assert admin_client.get(f'/api/devices/{device.pk}/').json()['ssh_host_key_fingerprint'] == ''
Device.objects.filter(pk=device.pk).update(ssh_host_key=HOST_KEY)

body = admin_client.get(f'/api/devices/{device.pk}/').json()

assert body['ssh_host_key_type'] == 'ssh-ed25519'
assert body['ssh_host_key_fingerprint'] == 'SHA256:kuehv88oMZA/RgHRswCgnfGgnwqVeQ91l54enX5Lzec'
assert 'ssh_host_key' not in body


def test_the_host_key_cannot_be_set_through_the_api(device, admin_client):
reply = admin_client.patch(f'/api/devices/{device.pk}/', {'ssh_host_key': HOST_KEY}, format='json')

assert reply.status_code == 200
assert Device.objects.get(pk=device.pk).ssh_host_key == ''


def test_group_admins_forget_a_host_key_and_it_is_audited(device, make_user, client_for):
from audit.models import AuditLog

Device.objects.filter(pk=device.pk).update(ssh_host_key=HOST_KEY)
viewer = client_for(make_user('viewer', grants={device.device_group: ['view', 'backup_now']}))
admin = client_for(make_user('groupadmin', grants={device.device_group: ['admin']}))

assert viewer.post(f'/api/devices/{device.pk}/forget_host_key/').status_code == 403
assert Device.objects.get(pk=device.pk).ssh_host_key == HOST_KEY

reply = admin.post(f'/api/devices/{device.pk}/forget_host_key/')

assert reply.status_code == 200 and reply.json()['ssh_host_key_fingerprint'] == ''
device.refresh_from_db()
assert device.ssh_host_key == '' and device.ssh_host_key_pinned_at is None
entry = AuditLog.objects.get(action='device.host_key_forget')
assert entry.actor_name == 'groupadmin' and entry.details['key_type'] == 'ssh-ed25519'
22 changes: 22 additions & 0 deletions orchestrator/api/views.py
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ class DeviceViewSet(viewsets.ModelViewSet):
``POST {id}/backup_now/`` needs ``backup_now``. Retiring a device
(``deactivate/``, ``reactivate/``, and ``DELETE`` once it is deletable) needs
``admin``; ``purge/`` is for global administrators (devices.lifecycle).
``POST {id}/forget_host_key/`` (``admin``) drops the SSH host key pinned at the device's first backup.
``?lifecycle=active|inactive|deletable`` filters the list.
"""

Expand Down Expand Up @@ -236,6 +237,27 @@ def reactivate(self, request, pk=None):
)
return self._detail()

@decorators.action(detail=True, methods=['post'], permission_classes=[HasDeviceGroupPermission('admin')])
def forget_host_key(self, request, pk=None):
"""Drop the pinned SSH host key (the device was replaced or re-keyed): its next backup pins the new one."""
device = self.get_object()
if device.ssh_host_key:
details = {
'device': device.pk,
'device_name': device.name,
'key_type': device.ssh_host_key_type,
'fingerprint': device.ssh_host_key_fingerprint,
}
Device.objects.filter(pk=device.pk).update(ssh_host_key='', ssh_host_key_pinned_at=None)
record(
request,
'device.host_key_forget',
resource=str(device.pk),
device_group=device.device_group_id,
details=details,
)
return self._detail()

@decorators.action(detail=True, methods=['post'], permission_classes=[IsGlobalAdmin])
def purge(self, request, pk=None):
"""End an inactive device's hold now: unpin and delete all of its backups."""
Expand Down
27 changes: 21 additions & 6 deletions orchestrator/audit/log.py
Original file line number Diff line number Diff line change
Expand Up @@ -3,19 +3,34 @@
import ipaddress
import logging

from django.conf import settings

from audit.models import AuditLog

logger = logging.getLogger('devicevault.audit')


def client_ip(request) -> str | None:
"""The client's address: the last X-Forwarded-For hop (added by our nginx), else the peer."""
"""The client's address, from X-Forwarded-For (our nginx appends the peer it saw), else the peer.

Read from the right, skipping ``DV_TRUSTED_PROXIES`` (a TLS proxy or load balancer in front of
nginx): the first hop that is not one of them is the client, since everything to its left is
whatever the client chose to send. With no trusted proxies that is the hop nginx added.
"""
forwarded = request.META.get('HTTP_X_FORWARDED_FOR', '')
address = forwarded.split(',')[-1].strip() if forwarded else request.META.get('REMOTE_ADDR', '')
try:
return str(ipaddress.ip_address(address))
except ValueError:
return None
hops = [hop.strip() for hop in forwarded.split(',')] if forwarded else [request.META.get('REMOTE_ADDR', '')]
address = None
for hop in reversed(hops):
try:
address = ipaddress.ip_address(hop)
except ValueError:
return None # a hop a trusted proxy would never write: nothing left of here can be believed
if address.version == 6 and address.ipv4_mapped:
address = address.ipv4_mapped
if not any(address in network for network in settings.DEVICEVAULT_TRUSTED_PROXIES):
break
# Every hop a trusted proxy: the request came from the outermost one itself.
return str(address) if address is not None else None


def _token_prefix(request) -> str:
Expand Down
Loading