shared_fs_disks has no 'nfs' type — how to define external NFS mounts for trix_home/easybuild/images in HA?

[peter200026]

I ran into a related question while setting up HA with TrinityX v16 on Rocky 9.8. I wanted to share what I found while digging through the code, since it might help clarify this issue.

Looking at roles/trinity/shared-fs/tasks/main.yaml, the shared_fs_disks variable is only processed for type: 'drbd', type: 'iscsi', and type: 'direct'. There is no type: 'nfs' branch in this role (there's even a comment # more types can be added here).

Separately, roles/trinity/nfs-mounts/tasks/main.yml handles a different variable, nfs_mounts, which is combined with nfs_remote_mask (set to shared_fs_disks in controller.yml) and an nfs_use_mask/xmount flag. This role is invoked in controller.yml for trix_shared, trix_ohpc, trix_ha... but it re-exports these paths from the active controller (trix_ctrl_hostname) to both nodes — it doesn't appear to consume an externally defined NFS remote for trix_home/trix_easybuild/etc.

Given this, could you clarify:

What is the exact, supported syntax to put trix_home (or trix_easybuild/trix_images) on an external NFS server (not a controller-exported share) in an HA setup?
Does this need to go through nfs_mounts directly with a remote: pointing at the external NFS server, bypassing shared_fs_disks entirely?

Any pointer to a working example all.yml snippet for this scenario would be very helpful.

Regards
Peter

[aphmschonewille]

Thanks for digging into the roles before filing — your reading of shared-fs is
exactly right, and it turns out you found a real bug on the way.

type: 'nfs' is supported, and it is the intended syntax. You are correct that
roles/trinity/shared-fs/tasks/main.yaml has no nfs branch, but that is by
design: that role provisions block devices (DRBD, iSCSI, direct), and a remote
share has nothing to provision. The nfs type is consumed further along, where
shared_fs_disks is handed to the nfs and nfs-mounts roles as
nfs_remote_mask (controller.yml, and the image/login imports). There,
nfs-mounts does resolve an externally defined remote:

{% if disk["type"] == "nfs" %}
  {{ disk["remote"] }}

So you do not need to bypass shared_fs_disks. The supported form is documented
in group_vars/all.yml.example:

shared_fs_disks:
  - name: '{{ trix_home }}'
    mount: '{{ trix_home }}'
    type: 'nfs'
    remote: '192.0.2.10:/homes'
    mountoptions: 'nfsvers=4,rw,retrans=4,_netdev'

lustre, beegfs, mmfs, gpfs and panfs are handled the same way.

Please keep the name: key. On the version you are running, an entry that
carries only mount: makes the whole mask stop matching — silently, with no
error — so both your external share and the cross-mounts of your DRBD disks
quietly revert to controller-exported paths. That one is already fixed on our
development line and will ship with 16.1, which has not been released yet, so on
any current release the name: key is what keeps it working.

Now the bug you have run into. With the above in place, compute and login nodes
mount the external server correctly, but the controllers do not mount it at all
— in HA and non-HA alike. On a controller the mount is gated on an xmount flag
that is deliberately not set for external share types, so the entry is resolved
and then skipped. The practical effect in your setup: the nodes see the NAS, while
anything the controller writes to /trinity/home directly (user creation, for
instance) lands on controller-local disk, and the controller re-exports an empty
directory.

This is a regression rather than a missing feature. On the 14.x line these mounts
were applied on the controller as well; the gate that suppresses them was
introduced in 15.x and is still present today. We have raised it internally as a
bug. Until it is fixed, if you need the controllers to see the share, mount it
outside TrinityX (fstab or a systemd mount unit) on both controllers at the same
path, and keep the shared_fs_disks entry so the nodes are configured from it.

Two further points on your list:

• trix_ohpc and trix_easybuild behave the same as trix_home — they work on
  the nodes, and are affected by the same controller-side gap.
• trix_images cannot be placed on an external NFS share this way at all. There
  is no mount entry for it for the mask to override, so shared_fs_disks has
  nothing to act on. That is covered by the same internal ticket.

On trix_ha, to correct the impression #557 may have left: it can live on
NFS, and has been run that way. The constraint is not that every share must come
from one particular server — it is that it has to be NFS across the board: all
TrinityX shares imported over NFS, with the controllers running no NFS service of
their own. That is the configuration we can confirm working. What is less desirable
is the mixed case, some shares local or on DRBD and others imported, although it should work. The
extended-attribute caveat raised in #557 still applies. Worth knowing that the
controller-side bug above is precisely what stands in the way of expressing that
all-NFS arrangement through shared_fs_disks today.