2.4 KiB
name, description, metadata
| name | description | metadata | ||
|---|---|---|---|---|
| nfs-pnfs-scsi-layout-wedge | NAS NFS writes wedging to a crawl is pNFS SCSI layouts failing to fence on an LVM RAID1 target; fix is blacklisting blocklayoutdriver on the client |
|
Symptom seen 2026-10-02: every write to the NAS mounts crawls or stalls for
60s+, reads stay fast, and the box bogs down. /proc/meminfo showed 8.5 GB
Dirty pinned on the NFS bdi with BdiWriteBandwidth: 28 kBps, and processes
stuck in D state on balance_dirty_pages / vfs_unlink.
Cause chain. The NAS exports XFS on /dev/mapper/raid-main (dm-4), an LVM
RAID1 target. Linux nfsd automatically offers pNFS SCSI layouts for XFS, and
the client accepts — /proc/self/mountstats showed pnfs=LAYOUT_SCSI.
Fencing a SCSI layout uses SCSI persistent reservations, which the dm RAID1
target does not implement, so the recall can never succeed:
nfsd4_layout_fence_worker: FENCE failed client[10.0.0.2:938] clid[13] device[dm-4]
The client lease then expires (lease_expired climbing, 90s lease) with dirty
pages pinned to a layout that will never be released.
There is no SAN here — mainframe reaches the NAS over plain Ethernet and could
never use a SCSI layout at all. Fix applied on mainframe:
/etc/modprobe.d/nfs-no-scsi-layout.conf blacklisting blocklayoutdriver,
which implements both block and SCSI layouts. Without it the client does
ordinary NFS I/O over the wire.
Recovery is NOT possible without rebooting the NAS once it wedges. Both
echo expire > /proc/fs/nfsd/clients/<id>/ctl and systemctl restart nfs-server hang in D state, because each has to release the same unfenceable
layout; the restart leaves nfs-server stuck in deactivating/final-sigterm
and the NAS then cannot write to its own /nas either.
Diagnosis commands worth reusing: sudo cat /sys/kernel/debug/bdi/0:58/stats
(all seven NAS mounts share bdi 0:58, so one fault stalls every export),
/proc/fs/nfsd/clients/*/info and /states on the NAS for stale client
records, and grep -o 'lease_expired=[0-9]*' /proc/self/mountstats.
Related: podman-compose-bind-mount-restart
CORRECTION (same day): this fault was real and did wedge the mount, but it was NOT the cause of the chronic slow NFS writes. That was the NAS routing all its traffic over WiFi — see nas-dual-homed-wifi-path. The layout fence most likely failed because of timeouts over that flaky link. Keep the blocklayoutdriver blacklist regardless; SCSI layouts can never work here.