Skip to content

Files storage reliability

Updated August 11, 2026.

This runbook covers the Multitracks folder index used by the staff portal and the public download host. The goal is for metadata browsing to remain responsive while large downloads and cloud backups use the same storage.

Request path

The staff Files page gets metadata from https://files.ro.church:8080 over the internal network. The portal connects directly to the storage address while keeping the public hostname for TLS verification.

  • GET /list/folders returns immediate folder names. It is intentionally O(number of folders) and does not recursively size the full archive.
  • GET /list/folder-files?folder=... walks only the selected folder and returns individual file names, paths, and exact sizes.
  • Downloads continue to stream directly from storage. The portal does not proxy ZIP or audio payloads.
  • Listing endpoints remain restricted to the portal host by nginx. Do not make them public to simplify monitoring.

size_bytes remains 0 on folder rows for compatibility with older portal code. The folder-picker no longer displays an aggregate size. Exact sizes are shown only after one folder is selected.

The former implementation recursively called stat for every file in every folder during each /list/folders request. Because njs filesystem work is synchronous, that blocked an nginx worker and made routine storage contention look like an outage.

Portal fallback

The portal caches successful listing responses:

  • fresh metadata is reused for two minutes by default;
  • last-known-good metadata may be displayed for seven days;
  • cache files contain metadata only and use restrictive permissions;
  • cache updates use a temporary file plus atomic rename, so an interrupted PHP worker cannot replace a good cache with a partial file;
  • creating a share always validates the selected folder against live storage.

Runtime controls are FILES_LISTING_CACHE_TTL, FILES_LISTING_STALE_TTL, and FILES_LISTING_MAX_BYTES.

An unavailable folder index must not prevent staff from opening existing shares or managing existing links.

Multitracks cloud sync

rclone-multitracks.service is the only scheduler allowed to own the Multitracks cloud copy. The older root cron entry is disabled.

The service runs /usr/local/bin/rclone-multitracks-loop.sh with:

  • nice level 15;
  • best-effort I/O priority 7;
  • a 100% CPU quota;
  • two transfers and four checkers;
  • 16 MB buffers and 32 MB Drive chunks.

Logs belong under /var/log/rclone/multitracks_service.log. A missing /var/log/rclone directory causes every service attempt to fail before syncing, so verify that directory when troubleshooting.

These limits favor active staff browsing and downloads over backup completion time. Do not re-enable the cron copy while the systemd service is enabled.

NFS trust boundary

The portal VM mounts only the multitrack subtree from rocc01:

  • source: 10.200.24.2:/main-pool/bulk-storage/multitracks;
  • target: /mnt/multitracks;
  • client: rocc-db (10.200.24.42);
  • client options: read-only NFSv4, hard mount, _netdev, nofail, and x-systemd.automount.

Both transcript orchestrators depend on mnt-multitracks.automount. The portal Files browser also reads this mount. The storage-mgr container receives the bulk-storage tree through a host-local bind mount, so its nginx, ZIP-helper, and rclone paths do not require a network export.

The August 11 inventory found only the portal VM connected to NFS. No other PVE guest configuration, host unit, cron job, mountd record, or repository path referenced the network export. This is strong current-state evidence, but NFSv4 does not provide a dependable historical list of offline clients.

The current server configuration exports the entire /main-pool parent to all clients as read/write. The intended least-privilege replacement is:

/main-pool/bulk-storage/multitracks 10.200.24.42(ro,sync,no_subtree_check,root_squash,insecure)

insecure is retained because the observed Linux NFS client uses a high source port. Do not retain crossmnt when the export root is the exact directory the portal needs.

Apply the change only in a named maintenance window:

  1. Preserve /etc/exports with its owner and mode.
  2. Add the narrow child export while the parent export still exists, then run exportfs -ra and exportfs -v.
  3. From the portal VM, mount the exact child export read-only at a temporary, empty directory. Perform bounded stat and directory-listing checks; do not copy or print sermon content.
  4. Replace the broad parent line with the narrow export and reload exports.
  5. Verify the existing /mnt/multitracks mount is read-only, the Files storage health check passes, both transcript configurations validate, and one content-free discovery run exits without stale leases.
  6. Remove the temporary test mount and directory.

If any verification fails, restore the exact preserved exports file, run exportfs -ra, and recheck the original portal mount. Do not troubleshoot by temporarily granting world read/write access.

Health checks

From the portal VM, the following should return HTTP 200 in well under the portal's six-second request timeout:

curl --silent --show-error --output /dev/null \
  --connect-timeout 2 --max-time 8 \
  --resolve files.ro.church:8080:10.200.24.5 \
  --write-out 'status=%{http_code} total=%{time_total}s\n' \
  https://files.ro.church:8080/list/folders

On storage, check the service without printing credentials or nginx secret-bearing directives:

systemctl is-active nginx rclone-multitracks.service
systemctl show rclone-multitracks.service \
  -p Nice -p CPUQuotaPerSecUSec \
  -p IOSchedulingClass -p IOSchedulingPriority
grep -Ec '^[^#].*rclone.*multitracks' /var/spool/cron/crontabs/root
journalctl -u nginx -u rclone-multitracks.service \
  --since '15 minutes ago' --no-pager -p warning

The active Multitracks cron count should be 0. A nonzero count means two schedulers may compete for the same copy job.

The Git-managed monitor is /usr/share/nginx/html/dev-portal/scripts/files-storage-watch.php. The riveroaks service account runs it once per minute. It:

  • keeps the IP-pinned storage-origin listing probe;
  • separately requests the unsigned /dl/__portal-healthcheck__/ path through normal DNS and requires a TLS-verified HTTP 403;
  • validates status, latency, bounded response size, non-empty listings, and every listing row's safe schema;
  • tracks outage/recovery state independently for each probe;
  • records state in /var/tmp/riveroaks-files-storage-watch.json;
  • alerts after three consecutive failures rather than one transient miss;
  • sends one recovery notice after an alerted outage;
  • alerts technologystaff@riveroaks.org;
  • never prints folder names or response bodies and never restarts storage.

Review its safe output with:

journalctl -t portal-files-watch --since '15 minutes ago' --no-pager
php /usr/share/nginx/html/dev-portal/scripts/files-storage-watch.php --dry-run
php /usr/share/nginx/html/dev-portal/scripts/files-storage-watch.php --status

As of July 30, internal resolver 10.200.200.200 (Pi-hole dns-1) incorrectly answers files.ro.church with portal address 10.200.24.42. The secondary resolver returns the public Files target. This inconsistent answer reproduces the staff-side TLS protocol failure and is now correctly reported by the client_path probe. Fix the Pi-hole local DNS record so the name reaches the storage Files listener (10.200.24.5:8080, or the approved public hairpin target). Do not pin the client-path monitor to an IP; doing so would hide the same failure again.

The service-account crontab backup from installation is /var/tmp/riveroaks-crontab.bak-20260730-files-watch on the portal VM.

Monitor latency as well as status; a response approaching six seconds is an early warning of renewed storage contention. When the portal code is promoted from dev, move the scheduler to the production Git checkout and set FILES_WATCH_ALERT_TO=technologystaff@riveroaks.org after the team approves the alert volume.

Rollback

Rollback copies from the July 30 change are stored on the storage host:

  • /etc/nginx/njs/listing.js.bak-20260730-fast-index
  • /usr/local/bin/rclone-multitracks-loop.sh.bak-20260730-codex
  • /etc/systemd/system/rclone-multitracks.service.bak-20260730-codex
  • /var/spool/cron/crontabs/root.bak-20260730-codex

Restore only the component under investigation. Always run nginx -t before reloading nginx, and bash -n plus systemd-analyze verify before restarting the sync service. Restoring the old root crontab also restores the duplicate, unconstrained sync and should be a last resort.

The portal application itself remains Git-managed. Portal changes go to the dev branch and deploy through the GitLab pipeline; do not hand-edit the portal checkout on the VM.