Skip to content

Storage manager automation

Updated August 10, 2026.

storage-mgr is unprivileged Proxmox LXC 109. It coordinates several independent recording, multitrack, and VM-backup flows. This page is the control-plane inventory: it identifies the active scheduler for each flow, the Git source of truth where one exists, safe health checks, and the legacy scripts that must not be re-enabled alongside their replacements.

Active schedules

Flow Active scheduler Runtime entry point Purpose Detailed runbook
Multitrack ingest root cron, daily at 22:00 /root/audio-backup.sh Copies completed service multitracks into bulk storage This page
Proxmox offsite backup root cron, daily at 00:00 /root/proxmox-backup-sync.sh Resumes the current ISO-week snapshot to Google Shared Drive and retains 13 weeks Proxmox backups
HyperDeck archive root cron, daily at 01:00 /root/hypedeck-backup.sh Pulls completed HyperDeck clips, verifies byte counts before device deletion, then syncs the dated archive riveroaks/infra/apps/hyperdeck/README.md
Multitrack cloud copy rclone-multitracks.service /usr/local/bin/rclone-multitracks-loop.sh Copies the bulk-storage multitrack archive to its approved cloud destination with bounded CPU and I/O Files storage reliability
OBS recording archive ro-obs-archive.timer, every 10 minutes /usr/local/sbin/ro-obs-archive Pulls stable OBS recordings, verifies SHA-256, publishes atomically, and only then removes the Mac source OBS recording archive

The root crontab is authoritative for the three cron rows. rclone-multitracks.service is the only active scheduler for the cloud multitrack copy; its older cron entries are intentionally commented out. ro-obs-archive.timer replaced /root/obs-backup.sh; all OBS cron variants are intentionally disabled.

Source of truth and secrets

  • Proxmox offsite sync is Git-managed at riveroaks/infra/storage/hosts/storage-mgr/files/usr/local/sbin/riveroaks-proxmox-backup-sync and deployed to /root/proxmox-backup-sync.sh.
  • HyperDeck automation is Git-managed under riveroaks/infra/apps/hyperdeck. Its Influx token belongs in the root-only /etc/hyperdeck/influx.env, never in the script or documentation.
  • OBS archive code, installer, rollback, and threat model are Git-managed under riveroaks/infra/apps/obs-archive.
  • The multitrack cloud service and its resource limits are documented in the Files storage runbook. Do not restore the disabled root cron while the service is active.
  • /root/audio-backup.sh remains a legacy host-managed script. Treat it as production-only technical debt: preserve it during routine work, do not copy its contents into tickets, and migrate it to reviewed infrastructure code before making behavioral changes.
  • rclone remote configuration and any service credentials remain local root-owned runtime configuration. Documentation names boundaries and variable names only.

Safe health checks

Run these from the Proxmox host. They inspect status and syntax without reading credentials or backup contents.

pct status 109
pct exec 109 -- crontab -l
pct exec 109 -- systemctl is-active rclone-multitracks.service ro-obs-archive.timer
pct exec 109 -- systemctl list-timers ro-obs-archive.timer --no-pager
pct exec 109 -- bash -n /root/audio-backup.sh
pct exec 109 -- bash -n /root/hypedeck-backup.sh
pct exec 109 -- bash -n /root/proxmox-backup-sync.sh
pct exec 109 -- /root/proxmox-backup-sync.sh --dry-run
pct exec 109 -- df -h /mnt/bulk-storage /mnt/proxmox-backups

The expected active root-cron times are 22:00 for multitrack ingest, 00:00 for Proxmox offsite sync, and 01:00 for HyperDeck. A duplicate active rclone or OBS cron is a fault, not redundancy.

Use bounded log queries when troubleshooting:

pct exec 109 -- journalctl -u rclone-multitracks.service --since "2 hours ago" --no-pager
pct exec 109 -- journalctl -u ro-obs-archive.service --since "2 hours ago" --no-pager
pct exec 109 -- tail -n 200 /var/log/proxmox-backup-sync.log

Logs and tickets must not include recording names, transcript text, rclone configuration, tokens, credentials, or backup contents. Prefer counts, byte totals, duration, bounded error codes, and object hashes.

Change and recovery rules

  1. Update the infrastructure repository first whenever a Git-managed flow changes.
  2. Run syntax and network-free regression tests, then require a passing merge-request pipeline.
  3. Preserve the installed artifact with a dated recovery copy before deployment.
  4. Compare SHA-256 between the reviewed artifact and the installed copy.
  5. Use a dry-run or no-delete mode before a live copy whenever the flow supports it.
  6. Never run a second scheduler in parallel to “test” an archive. Respect the flow's existing lock.
  7. Never delete a source recording or backup to repair a cloud quota failure. Resume from the verified local copy after the provider limit resets.

The August 10 audit verified all four root scripts above with bash -n, confirmed the expected cron ownership, and verified the two replacement systemd workflows. The Proxmox sync is now idempotent and resumable; one historical week-32 object remains tracked separately until Google Drive's upload quota resets.