Skip to content

Q-SYS core monitoring and alerts

Updated August 11, 2026.

Purpose

The production portal VM continuously monitors the Auditorium and Dark Room Q-SYS cores. The listener detects real outages, recovery, restarts, new design deployments, and same-design redeployments without relying on a once-per-minute poll that can miss short state changes.

The monitor is an operational safety net. It does not change a Q-SYS design, control a core, or automatically restart AV equipment.

Production architecture

  • Systemd unit: qsys-core-watch.service
  • Process: scripts/qsys-core-listen.php in the production portal release
  • Protocol: persistent QRC connections with SNMP uptime as independent reboot evidence
  • State table: logs.qsys_core_state
  • Append-only event table: logs.qsys_core_events
  • Alert transport: the protected portal mail configuration
  • Default alert route: River Oaks Technology staff; QSYS_ALERT_TO may provide an explicit protected override

The listener tracks each core independently. A failure on one core does not make the other core unhealthy.

Event rules

Event Rule Email
Design deployment Design identity changes after the settling window Yes
Same-design redeploy The same design stops and starts without a new identity Yes
Offline Continuous unreachability exceeds the configured offline threshold Yes
Online A previously offline core reconnects Audit record only
Reboot SNMP uptime resets, or recovery provides equivalent restart evidence Yes

The default settling window is 30 seconds and the default offline threshold is 120 seconds. This prevents an ordinary design push from generating a false offline/reboot sequence. Do not reduce the offline threshold below the settling window.

Logging and audit trail

qsys_core_state stores one current row per core: friendly label, reachability, consecutive failure count, current design identity, SNMP uptime, last successful contact, and update time.

qsys_core_events is append-only. It records the core, event type, old/new design identity where relevant, platform/state metadata, bounded evidence, and detection time. Use aggregate queries for routine health checks; do not paste design details or mail credentials into tickets.

The systemd journal contains connection, keepalive, state-transition, retry, database, and mail-delivery diagnostics. Successful steady-state keepalives are expected. Repeated connection failures for one core should correspond to its failure count and, after the threshold, one offline event rather than repeated alert mail.

Routine health check

On the portal VM:

systemctl status qsys-core-watch.service --no-pager
journalctl -u qsys-core-watch.service --since "30 minutes ago" --no-pager

Healthy means:

  • the unit is enabled and active;
  • the PHP listener has one long-running process;
  • each expected core has recent keepalive or reconnect activity;
  • qsys_core_state.reachable is 1 and its failure count is 0;
  • no current database or mail-delivery errors appear in the journal.

The tracked source includes a dry-run mode that connects to the cores and prints intended actions without database writes or email:

sudo -u www-data env APP_ENV=production \
  php /usr/share/nginx/portal/scripts/qsys-core-listen.php --dry-run

Do not run the dry-run for an extended period beside production. It creates additional QRC connections and is intended only for a bounded diagnostic.

Response guide

Unit is stopped or failed

  1. Read the latest journal entries before restarting.
  2. Verify the production release and protected environment are present.
  3. Confirm database and mail variable names exist without printing their values.
  4. Start the unit once and confirm both core connections stabilize.
  5. If it fails again, leave the evidence intact and escalate rather than using a restart loop.

One core is offline

  1. Confirm the other core remains healthy.
  2. Check switching, VLAN, power, and the core itself before changing the monitor.
  3. Compare the journal transition with the append-only event timestamp.
  4. After recovery, verify an online audit record and current state row.

Both cores are offline

Treat this first as a shared network, gateway, power, or portal-host path failure. Check shared dependencies before touching either Q-SYS design.

Event logged but email missing

  1. Confirm the event row exists.
  2. Check the journal at the same timestamp for a bounded MailerSend HTTP or network error.
  3. Verify only the presence and file metadata of protected mail variables.
  4. Use the approved portal mail smoke-test path; never print or copy the API credential into a shell command or ticket.

Change safety

  • Keep the persistent listener and legacy minute poller mutually exclusive; running both would duplicate state transitions.
  • Preserve the settling/offline relationship when tuning thresholds.
  • Apply schema changes additively and retain the append-only event history.
  • Test state-machine changes with dry-run and automated tests before production.
  • Never simulate an outage during a service or restart a core solely to test an alert.