Focused portal workflow alerts

Updated August 12, 2026.

Purpose

The portal VM runs a content-free health monitor for the sermon-slide and service-transcription workflows. It complements the portal's SQL audit history and systemd journals; it does not replace either source of diagnostic detail.

The monitor checks every five minutes and alerts only after the same condition is present on two consecutive checks. It sends one aggregate outage message, deduplicates the condition while it remains active, and sends one recovery message after it clears.

Sermon checks filter every run, event, and notification through the upload's trusted source_environment. Production incidents cannot be opened by a development upload, and development incidents cannot suppress or satisfy a production recovery. Transcript jobs are a single production-backed corpus. The development monitor is sermon-only; the production monitor owns transcript job, timer, and Mac-pool checks.

Development alerts go to cgood@riveroaks.org. Production alerts go to productionstaff@riveroaks.org. WORKFLOW_ALERT_TO may explicitly override the recipient in a protected environment file. Never put recipient overrides, MailerSend credentials, database credentials, SSH keys, or their values in Git.

The notification channel is part of service readiness. Every delivery attempt records only the provider name, HTTP status or bounded error code, and recipient count in the systemd journal. It never logs a token, address, body, or provider response. An actionable incident remains pending until its outage or recovery message is acknowledged. If MailerSend is unavailable, the public sermon readiness endpoint becomes unavailable so the independent Uptime Kuma route can still notify operators.

Conditions monitored

Sermon-slide checks cover:

any extraction, lower-third delivery, or notification worker service that is not running;
protected cache and ProPresenter-output directories that are missing, read-only, or cannot complete an atomic write-and-sync probe;
portal storage at 85% used or below 5 GiB free as a warning, and at 95% used or below 1 GiB free as a critical condition;
missing notification-provider configuration;
a missing LibreOffice executable or unusable protected sermon-notes conversion workspace (the monitor does not launch LibreOffice);
invalid or absent active destination rules, fit limits, theme keys, immutable template files, or template SHA-256 values;
loss of the service account's ability to add files beneath any active Google Shared Drive destination;
the restricted receiver, writable destination, and free space on both the broadcast-gfx and worship slides Macs;
a recent processing stage or computer deployment that failed and has no later successful event for the same stage;
a current upload waiting for a sermon-series theme mapping;
a current upload whose broadcast-gfx delivery is deferred because delivery is disabled;
a processing run that remains active beyond the configured stale threshold;
a terminal run that did not send both its uploader and production-staff completion notifications;
a sermon trace export that exhausts its bounded retry budget or remains undelivered beyond the stale-run window. This condition protects diagnostic evidence but does not change sermon processing readiness.

The ordinary checks are intentionally cheap and do not open ProPresenter, import a presentation, modify a playlist, create Drive folders, or touch sermon files. Two consecutive failures are required before an ordinary alert is sent. Warnings and failures retain one incident key until recovery, so a persistent dependency outage does not generate repeated email.

The initial operating objectives are measurable detection and recovery objectives, not an unmeasured availability percentage:

Signal	Objective
Portal, PHP, or monitor-state loss	Uptime Kuma detects within about 3 minutes (60-second interval plus two retries)
Ordinary dependency failure	Focused alert within 10 minutes (two five-minute checks)
Recovery after repair	Recovery notice and public state update within 5 minutes
Deep readiness	Runs at 5:00, 11:00, 17:00, and 23:00 Monday through Saturday, plus 5:00 Sunday; older than 30 hours is critical

Review actual incident and recovery timestamps before adopting a formal monthly availability SLO. Do not claim an uptime percentage that this data has not yet measured.

Deep sermon readiness

Production also runs riveroaks-sermon-readiness-production.timer at 5:00, 11:00, 17:00, and 23:00 Monday through Saturday, plus 5:00 Sunday. Its non-mutating canary performs two deeper checks:

The portal generates a tiny synthetic point-and-Scripture presentation with the real generator, pinned Python environment, protobuf descriptor, first valid active immutable theme, and that theme's fit limits. The output and manifest must preserve content, counts, template SHA-256, and output SHA-256. All temporary files are removed when the check exits.
The broadcast-gfx receiver starts ProPresenter only when it was not already running, verifies that the API identifies the exact configured application version after the startup warm-up interval, confirms at least one compatible theme, resolves the configured library, and confirms the managed playlist slot still points to the last acknowledged presentation. It imports nothing and changes no playlist item. ProPresenter is closed only when the canary launched it.

A deep failure alerts immediately. The five-minute monitor retains the last deep result; a missing result or one older than 30 hours is itself critical. This catches a broken ProPresenter launch, API, theme catalog, library, playlist mapping, or generator before the next pastor upload.

The non-mutating deep check does not claim to prove that ProPresenter can persist a new playlist value: an unchanged playlist PUT is optimized without a disk write, while changing the live Sunday playlist for a canary is unsafe. The August 14 incident demonstrated this boundary. Until a separate isolated canary playlist is provisioned, persistence is verified during every real deployment with a 90-second on-disk acknowledgement and exact API-versus-store error codes. Uptime Kuma can warn before an upload for launch, API, mapping, theme, library, worker, Drive, storage, and monitor-state failures; it cannot preannounce a write-path defect that exists only when changing the live playlist.

After repairing a deep-readiness condition, run the production service once so operators do not wait for the next morning's timer before receiving recovery:

sudo systemctl start riveroaks-sermon-readiness-production.service
systemctl status riveroaks-sermon-readiness-production.service --no-pager

The service records the new bounded result in the same protected state and sends the normal deduplicated recovery notice. Do not clear the incident JSON manually.

Service-transcription checks cover:

a recent terminal transcript job failure;
an expired lease that is still marked leased or processing;
a disabled transcript timer or a timer whose last trigger is outside the freshness window;
either configured Mac becoming unavailable; if both are unavailable, the alert is summarized as a complete worker-pool outage.

These transcript conditions run only in the production monitor. The development unit sets WORKFLOW_ALERT_TRANSCRIPTS_ENABLED=0 because its acceptance timer is installed but disabled and both environments otherwise inspect the same queue. The production unit explicitly sets the flag to 1 and checks the production transcript service and timer.

Database failures and missing workflow tables are monitor failures and are reported separately. Alerts contain only environment, bounded error codes, counts, safe remediation guidance, and safe portal links. Retry-attempt failures are deduplicated by upload, so one upload does not open a separate incident for each run UUID. They do not contain sermon text, transcript text, filenames, paths, email bodies, prompts, provider responses, or credentials.

Installed units

Environment	Service	Timer	State file
Development	`riveroaks-workflow-health-dev.service`	`riveroaks-workflow-health-dev.timer`	`/var/lib/riveroaks-workflow-health/development.json`
Production	`riveroaks-workflow-health-production.service`	`riveroaks-workflow-health-production.timer`	`/var/lib/riveroaks-workflow-health/production.json`

The production-only deep service and timer are riveroaks-sermon-readiness-production.service and riveroaks-sermon-readiness-production.timer.

The state files contain only hashed incident keys and bounded operational metadata. They are writable only by the portal worker account. Install from the matching deployed portal root:

sudo scripts/install-workflow-health-watch.sh development
sudo scripts/install-workflow-health-watch.sh production

Do not install the production unit from a development checkout. Production must use code already merged and deployed to the production root.

Routine checks

Check the production timer and most recent execution without reading environment files:

systemctl status riveroaks-workflow-health-production.timer --no-pager
journalctl -u riveroaks-workflow-health-production.service -n 30 --no-pager

Run the content-safe --status check through a transient systemd unit so it receives the same protected environment file and non-secret unit settings as the monitor. Invoking the Python file directly does not load the unit settings and will correctly report environment_invalid.

sudo systemd-run --quiet --wait --pipe --collect \
  --unit=riveroaks-workflow-health-production-status \
  --uid=www-data \
  --property=EnvironmentFile=/etc/riveroaks-portal/production.env \
  --setenv=WORKFLOW_ALERT_ENVIRONMENT=production \
  --setenv=WORKFLOW_ALERT_PORTAL_BASE_URL=https://portal.ro.church \
  --setenv=WORKFLOW_ALERT_TRANSCRIPTS_ENABLED=1 \
  --setenv=WORKFLOW_ALERT_TRANSCRIPT_TIMER_UNIT=riveroaks-service-transcripts-production.timer \
  --setenv=WORKFLOW_ALERT_TRANSCRIPT_SERVICE_UNIT=riveroaks-service-transcripts-production.service \
  /opt/riveroaks-service-transcripts/bin/python \
  /usr/share/nginx/portal/scripts/workflow-health-watch.py --status

A healthy monitor execution exits successfully even when it finds a workflow incident; the alert state and email communicate the workflow failure. The content-free --status --json contract exits nonzero when the monitor is stale or a systemic sermon dependency is unavailable. One failed upload does not falsely mark the entire service offline.

The portal exposes the same bounded contract at /health/sermon-slides.php. It returns only service, environment, aggregate status, and monitor age. It never returns incident details, paths, hostnames, sermon metadata, content, email addresses, or configuration. Uptime Kuma should use an HTTP keyword monitor, require the literal "service":"sermon-slides", check the production URL every 60 seconds with two retries, use a 15-second timeout, and accept only 200 through 299. HTTP 503 means the monitor is stale or a systemic dependency is unavailable. A capacity warning returns HTTP 200 with status=degraded; the focused email contains the safe metric and remediation.

The installed monitor is named Sermon Slides Readiness. It uses the existing NTFY notification route. Keep that route independent from MailerSend; otherwise an email-provider outage could silence both layers.

Portal capacity retention and recovery

The portal VM's root filesystem is part of sermon-slide readiness. A warning is raised at 85% used or below 5 GiB free; a critical condition is raised at 95% used or below 1 GiB free. Do not lower these thresholds to clear an incident.

The local PM2 Node-RED stdout and stderr files are operational logs, not an authoritative sermon archive. They are managed from riveroaks/infra by deploy/targets/portal/rocc-db-log-retention.json. The installed timer checks the files every 15 minutes, rotates each at 50 MiB, compresses archives, and retains seven rotations. It uses copytruncate, so rotation does not restart Node-RED. Google Drive originals, protected sermon uploads, SQL processing history, databases, and the read-only multitrack mount are outside this policy.

Routine checks are content-free:

systemctl status riveroaks-pm2-logrotate.timer --no-pager
systemctl show riveroaks-pm2-logrotate.service --property=Result --value
sudo du -x -m --max-depth=1 /home/riveroaks/.pm2 | sort -n
df -h /

If legacy unbounded PM2 logs already consumed the filesystem, first prove the exact live bindings without displaying log content:

PM2_HOME=/home/riveroaks/.pm2 \
  /home/riveroaks/.nvm/versions/node/v18.18.2/bin/node \
  /home/riveroaks/.nvm/versions/node/v18.18.2/lib/node_modules/pm2/bin/pm2 jlist \
  | jq -r '.[] | [.name, .pm2_env.status, .pm2_env.pm_out_log_path, .pm2_env.pm_err_log_path] | @tsv'
sudo find /home/riveroaks/.pm2/logs -maxdepth 1 -type f \
  -name 'node-red-*.log' -printf '%f\t%u:%g\t%s bytes\n'

Only after the process is online and the two paths match the active PM2 metadata may an operator truncate those exact files in place. Truncation keeps the open file descriptors valid and avoids a Node-RED restart, but the removed log history is not recoverable through the application. Do not archive an oversized error log merely to retain it: legacy logs can contain provider credentials or message payloads. Capture only bounded error categories and timestamps, never content or secrets. Any credential found in a log must be treated as compromised, rotated at the provider, replaced through the protected runtime boundary, and proven rejected after cutover.

After capacity returns, deploy or verify the managed timer, then run the production workflow monitor once. Confirm that /health/sermon-slides.php returns HTTP 200 without storage_warning, and confirm the deduplicated recovery email reaches the currently configured workflow-alert recipient. Do not delete the monitor state file to force recovery; the previous incident state is what causes the legitimate recovery message.

Use three separate sources during triage:

Uptime Kuma answers whether the monitoring loop and systemic sermon service are currently available.
Focused email identifies the bounded dependency and safe next action.
SQL processing events and the systemd journal provide run-level history and correlation IDs.

Alert triage

Confirm the environment in the subject before changing anything.
Open the safe run link in the alert. For slides, inspect the newest portal processing timeline and its first failed stage. For transcripts, inspect queue status and the systemd transcript-worker journal.
If the alert identifies a worker, restore the named systemd service first. Check whether a later retry already completed. The monitor clears a failed stage when a later successful event for that upload and stage exists, when a newer run for the upload reaches ready or stored, or when every notification belonging to the affected run has subsequently reached sent. This preserves the failed event as audit history without keeping a recovered upload in an outage state.
Repair the dependency or use the portal's normal redeploy/retry path. Do not ask a pastor to upload again for a Mac, timer, playlist, Drive, or provider outage.
Leave the incident state file intact. The next two checks verify the cleared condition and the monitor sends the recovery message automatically.

For propresenter_playlist_api_timeout, inspect the broadcast-gfx ProPresenter API and configured app version. For propresenter_playlist_store_timeout, keep ProPresenter closed, run deep readiness, then redeploy the existing upload. A successful redeploy sends a PROCESSING FIXED production report; downstream worship and Drive steps that were not reached are no longer described as independent computer failures.

Before planned work that is expected to interrupt a receiver, enable a bounded maintenance window. This suppresses new outage email and reports the public service as degraded, but it does not stop checks or erase incidents. The window must be between 15 minutes and 24 hours and expires automatically:

sudo systemd-run --quiet --wait --pipe --collect \
  --unit=riveroaks-workflow-health-production-maintenance \
  --uid=www-data \
  --property=EnvironmentFile=/etc/riveroaks-portal/production.env \
  --setenv=WORKFLOW_ALERT_ENVIRONMENT=production \
  --setenv=WORKFLOW_ALERT_PORTAL_BASE_URL=https://portal.ro.church \
  /opt/riveroaks-service-transcripts/bin/python \
  /usr/share/nginx/portal/scripts/workflow-health-watch.py \
  --maintenance-minutes 60 --maintenance-reason planned_receiver_work

Clear it early with the same transient-unit boundary and --clear-maintenance. Do not edit the maintenance JSON by hand.

If the monitor itself reports database or schema failure, validate the deployed migrations and protected environment-file permissions before investigating individual workflow rows. If one transcript Mac is unavailable, processing can continue on the other Mac, but the degraded host still needs attention. If the complete Mac pool is unavailable, no local transcript job can run.

A changed or invalid managed ProPresenter playlist slot is a safety failure, not a transient outage. Restore or remap the slot before using Redeploy; the worker intentionally does not keep rewriting an unexpected slot. Theme mapping and delivery-deferred alerts also require an administrator action and remain active until a newer run clears the condition.

For propresenter_playlist_persistence_timeout, leave ProPresenter available, confirm its playlist store is writable, and redeploy through the portal. Do not quit or terminate ProPresenter immediately after an API update. Successful placement requires both an API readback and an observed on-disk playlist-store change before an automation-owned process is closed.

August 12, 2026 production acceptance

The live rollout completed the following checks:

the development content-free readiness endpoint returned HTTP 200;
a controlled development alert was accepted by MailerSend with HTTP 202 and arrived in Gmail;
Uptime Kuma monitor 15 was configured with the documented interval, retries, keyword, timeout, HTTP range, and NTFY route;
a normal portal redeploy completed lower-third generation, both Mac deliveries, render verification, Drive preview publishing, and user/admin notifications;
the deep canary detected the non-persistent playlist update instead of reporting a false ready state;
the UUID-addressed repair survived a complete ProPresenter close/reopen;
the production alert and recovery messages were both accepted by MailerSend;
the deep playlist incident recovered without editing monitor state.

The same rollout raised a real storage_warning: the portal root filesystem was approximately 90% utilized with about 6.7 GiB available. This is expected preemptive behavior. Reduce utilization or extend the volume; do not weaken the threshold merely to make the readiness status green.

Controlled tests

Preview a development synthetic alert without sending email or changing live state. Use systemd-run for the controlled commands as well; never source or print the environment file in an operator shell.

sudo systemd-run --quiet --wait --pipe --collect \
  --unit=riveroaks-workflow-health-dev-dry-run \
  --uid=www-data \
  --property=EnvironmentFile=/etc/riveroaks-portal/dev.env \
  --setenv=WORKFLOW_ALERT_ENVIRONMENT=development \
  --setenv=WORKFLOW_ALERT_PORTAL_BASE_URL=https://dev.portal.ro.church \
  --setenv=WORKFLOW_ALERT_TRANSCRIPTS_ENABLED=0 \
  --setenv=WORKFLOW_ALERT_TRANSCRIPT_TIMER_UNIT=riveroaks-service-transcripts-dev.timer \
  --setenv=WORKFLOW_ALERT_TRANSCRIPT_SERVICE_UNIT=riveroaks-service-transcripts-dev.service \
  /opt/riveroaks-service-transcripts/bin/python \
  /usr/share/nginx/html/dev-portal/scripts/workflow-health-watch.py \
  --dry-run --scenario sermon-failure

Supported scenarios are healthy, recovery, sermon-failure, transcript-failure, timer-stale, and mac-pool-down. A controlled email test must name its recipient explicitly and never changes live monitor state:

sudo systemd-run --quiet --wait --pipe --collect \
  --unit=riveroaks-workflow-health-dev-test-send \
  --uid=www-data \
  --property=EnvironmentFile=/etc/riveroaks-portal/dev.env \
  --setenv=WORKFLOW_ALERT_ENVIRONMENT=development \
  --setenv=WORKFLOW_ALERT_PORTAL_BASE_URL=https://dev.portal.ro.church \
  --setenv=WORKFLOW_ALERT_TRANSCRIPTS_ENABLED=0 \
  --setenv=WORKFLOW_ALERT_TRANSCRIPT_TIMER_UNIT=riveroaks-service-transcripts-dev.timer \
  --setenv=WORKFLOW_ALERT_TRANSCRIPT_SERVICE_UNIT=riveroaks-service-transcripts-dev.service \
  /opt/riveroaks-service-transcripts/bin/python \
  /usr/share/nginx/html/dev-portal/scripts/workflow-health-watch.py \
  --scenario transcript-failure --test-send cgood@riveroaks.org

sudo systemd-run --quiet --wait --pipe --collect \
  --unit=riveroaks-workflow-health-dev-recovery-test-send \
  --uid=www-data \
  --property=EnvironmentFile=/etc/riveroaks-portal/dev.env \
  --setenv=WORKFLOW_ALERT_ENVIRONMENT=development \
  --setenv=WORKFLOW_ALERT_PORTAL_BASE_URL=https://dev.portal.ro.church \
  --setenv=WORKFLOW_ALERT_TRANSCRIPTS_ENABLED=0 \
  --setenv=WORKFLOW_ALERT_TRANSCRIPT_TIMER_UNIT=riveroaks-service-transcripts-dev.timer \
  --setenv=WORKFLOW_ALERT_TRANSCRIPT_SERVICE_UNIT=riveroaks-service-transcripts-dev.service \
  /opt/riveroaks-service-transcripts/bin/python \
  /usr/share/nginx/html/dev-portal/scripts/workflow-health-watch.py \
  --scenario recovery --test-send cgood@riveroaks.org

Test messages are prefixed CONTROLLED TEST. Run them in development unless a production-specific test has been approved. A real incident test should use an isolated fixture and must not alter sermon content, transcript text, live queue ownership, timers, Mac authorization, or delivery artifacts.