Skip to content

Lyrics and service systems

Live lyric display pipeline

propresenter-poller reads ProPresenter slide/presentation state and writes the shared lyric cache; lyrics-websocket reads that cache and broadcasts changes. The poller also supplies countdown fields. The cache writer uses a temporary file and rename to avoid partial-reader JSON. Sources: portal · lyrics/poll_propresenter.php, portal · lyrics/countdown_poller.php, portal · lyrics/cache_writer.php, portal · lyrics/server.php.

Systemd runs both PHP daemons from the production Portal checkout. Their cache drop-ins select /run/riveroaks-lyrics; the broadcaster listens on all interfaces at 8081. Public/static displays and LAN/iPad clients share that broadcaster, while nginx supplies local /ws proxy routes. Sources: infra · portal/hosts/rocc-db/files/etc/systemd/system/lyrics-websocket.service, infra · portal/hosts/rocc-db/files/etc/systemd/system/lyrics-websocket.service.d/lyrics-cache.conf, infra · portal/hosts/rocc-db/files/etc/systemd/system/propresenter-poller.service.d/lyrics-cache.conf, portal · lyrics/server.php, infra · nginx/hosts/rocc-db/sites-enabled/lyrics.riveroaks.cc.conf.

The normal broadcaster is receive-only; inbound aligner-control writes require the explicit demo opt-in. A socket heartbeat proves broadcaster activity, not fresh ProPresenter data. The separate watchdog checks pipeline health and can restart a failed lyric daemon; inspect its bounded recovery/status logic before manual intervention. Sources: portal · lyrics/server.php, infra · portal/hosts/rocc-db/files/usr/local/sbin/riveroaks-lyrics-watchdog.

Read-only ProPresenter proxy for iPads

The private lyric vhost exposes exact GET proxy routes for timers, playlists, slide index and presentation state. Some routes named focus can alter focus on the ProPresenter side despite using GET; the proxy is narrow, not a general permission to call arbitrary ProPresenter endpoints. These routes stay behind the private source ACL. Source: infra · nginx/hosts/rocc-db/sites-enabled/lyrics.riveroaks.cc.conf.

Song viewer and Planning Center adapter

The PHP adapter retrieves PCO service types/plans/elements server-side. The LAN lyric vhost allows the exact adapter for dashboards; the interactive songs viewer has its own Authentik-protected hostname. PCO credentials are not put in the iPad configuration. Sources: portal · lyrics/songs/api.php, portal · lyrics/songs/service_elements.php, infra · nginx/hosts/rocc-db/sites-enabled/lyrics.riveroaks.cc.conf, infra · nginx/hosts/rocc-db/sites-enabled/songs.riveroaks.cc.conf.

Lyric follower (riveroaks/infrastructure/lyric-follower)

The persistent shadow service consumes local lyric WebSocket/ProPresenter state and Q-SYS LTC over QRC. The unit runs a constrained, read-only-root container, binds the observer at loopback 8094 and playback support at 8095, and mounts a persistent state directory. This is a separate application from the PHP display poller. Sources: infrastructure/lyric-follower · systemd/ro-lyric-shadow.service, infrastructure/lyric-follower · src/lyric_follower/rehearsal.py, infrastructure/lyric-follower · src/lyric_follower/qrc.py.

LIVE_TRIGGER_ENABLED = False blocks persistent live triggering. OFF, SHADOW and ADVISORY are the current safe operator modes; attended replay has a separate controlled route. The safety model's Assisted/Auto ladder describes future qualification, not active autonomous service operation. Sources: infrastructure/lyric-follower · src/lyric_follower/pp_agent.py, infrastructure/lyric-follower · src/lyric_follower/assisted_replay.py, infrastructure/lyric-follower · src/lyric_follower/rehearsal.py, infrastructure/lyric-follower · docs/safety-model.md.

Browser control passes through Authentik; narrow LAN controls and capture-feed access are separate exceptions. See trust boundaries. Source: infra · nginx/hosts/rocc-db/sites-enabled/lyrics.riveroaks.cc.conf.

The lyric-text decision changed during this project: ADR 0008 records approval to retain text for training, and lyric_text.py implements storage helpers. Do not use the event spine's hash-only contract as a blanket statement about all follower/capture storage. Sources: infrastructure/lyric-follower · docs/adr/0008-store-lyric-text-for-training.md, infrastructure/lyric-follower · src/lyric_follower/lyric_text.py.

Default-off hybrid evaluator

The current main branch also contains a hybrid ASR/timecode follower library and offline evaluation CLI. It returns hold/track/propose decisions and has no actuator or network transport. Configuration rejects enabled production use; all decisions are ineligible for live action. Rehearsal and persistent playback do not import this evaluator. Merging the library does not enable Assisted or Auto operation. Sources: infrastructure/lyric-follower · src/lyric_follower/hybrid_follower.py, infrastructure/lyric-follower · docs/hybrid-dev-test.md.

The NDI follow-up likewise distinguishes native development qualification from attended capture-Mac installation. Fixture/CI success does not establish a production recorder version or authorize replacing a separately held capture trial. Source: infrastructure/lyric-follower · docs/NDI_DAYTIME_ACCEPTANCE.md.

Worship Data Capture

The same repo contains a native macOS audio/NDI recorder. It obtains filtered PCO recording slots through the capture scheduling service at loopback 8096 behind a dedicated bearer-protected nginx route; the feed generator stays server-side. It does not use the canonical staff API for schedule reads. Sources: infrastructure/lyric-follower · macos/WorshipDataCaptureApp/Sources/WorshipDataCaptureApp/PlanningCenterCatalog.swift, infrastructure/lyric-follower · server/planning_center_api.py, infrastructure/lyric-follower · server/planning-center-feed.php, infra · nginx/hosts/rocc-db/sites-enabled/lyrics.riveroaks.cc.conf.

Linux CI builds/tests the wheel; native qualification, signing, package release, Jamf scope, safe activation and runtime verification are separate operator steps. Installing a package does not establish the running process version. Source: infrastructure/lyric-follower · .gitlab-ci.yml, infrastructure/lyric-follower · docs/deployment-pipeline.md.

Comms and panic feeds

The transcription feature branch contains the producer: it signs bounded snapshots for the selected API lane, publishes changed/latest state and heartbeats, and continues local transcription if publication fails. This source is feature-branch evidence, not proof of deployment from main. Sources: infrastructure/lyric-follower · macos/WorshipDataCaptureApp/Sources/WorshipDataCaptureApp/CommsPublisher.swift, infrastructure/lyric-follower · macos/WorshipDataCaptureApp/Sources/WorshipDataCaptureCore/Transcription.swift, infrastructure/lyric-follower · docs/deployment-pipeline.md.

The API routes comms/panic ingest to the dev Portal. PHP verifies HMAC and ordering and atomically writes /run/riveroaks-comms-<lane>/{state,panic}.json. The private nginx aliases point at development snapshots. Sources: portal-api · src/contract.ts, portal · dashboard/api/mobile/v1/comms-ingest.php, portal · dashboard/api/mobile/v1/panic-ingest.php, portal · dashboard/api/mobile/v1/lib/comms.php, infra · nginx/hosts/rocc-db/sites-enabled/lyrics.riveroaks.cc.conf.

The broadcaster sends type=comms, kind=comms|panic only to opt-in socket clients; iPad REST reads remain a fallback. Snapshots older than 15 seconds are marked offline; final lines can remain visible. Opt-in is not access control: see the public-edge verification gap. Sources: portal · lyrics/comms_feed.php, portal-ios · RiverOaksServiceDashboard/Models/DashboardModels.swift, portal-ios · RiverOaksServiceDashboard/ViewModel/DashboardViewModel.swift.

Service timeline (riveroaks/infrastructure/service-timeline)

The Phase 0 event spine writes observations to local SQLite WAL before network shipping. Ordered batches cross mTLS plus HMAC into a collector and MariaDB. The event log is append-only; producer checkpoints are mutable and updated in the same transaction as event inserts. Duplicates are checked and sequence gaps rejected. Sources: infrastructure/service-timeline · src/service_timeline/spool.py, infrastructure/service-timeline · src/service_timeline/client.py, infrastructure/service-timeline · src/service_timeline/signing.py, infrastructure/service-timeline · src/service_timeline/storage.py, infrastructure/service-timeline · database/roles.sql.example.

The ProPresenter producer is read-only and records cue identity/text hashes. Media registration and operator markers have their own local spool commands. There is no central API connection or automatic deployment stage in the reviewed implementation. Sources: infrastructure/service-timeline · src/service_timeline/propresenter.py, infrastructure/service-timeline · src/service_timeline/media.py, infrastructure/service-timeline · src/service_timeline/marker.py, infrastructure/service-timeline · .gitlab-ci.yml, infrastructure/service-timeline · README.md.

AV monitoring

The dashboard reads acoustics and timecode through the Service Monitor LAN feeds. The separate AV signal-health collector provides a Q-SYS acoustic bridge; Q-SYS/Node-RED can publish signed Portal alerts. These monitoring paths do not put the central API in the synchronous lyric-control loop. Sources: portal-ios · docs/IPAD_SERVICE_DASHBOARD.md, infrastructure/av-signal-health · README.md, infrastructure/av-signal-health · docs/qsys-acoustics-rta-bridge.md, infra · nodered/instances/telegram/qsys-portal-alert.js, infrastructure/lyric-follower · docs/safety-model.md.

What depends on what

Failure impact is an inference from the source paths above, not a performed outage test. See failure matrix and runbooks.

Dependency lost Expected impact
Central API/VPC path Staff native calls and API-backed web pages fail; new comms publication fails. Local lyric display and local control can continue if their LAN dependencies remain healthy
rocc-db/PHP/SQL Broad shared failure domain for Portal data/auth and local lyric/control services; independent Mac/Pi processes may remain running
ProPresenter New live lyric/slide/countdown observations fail; transport heartbeat alone can still look alive
PCO New plan/schedule retrieval fails; cache/fallback behavior depends on the individual adapter, so do not promise universal offline operation
Collector/network for timeline Producers can spool locally within configured capacity; drain/replay requires collector recovery without skipping gaps
APNs Foreground alert API remains a separate transport; background notifications can be delayed/absent