Lyrics and service systems
Live lyric display pipeline
propresenter-poller reads ProPresenter slide/presentation state and writes the
shared lyric cache; lyrics-websocket reads that cache and broadcasts changes.
The poller also supplies countdown fields. The cache writer uses a temporary
file and rename to avoid partial-reader JSON. Sources:
portal · lyrics/poll_propresenter.php, portal · lyrics/countdown_poller.php,
portal · lyrics/cache_writer.php, portal · lyrics/server.php.
Systemd runs both PHP daemons from the production Portal checkout. Their cache
drop-ins select /run/riveroaks-lyrics; the broadcaster listens on all
interfaces at 8081. Public/static displays and LAN/iPad clients share that
broadcaster, while nginx supplies local /ws proxy routes. Sources:
infra · portal/hosts/rocc-db/files/etc/systemd/system/lyrics-websocket.service,
infra · portal/hosts/rocc-db/files/etc/systemd/system/lyrics-websocket.service.d/lyrics-cache.conf,
infra · portal/hosts/rocc-db/files/etc/systemd/system/propresenter-poller.service.d/lyrics-cache.conf,
portal · lyrics/server.php,
infra · nginx/hosts/rocc-db/sites-enabled/lyrics.riveroaks.cc.conf.
The normal broadcaster is receive-only; inbound aligner-control writes require
the explicit demo opt-in. A socket heartbeat proves broadcaster activity, not
fresh ProPresenter data. The separate watchdog checks pipeline health and can
restart a failed lyric daemon; inspect its bounded recovery/status logic before
manual intervention. Sources: portal · lyrics/server.php,
infra · portal/hosts/rocc-db/files/usr/local/sbin/riveroaks-lyrics-watchdog.
Read-only ProPresenter proxy for iPads
The private lyric vhost exposes exact GET proxy routes for timers, playlists,
slide index and presentation state. Some routes named focus can alter focus
on the ProPresenter side despite using GET; the proxy is narrow, not a general
permission to call arbitrary ProPresenter endpoints. These routes stay behind
the private source ACL. Source:
infra · nginx/hosts/rocc-db/sites-enabled/lyrics.riveroaks.cc.conf.
Song viewer and Planning Center adapter
The PHP adapter retrieves PCO service types/plans/elements server-side. The LAN
lyric vhost allows the exact adapter for dashboards; the interactive songs
viewer has its own Authentik-protected hostname. PCO credentials are not put in
the iPad configuration. Sources: portal · lyrics/songs/api.php,
portal · lyrics/songs/service_elements.php,
infra · nginx/hosts/rocc-db/sites-enabled/lyrics.riveroaks.cc.conf,
infra · nginx/hosts/rocc-db/sites-enabled/songs.riveroaks.cc.conf.
Lyric follower (riveroaks/infrastructure/lyric-follower)
The persistent shadow service consumes local lyric WebSocket/ProPresenter state
and Q-SYS LTC over QRC. The unit runs a constrained, read-only-root container,
binds the observer at loopback 8094 and playback support at 8095, and mounts a
persistent state directory. This is a separate application from the PHP display
poller. Sources: infrastructure/lyric-follower · systemd/ro-lyric-shadow.service,
infrastructure/lyric-follower · src/lyric_follower/rehearsal.py,
infrastructure/lyric-follower · src/lyric_follower/qrc.py.
LIVE_TRIGGER_ENABLED = False blocks persistent live triggering. OFF, SHADOW
and ADVISORY are the current safe operator modes; attended replay has a separate
controlled route. The safety model's Assisted/Auto ladder describes future
qualification, not active autonomous service operation. Sources:
infrastructure/lyric-follower · src/lyric_follower/pp_agent.py,
infrastructure/lyric-follower · src/lyric_follower/assisted_replay.py,
infrastructure/lyric-follower · src/lyric_follower/rehearsal.py, infrastructure/lyric-follower · docs/safety-model.md.
Browser control passes through Authentik; narrow LAN controls and capture-feed
access are separate exceptions. See trust boundaries.
Source: infra · nginx/hosts/rocc-db/sites-enabled/lyrics.riveroaks.cc.conf.
The lyric-text decision changed during this project: ADR 0008 records approval
to retain text for training, and lyric_text.py implements storage helpers.
Do not use the event spine's hash-only contract as a blanket statement about
all follower/capture storage. Sources:
infrastructure/lyric-follower · docs/adr/0008-store-lyric-text-for-training.md,
infrastructure/lyric-follower · src/lyric_follower/lyric_text.py.
Default-off hybrid evaluator
The current main branch also contains a hybrid ASR/timecode follower library
and offline evaluation CLI. It returns hold/track/propose decisions and has no
actuator or network transport. Configuration rejects enabled production use;
all decisions are ineligible for live action. Rehearsal and persistent playback
do not import this evaluator. Merging the library does not enable Assisted or
Auto operation. Sources:
infrastructure/lyric-follower · src/lyric_follower/hybrid_follower.py,
infrastructure/lyric-follower · docs/hybrid-dev-test.md.
The NDI follow-up likewise distinguishes native development qualification from
attended capture-Mac installation. Fixture/CI success does not establish a
production recorder version or authorize replacing a separately held capture
trial. Source:
infrastructure/lyric-follower · docs/NDI_DAYTIME_ACCEPTANCE.md.
Worship Data Capture
The same repo contains a native macOS audio/NDI recorder. It obtains filtered
PCO recording slots through the capture scheduling service at loopback 8096
behind a dedicated bearer-protected nginx route; the feed generator stays
server-side. It does not use the canonical staff API for schedule reads.
Sources: infrastructure/lyric-follower · macos/WorshipDataCaptureApp/Sources/WorshipDataCaptureApp/PlanningCenterCatalog.swift,
infrastructure/lyric-follower · server/planning_center_api.py,
infrastructure/lyric-follower · server/planning-center-feed.php,
infra · nginx/hosts/rocc-db/sites-enabled/lyrics.riveroaks.cc.conf.
Linux CI builds/tests the wheel; native qualification, signing, package release,
Jamf scope, safe activation and runtime verification are separate operator
steps. Installing a package does not establish the running process version.
Source: infrastructure/lyric-follower · .gitlab-ci.yml, infrastructure/lyric-follower · docs/deployment-pipeline.md.
Comms and panic feeds
The transcription feature branch contains the producer: it signs bounded
snapshots for the selected API lane, publishes changed/latest state and
heartbeats, and continues local transcription if publication fails. This source
is feature-branch evidence, not proof of deployment from main. Sources:
infrastructure/lyric-follower · macos/WorshipDataCaptureApp/Sources/WorshipDataCaptureApp/CommsPublisher.swift,
infrastructure/lyric-follower · macos/WorshipDataCaptureApp/Sources/WorshipDataCaptureCore/Transcription.swift,
infrastructure/lyric-follower · docs/deployment-pipeline.md.
The API routes comms/panic ingest to the dev Portal. PHP verifies HMAC and
ordering and atomically writes /run/riveroaks-comms-<lane>/{state,panic}.json.
The private nginx aliases point at development snapshots. Sources:
portal-api · src/contract.ts, portal · dashboard/api/mobile/v1/comms-ingest.php,
portal · dashboard/api/mobile/v1/panic-ingest.php,
portal · dashboard/api/mobile/v1/lib/comms.php,
infra · nginx/hosts/rocc-db/sites-enabled/lyrics.riveroaks.cc.conf.
The broadcaster sends type=comms, kind=comms|panic only to opt-in socket
clients; iPad REST reads remain a fallback. Snapshots older than 15 seconds are
marked offline; final lines can remain visible. Opt-in is not access control:
see the public-edge verification gap.
Sources: portal · lyrics/comms_feed.php,
portal-ios · RiverOaksServiceDashboard/Models/DashboardModels.swift,
portal-ios · RiverOaksServiceDashboard/ViewModel/DashboardViewModel.swift.
Service timeline (riveroaks/infrastructure/service-timeline)
The Phase 0 event spine writes observations to local SQLite WAL before network
shipping. Ordered batches cross mTLS plus HMAC into a collector and MariaDB.
The event log is append-only; producer checkpoints are mutable and updated in
the same transaction as event inserts. Duplicates are checked and sequence gaps
rejected. Sources: infrastructure/service-timeline · src/service_timeline/spool.py,
infrastructure/service-timeline · src/service_timeline/client.py,
infrastructure/service-timeline · src/service_timeline/signing.py,
infrastructure/service-timeline · src/service_timeline/storage.py,
infrastructure/service-timeline · database/roles.sql.example.
The ProPresenter producer is read-only and records cue identity/text hashes.
Media registration and operator markers have their own local spool commands.
There is no central API connection or automatic deployment stage in the reviewed
implementation. Sources: infrastructure/service-timeline · src/service_timeline/propresenter.py,
infrastructure/service-timeline · src/service_timeline/media.py,
infrastructure/service-timeline · src/service_timeline/marker.py,
infrastructure/service-timeline · .gitlab-ci.yml, infrastructure/service-timeline · README.md.
AV monitoring
The dashboard reads acoustics and timecode through the Service Monitor LAN
feeds. The separate AV signal-health collector provides a Q-SYS acoustic bridge;
Q-SYS/Node-RED can publish signed Portal alerts. These monitoring paths do not
put the central API in the synchronous lyric-control loop. Sources:
portal-ios · docs/IPAD_SERVICE_DASHBOARD.md,
infrastructure/av-signal-health · README.md,
infrastructure/av-signal-health · docs/qsys-acoustics-rta-bridge.md,
infra · nodered/instances/telegram/qsys-portal-alert.js,
infrastructure/lyric-follower · docs/safety-model.md.
What depends on what
Failure impact is an inference from the source paths above, not a performed outage test. See failure matrix and runbooks.
| Dependency lost | Expected impact |
|---|---|
| Central API/VPC path | Staff native calls and API-backed web pages fail; new comms publication fails. Local lyric display and local control can continue if their LAN dependencies remain healthy |
rocc-db/PHP/SQL |
Broad shared failure domain for Portal data/auth and local lyric/control services; independent Mac/Pi processes may remain running |
| ProPresenter | New live lyric/slide/countdown observations fail; transport heartbeat alone can still look alive |
| PCO | New plan/schedule retrieval fails; cache/fallback behavior depends on the individual adapter, so do not promise universal offline operation |
| Collector/network for timeline | Producers can spool locally within configured capacity; drain/replay requires collector recovery without skipping gaps |
| APNs | Foreground alert API remains a separate transport; background notifications can be delayed/absent |