Skip to content

Repositories, deployment, and change guide

This page records declared release behavior at the reviewed commits. Re-read current CI rules before operating a lane; a green source pipeline does not by itself establish device activation or successful background processing.

Repository map and release lanes

Repository Integration / release path Source
riveroaks/portal Feature/bug MR → dev → development origin; accepted dev → main promotion → production origin portal · .gitlab-ci.yml, portal · .gitlab-ci.yml
riveroaks/portal-api MR → dev, automatic plan/apply/smoke; reviewed promotion to main, production apply requires gate portal-api · CONTRIBUTING.md, portal-api · .gitlab-ci.yml
riveroaks/portal-ios Development work targets dev; protected-dev TestFlight and Portal-iPad Jamf jobs are automatic. Standalone dashboard and Utility releases are manual; the Utility expiry schedule cannot deploy. Release config alone is not App Store publication portal-ios · CONTRIBUTING.md, portal-ios · .gitlab-ci.yml, portal-ios · Configuration/Release.xcconfig
riveroaks/infra main owns host manifests/scripts; deployment jobs are target-specific and gated. Inspect the selected job before running it infra · .gitlab-ci.yml, infra · docs/deploy/gitlab-ci.md
riveroaks/infrastructure/lyric-follower main Linux validation/wheel; operator-controlled Mac signing/Jamf activation. Observer host deployment is a separate infra job infrastructure/lyric-follower · .gitlab-ci.yml, infrastructure/lyric-follower · docs/deployment-pipeline.md, infra · deploy/scripts/deploy-lyric-follower-shadow.sh
riveroaks/infrastructure/service-timeline main validation/wheel; attended deployment and acceptance, no automatic deploy infrastructure/service-timeline · .gitlab-ci.yml, infrastructure/service-timeline · README.md
Forms, Email, Short Links Independent main pipelines with dev/production IaC and protected/manual production gates; not a Portal dev→main promotion cloudflare-forms · .gitlab-ci.yml, cloudflare-email · .gitlab-ci.yml, cloudflare-short-links · .gitlab-ci.yml
riveroaks/docs Feature/bug MR → strict validation → main; translation, static build, Pages deployment, private deploy, image publication docs · .gitlab-ci.yml, docs · README.md

Private docs image promotion

The current private docs vhost proxies to the cluster. Docs main CI refreshes an immutable local rollback release and publishes a new static-site image; cluster-image.txt supplies the digest. A reviewed riveroaks/infra MR must update the docs Deployment's pinned digest before Flux rolls the candidate out. A successful deploy_internal job alone does not prove the private hostname is serving its new content. Verify a changed page after image promotion. Keep the previous digest for an app-only rollback through the same GitOps path. Sources: docs · tools/publish-cluster-image.sh, docs · README.md, infra · k3s/gitops/apps/docs/workload.yaml, infra · nginx/hosts/rocc-db/sites-enabled/docs.riveroaks.cc.conf.

Releasing a change that spans repositories

  1. Implement and deploy additive PHP adapter/schema changes through Portal dev. Confirm migration apply/check and runtime health.
  2. Update the API catalog/edge behavior through API dev; verify edge and private origin smoke checks and authenticated/RBAC fixtures.
  3. Integrate the native client on iOS dev. Check its configured hostname pair, decoding fixtures and actual release jobs before merging: dev can reach managed campus iPads automatically.
  4. Accept on development with controlled data. Budget data can be real production ledger data despite the development hostname.
  5. For an approved web/API production release, promote the Portal origin before the API; inspect the production plan and gate. Native production release is a separate owner-approved process.

This order is the recommended compatibility sequence derived from the adapter boundary and release controls. Sources: portal · .gitlab-ci.yml, portal · docs/technology-budget-sql-ledger.md, portal-api · docs/MIGRATION.md, portal-api · .gitlab-ci.yml, portal-ios · .gitlab-ci.yml.

Activation and rollback boundaries

Component Recovery boundary Runbook / implementation
API Re-deploy prior reviewed version through the lane; preserve Worker/tunnel resources and inspect schema compatibility portal-api · docs/OPERATIONS.md, portal-api · infra/modules/portal-platform/main.tf
Portal Reconcile via CI and required migrations; do not blindly reverse additive schema. Token/schema mismatch has caused sign-in failure portal · .gitlab-ci.yml, portal · docs/PRODUCTION_API_SESSION_INCIDENT_2026-09-08.md
PHP lyric daemons Copying code does not replace running processes; guarded activation restarts and checks services/socket portal · .gitlab/activate-lyrics-services.sh
Lyric shadow Infra script pins candidate/previous image IDs, retains previous unit and attempts rollback on failed activation infra · deploy/scripts/deploy-lyric-follower-shadow.sh
Capture Mac Retain signed previous package and compatible config; idle/drained activation then verify the running version, not only installed files infrastructure/lyric-follower · docs/deployment-pipeline.md
iPad Verify Jamf install state and device acceptance separately; release evidence and target scope matter portal-ios · docs/JAMF.md, portal-ios · docs/DASHBOARD_INSTALL_VERIFICATION.md
Docs private site Immutable release directory + atomic symlink; health/home failures restore previous release docs · tools/deploy-internal.sh

Where to look when something is wrong

The expected impacts below follow the implementation dependency graph. Start with read-only health and the matching lane; follow the linked runbook before restarts, replay, migrations or deployment.

Symptom / dependency Evidence and first checks Runbook
All staff API calls fail Compare release/request ID, Worker liveness, then private-origin readiness; liveness alone does not prove SQL or every adapter portal-api · src/index.ts, portal-api · docs/OPERATIONS.md
Sign-in/refresh fails Distinguish invalid_grant from origin 503; check adapter/schema pair and configured API/web origins portal · dashboard/api/mobile/v1/token.php, portal · docs/PRODUCTION_API_SESSION_INCIDENT_2026-09-08.md, portal-ios · docs/AUTHENTICATION.md
One resource fails Correlate request ID with origin logs, migration and resource-specific permission checks portal · docs/PORTAL_OBSERVABILITY.md, portal-api · docs/REGRESSION-TESTING.md
Ticket socket fails Generic invalidation/polling fallback is separate from authoritative ticket fetches portal-api · docs/TICKETS_LIVE.md, portal · docs/ticket-portal-live-updates.md
Tasks gateway fails Portal identity then Tasks service binding/assertion; assertion rejection is 502, not a user-token failure portal-api · src/index.ts, portal-api · docs/TASKS.md
Lyrics freeze Check ProPresenter observations/cache freshness, poller and socket independently; heartbeat is not fresh lyrics portal · lyrics/poll_propresenter.php, portal · lyrics/server.php, infra · portal/hosts/rocc-db/files/usr/local/sbin/riveroaks-lyrics-watchdog
Comms/panic stale Check branch/deployed publisher provenance, ingest result, RAM snapshot age, socket opt-in and REST fallback; do not copy transcript text portal · dashboard/api/mobile/v1/lib/comms.php, portal · lyrics/docs/comms-push.md, portal-ios · RiverOaksServiceDashboard/Models/DashboardModels.swift
iPad panel absent Test that panel's configured LAN feed; local network permission, Wi-Fi/VPN and freshness are independent of staff sign-in portal-ios · docs/IPAD_SERVICE_DASHBOARD.md, portal-ios · docs/QSYS_ROOM_CONTROLS.md
Alerts/push absent Publisher policy/RBAC/preferences, then ingest and APNs worker; foreground fetch and APNs are separate paths portal · docs/mobile-alerts.md, infra · docs/alerting/qsys-portal-alerts.md, infra · docs/alerting/grafana-portal-adapter.md
Capture schedule missing Feed generation, bearer route/backend, cached feed and Mac selected schedule infrastructure/lyric-follower · docs/planning-center-recording-feed.md
Timeline backlog Local spool capacity, mTLS/HMAC, collector readiness and sequence gap; retain ordered replay infrastructure/service-timeline · docs/runbook.md
Sermon delivery stuck Outbox/worker/destination state; preserve idempotency and service-safe receiver behavior portal · docs/SERMON_WORKFLOW_ALERT_RECOVERY.md, portal · docs/SERMON_PROPRESENTER_LIVE_DEPLOYMENT.md
Files unavailable Storage source/mount, delivery contract and storage-health worker portal · docs/files-storage-reliability.md, infra · docs/runbooks/multitracks-files-delivery.md
Origin secret provider unavailable Recover through the scoped runtime procedure; do not print runtime secret files infra · docs/deploy/portal-openbao-runtime-recovery.md
Form/email/short-link failure Independent Worker storage/queue/quota/auth lane; central API health may still pass cloudflare-forms · docs/OPERATIONS.md, cloudflare-email · docs/OPERATIONS.md, cloudflare-short-links · README.md
Docs not published Validation then translation/build/Pages, then private deploy/image jobs; image publication is not Flux promotion docs · .gitlab-ci.yml, docs · README.md, Docs hosting

Documentation release lane

MR/branch CI runs release-retention checks and mkdocs build --strict. Main's translation/build job publishes Pages and commits generated Spanish siblings; the private job fetches main, rebuilds strictly, activates the immutable static release and publishes a private registry image. The image digest needs a separate infra promotion; this documentation task does not change cluster routing or workloads. Sources: docs · .gitlab-ci.yml, docs · tools/commit-translations.sh, docs · tools/deploy-internal.sh, docs · tools/publish-cluster-image.sh, docs · README.md.

Not verified in this review

Source inspection does not prove live hostname routing, a capture trial's running version, device acceptance, current publisher commissioning or public socket filtering. The exact source/deployment distinction and known contradictions are recorded in Source snapshots.