Repositories, deployment, and change guide
This page records declared release behavior at the reviewed commits. Re-read current CI rules before operating a lane; a green source pipeline does not by itself establish device activation or successful background processing.
Repository map and release lanes
| Repository | Integration / release path | Source |
|---|---|---|
riveroaks/portal |
Feature/bug MR → dev → development origin; accepted dev → main promotion → production origin |
portal · .gitlab-ci.yml, portal · .gitlab-ci.yml |
riveroaks/portal-api |
MR → dev, automatic plan/apply/smoke; reviewed promotion to main, production apply requires gate |
portal-api · CONTRIBUTING.md, portal-api · .gitlab-ci.yml |
riveroaks/portal-ios |
Development work targets dev; protected-dev TestFlight and Portal-iPad Jamf jobs are automatic. Standalone dashboard and Utility releases are manual; the Utility expiry schedule cannot deploy. Release config alone is not App Store publication |
portal-ios · CONTRIBUTING.md, portal-ios · .gitlab-ci.yml, portal-ios · Configuration/Release.xcconfig |
riveroaks/infra |
main owns host manifests/scripts; deployment jobs are target-specific and gated. Inspect the selected job before running it |
infra · .gitlab-ci.yml, infra · docs/deploy/gitlab-ci.md |
riveroaks/infrastructure/lyric-follower |
main Linux validation/wheel; operator-controlled Mac signing/Jamf activation. Observer host deployment is a separate infra job |
infrastructure/lyric-follower · .gitlab-ci.yml, infrastructure/lyric-follower · docs/deployment-pipeline.md, infra · deploy/scripts/deploy-lyric-follower-shadow.sh |
riveroaks/infrastructure/service-timeline |
main validation/wheel; attended deployment and acceptance, no automatic deploy |
infrastructure/service-timeline · .gitlab-ci.yml, infrastructure/service-timeline · README.md |
| Forms, Email, Short Links | Independent main pipelines with dev/production IaC and protected/manual production gates; not a Portal dev→main promotion |
cloudflare-forms · .gitlab-ci.yml, cloudflare-email · .gitlab-ci.yml, cloudflare-short-links · .gitlab-ci.yml |
riveroaks/docs |
Feature/bug MR → strict validation → main; translation, static build, Pages deployment, private deploy, image publication |
docs · .gitlab-ci.yml, docs · README.md |
Private docs image promotion
The current private docs vhost proxies to the cluster. Docs main CI refreshes
an immutable local rollback release and publishes a new static-site image;
cluster-image.txt supplies the digest. A reviewed riveroaks/infra MR must
update the docs Deployment's pinned digest before Flux rolls the candidate out.
A successful deploy_internal job alone does not prove the private hostname is
serving its new content. Verify a changed page after image promotion. Keep the
previous digest for an app-only rollback through the same GitOps path.
Sources:
docs · tools/publish-cluster-image.sh,
docs · README.md,
infra · k3s/gitops/apps/docs/workload.yaml,
infra · nginx/hosts/rocc-db/sites-enabled/docs.riveroaks.cc.conf.
Releasing a change that spans repositories
- Implement and deploy additive PHP adapter/schema changes through Portal
dev. Confirm migration apply/check and runtime health. - Update the API catalog/edge behavior through API
dev; verify edge and private origin smoke checks and authenticated/RBAC fixtures. - Integrate the native client on iOS
dev. Check its configured hostname pair, decoding fixtures and actual release jobs before merging:devcan reach managed campus iPads automatically. - Accept on development with controlled data. Budget data can be real production ledger data despite the development hostname.
- For an approved web/API production release, promote the Portal origin before the API; inspect the production plan and gate. Native production release is a separate owner-approved process.
This order is the recommended compatibility sequence derived from the adapter
boundary and release controls. Sources: portal · .gitlab-ci.yml,
portal · docs/technology-budget-sql-ledger.md, portal-api · docs/MIGRATION.md,
portal-api · .gitlab-ci.yml, portal-ios · .gitlab-ci.yml.
Activation and rollback boundaries
| Component | Recovery boundary | Runbook / implementation |
|---|---|---|
| API | Re-deploy prior reviewed version through the lane; preserve Worker/tunnel resources and inspect schema compatibility | portal-api · docs/OPERATIONS.md, portal-api · infra/modules/portal-platform/main.tf |
| Portal | Reconcile via CI and required migrations; do not blindly reverse additive schema. Token/schema mismatch has caused sign-in failure | portal · .gitlab-ci.yml, portal · docs/PRODUCTION_API_SESSION_INCIDENT_2026-09-08.md |
| PHP lyric daemons | Copying code does not replace running processes; guarded activation restarts and checks services/socket | portal · .gitlab/activate-lyrics-services.sh |
| Lyric shadow | Infra script pins candidate/previous image IDs, retains previous unit and attempts rollback on failed activation | infra · deploy/scripts/deploy-lyric-follower-shadow.sh |
| Capture Mac | Retain signed previous package and compatible config; idle/drained activation then verify the running version, not only installed files | infrastructure/lyric-follower · docs/deployment-pipeline.md |
| iPad | Verify Jamf install state and device acceptance separately; release evidence and target scope matter | portal-ios · docs/JAMF.md, portal-ios · docs/DASHBOARD_INSTALL_VERIFICATION.md |
| Docs private site | Immutable release directory + atomic symlink; health/home failures restore previous release | docs · tools/deploy-internal.sh |
Where to look when something is wrong
The expected impacts below follow the implementation dependency graph. Start with read-only health and the matching lane; follow the linked runbook before restarts, replay, migrations or deployment.
| Symptom / dependency | Evidence and first checks | Runbook |
|---|---|---|
| All staff API calls fail | Compare release/request ID, Worker liveness, then private-origin readiness; liveness alone does not prove SQL or every adapter | portal-api · src/index.ts, portal-api · docs/OPERATIONS.md |
| Sign-in/refresh fails | Distinguish invalid_grant from origin 503; check adapter/schema pair and configured API/web origins |
portal · dashboard/api/mobile/v1/token.php, portal · docs/PRODUCTION_API_SESSION_INCIDENT_2026-09-08.md, portal-ios · docs/AUTHENTICATION.md |
| One resource fails | Correlate request ID with origin logs, migration and resource-specific permission checks | portal · docs/PORTAL_OBSERVABILITY.md, portal-api · docs/REGRESSION-TESTING.md |
| Ticket socket fails | Generic invalidation/polling fallback is separate from authoritative ticket fetches | portal-api · docs/TICKETS_LIVE.md, portal · docs/ticket-portal-live-updates.md |
| Tasks gateway fails | Portal identity then Tasks service binding/assertion; assertion rejection is 502, not a user-token failure |
portal-api · src/index.ts, portal-api · docs/TASKS.md |
| Lyrics freeze | Check ProPresenter observations/cache freshness, poller and socket independently; heartbeat is not fresh lyrics | portal · lyrics/poll_propresenter.php, portal · lyrics/server.php, infra · portal/hosts/rocc-db/files/usr/local/sbin/riveroaks-lyrics-watchdog |
| Comms/panic stale | Check branch/deployed publisher provenance, ingest result, RAM snapshot age, socket opt-in and REST fallback; do not copy transcript text | portal · dashboard/api/mobile/v1/lib/comms.php, portal · lyrics/docs/comms-push.md, portal-ios · RiverOaksServiceDashboard/Models/DashboardModels.swift |
| iPad panel absent | Test that panel's configured LAN feed; local network permission, Wi-Fi/VPN and freshness are independent of staff sign-in | portal-ios · docs/IPAD_SERVICE_DASHBOARD.md, portal-ios · docs/QSYS_ROOM_CONTROLS.md |
| Alerts/push absent | Publisher policy/RBAC/preferences, then ingest and APNs worker; foreground fetch and APNs are separate paths | portal · docs/mobile-alerts.md, infra · docs/alerting/qsys-portal-alerts.md, infra · docs/alerting/grafana-portal-adapter.md |
| Capture schedule missing | Feed generation, bearer route/backend, cached feed and Mac selected schedule | infrastructure/lyric-follower · docs/planning-center-recording-feed.md |
| Timeline backlog | Local spool capacity, mTLS/HMAC, collector readiness and sequence gap; retain ordered replay | infrastructure/service-timeline · docs/runbook.md |
| Sermon delivery stuck | Outbox/worker/destination state; preserve idempotency and service-safe receiver behavior | portal · docs/SERMON_WORKFLOW_ALERT_RECOVERY.md, portal · docs/SERMON_PROPRESENTER_LIVE_DEPLOYMENT.md |
| Files unavailable | Storage source/mount, delivery contract and storage-health worker | portal · docs/files-storage-reliability.md, infra · docs/runbooks/multitracks-files-delivery.md |
| Origin secret provider unavailable | Recover through the scoped runtime procedure; do not print runtime secret files | infra · docs/deploy/portal-openbao-runtime-recovery.md |
| Form/email/short-link failure | Independent Worker storage/queue/quota/auth lane; central API health may still pass | cloudflare-forms · docs/OPERATIONS.md, cloudflare-email · docs/OPERATIONS.md, cloudflare-short-links · README.md |
| Docs not published | Validation then translation/build/Pages, then private deploy/image jobs; image publication is not Flux promotion | docs · .gitlab-ci.yml, docs · README.md, Docs hosting |
Documentation release lane
MR/branch CI runs release-retention checks and mkdocs build --strict. Main's
translation/build job publishes Pages and commits generated Spanish siblings;
the private job fetches main, rebuilds strictly, activates the immutable static
release and publishes a private registry image. The image digest needs a
separate infra promotion; this documentation task does not change cluster
routing or workloads. Sources: docs · .gitlab-ci.yml,
docs · tools/commit-translations.sh, docs · tools/deploy-internal.sh,
docs · tools/publish-cluster-image.sh, docs · README.md.
Not verified in this review
Source inspection does not prove live hostname routing, a capture trial's running version, device acceptance, current publisher commissioning or public socket filtering. The exact source/deployment distinction and known contradictions are recorded in Source snapshots.