Skip to content

Searchable service transcription

Updated August 11, 2026.

See Focused portal workflow alerts for automated job, lease, timer, Mac-pool, and recovery monitoring.

Status

The portal foundation and restricted local execution transport are implemented behind environment feature flags. They include the additive database schema, leased-job trust boundary, full-text Services search, time-coded viewer, append-only corrections, hashed access/search auditing, deterministic MPR PA source discovery, and two failover M4 workers. Seventeen real historical services now pass end to end across both Macs. Production staff search, administrator corrections, and the nightly worker timer were enabled on August 10 through portal MRs !436 and !437. Seventeen transcripts are available and one older version is superseded. The first controlled production run verified both Macs, found no duplicate work, and exited successfully. On August 11, both environment timers passed their first unattended acceptance runs. Production is now the sole nightly schedule owner; its two consumers still engage both Macs, while the development timer remains installed but disabled. Access and retention remain an explicit governance follow-up rather than an implicit release assumption.

The August 10 source-path blocker is resolved. The portal VM now mounts 10.200.24.2:/main-pool/bulk-storage/multitracks at /mnt/multitracks through the existing systemd automount with client-side ro options. The prior /etc/fstab is preserved as /etc/fstab.before-transcripts-20260810. This avoids making unprivileged storage-mgr LXC 109 privileged merely to run kernel NFS.

Infrastructure hardening follow-up is still required: the Proxmox host currently exposes the parent /main-pool export broadly as read-write. The transcript client is mounted read-only, but the server-side export should be inventoried and narrowed without disrupting its existing clients. This is an existing storage-boundary issue, not a reason to grant the Macs storage access; the Macs continue to receive only UUID-scoped audio from the portal VM.

Implemented portal foundation

The feature defaults fail-closed with:

  • SERVICES_TRANSCRIPTS_ENABLED=0, which controls transcript search, cards, and the transcript viewer;
  • SERVICES_TRANSCRIPT_CORRECTIONS_ENABLED=0, which separately controls the administrator correction form;
  • the dedicated services_transcripts.manage permission, granted to portal administrators by default rather than to every staff viewer.

Production explicitly sets the search, correction, and worker flags to 1. The environment file remains protected outside the repository.

The Services archive query can search original machine segments and reviewed corrections only when the feature flag and required tables are available. A matching service shows at most three bounded excerpts and links directly to the exact transcript moment. Results remain grouped by service and date. The full viewer groups immutable machine segments into readable paragraphs, labels the text as automatic, supports multipart MPR PA audio, and maps every paragraph timestamp to its part-relative time. Paragraph grouping is presentational and does not rewrite or merge stored segments. The viewer also has local word-based search, highlighted matches, previous/next navigation, and shareable query plus segment URLs. Corrections append a new row, retain the machine wording, record the reviewer and reason, and may supersede an earlier correction without deleting it.

Reviewed service markers

The transcript viewer supports a reviewed outline of major service moments. Portal administrators can anchor a marker to an existing immutable transcript segment and classify it as Worship, Sermon, Announcements, Prayer, Offering, or Other. Staff can then select the marker to seek the existing multipart MPR PA player to the correct source file and part-relative timestamp.

To add a marker:

  1. Open the service transcript and find the paragraph containing the major moment.
  2. Select Review beside that paragraph, then choose the exact timestamped transcript moment in the review dialog.
  3. Under Add to the service outline, select the moment type, optionally add a short staff-facing label, and save.
  4. Verify the new Major moments entry starts playback at the intended boundary.

The same shared review dialog contains correction controls. It replaces the former correction and marker forms repeated beneath every raw segment. Review controls are available only to users with services_transcripts.manage and are visually quiet until the paragraph is hovered or focused on larger screens.

Marker writes require services_transcripts.manage and the same session CSRF boundary used for reviewed transcript corrections. The server resolves the submitted segment against the latest available transcript and stores both the service-wide time and immutable multipart playback coordinates. Staff-entered labels are length-bounded and escaped when rendered.

Retiring a marker removes it from the current outline without deleting its audit record. The database retains its creator and creation time plus the administrator and time of retirement. Marker actions never change transcript wording, source audio, or an existing correction.

The portal worker core now provides:

  • one deduplicated job per source-manifest hash, workflow, model, and language;
  • ordered immutable source-part manifests;
  • conditional database leases with a random token stored only as SHA-256, expiry recovery, bounded retries, and FOR UPDATE SKIP LOCKED claiming;
  • strict result validation for schema version, job/source/model identity, language, audio duration, segment count, UTF-8 bytes, timestamps, multipart ordering, confidence, and total transcript size;
  • an atomic commit of immutable transcript metadata and ordered segments only while the submitting worker still owns an unexpired lease;
  • processing time, audio duration, real-time factor, optional inference cost, and bounded content-free error codes.

The command-line worker supports configuration validation, explicit-date or bounded-lookback enqueueing, one-job execution, and draining the queue with up to one independently leased consumer per configured Mac. Each consumer starts with a different preferred Mac and may fail over to another healthy worker. Actual claim/dispatch is additionally gated by SERVICES_TRANSCRIPT_WORKER_ENABLED=1; configuration validation remains safe while the gate is off.

Intended user experience

Authorized staff can search the existing Services archive for a word or phrase. A match will show the service date, a short transcript excerpt, and a time offset. Selecting the result opens the readable transcript at that exact segment, preserves the search terms, and highlights the meaningful words. The local reader ignores common filler words so its behavior remains aligned with the archive full-text index. Search permissions remain at least as restrictive as the current staff-only Services archive.

Machine transcripts are an index into the recording, not an authoritative record of what a person said. The interface must label them as automated, preserve the original machine result, and retain an audit trail for reviewed corrections.

Source selection

Use the mono/stereo PA program channel, not isolated vocal microphones. The current browser-preview job recognizes WAV names shaped like MPR PA or PA S, including numbered parts, and writes immutable derived files under Portal Playback/ without changing the WAV masters.

Source selection must record:

  • service-folder identity and date;
  • exact source relative path, byte size, modification time, and SHA-256;
  • detected channel label and part number;
  • selection rule/version and a confidence value;
  • whether a human approved an ambiguous selection.

If zero or multiple plausible PA channels remain after deterministic ranking, the job enters needs_source_review. It must not silently choose a channel.

Processing architecture

The portal VM is the orchestrator and system of record. The two idle M4 Pro Macs at 10.200.21.35 and 10.200.5.5 are an active/active execution pool, not two independent schedulers scanning storage.

  1. After service ingest is stable, the VM creates one deduplicated transcript job for the recording hash.
  2. A database lease assigns the job to one healthy Mac. The other Mac may take a different job but cannot process the same lease.
  3. The VM selects the existing immutable Portal Playback/MPR PA*.m4a derivative, verifies its path, size, and SHA-256 against the job manifest, and leaves WAV masters read-only.
  4. A restricted SSH command streams the bounded artifact to the assigned Mac. The key cannot open a shell, forward ports, select arbitrary paths, or execute a caller-supplied command.
  5. The Mac verifies every received byte, converts audio to 16 kHz mono PCM with macOS afconvert, and runs mlx-whisper 0.4.3 with the immutable whisper-large-v3-turbo model revision a4aaeec0636e6fef84abdcbe3544cb2bf7e9f6fb. It returns time-coded JSON plus model/version, duration, language, and hashes. It receives no portal or storage credentials.
  6. The VM validates size, schema, ordering, timestamps, and source/result hashes before committing segments.
  7. Search indexes are updated only after validation. A failed or expired lease is retried with bounded backoff and may be reassigned safely.

The initial preference is local transcription because the M4 Pro hosts are idle overnight and local processing avoids per-minute inference cost. Azure speech or another approved service remains a measured fallback, not an automatic dependency. A representative ten-service benchmark must compare word accuracy, timestamp accuracy, processing time, failure rate, and cost before that choice is finalized.

Implemented data model

The additive schema should separate jobs from searchable text:

  • service_transcript_jobs: media/service link, source-manifest identity and hash, state, attempts, lease owner/expiry, model and worker version, language, timing, cost, and bounded error code;
  • service_transcripts: immutable machine-transcript version, full duration, model metadata, quality summary, and review state;
  • service_transcript_segments: start/end milliseconds, text, confidence when supported, sequence number, and a full-text index;
  • service_transcript_corrections: original segment reference, reviewed text, editor, timestamp, and reason without overwriting machine output.
  • service_transcript_job_sources: ordered source parts, path, size, modified time, SHA-256, and part number for the complete service input.
  • service_transcript_access_log: staff search/view/correction events; search terms are represented only by SHA-256 and length, never raw query text.
  • service_archive_markers: reviewed service boundary, immutable segment anchor, multipart playback coordinates, creator, and auditable retirement.

One unique key deduplicates a source-manifest SHA-256 plus model/workflow version. Lease updates must be conditional on the current owner and unexpired lease.

Scheduling and resource limits

  • Discover only folders whose ingest metadata and source size have remained stable for a configured window.
  • Run overnight by default, with a small maximum number of jobs per night.
  • Allow one transcription process per Mac initially.
  • Bound input duration, bytes, output segments, transcript bytes, and execution time.
  • Pause automatically when a Mac is on battery, low on disk, thermally constrained, or has an interactive production application running.
  • Record wall-clock processing duration and real-time factor for every run.

The systemd installer defines a daily 2:30 a.m. Eastern timer with up to twenty minutes of jitter. Every scheduled run first checks the restricted worker pool, then starts two queue consumers. Database row leases prevent the consumers from taking the same job; each prefers a different Mac and retains failover. Installing the unit does not enable it. After the initial ten-service cohort passed, the development worker flag and timer were enabled on August 10. Production was then enabled after an additive schema check, both-Mac health preflight, and a controlled deduplicated run.

Both timers then completed their first unattended runs on August 11 inside the configured jitter window. Each validated both Macs, started two consumers, recorded the exhausted backlog as a healthy empty queue, and exited with status zero. Because both environments share the same deduplicated queue, the development timer was disabled after acceptance. The production timer is the only nightly scheduler and still starts one consumer per Mac. The development workflow monitor sets WORKFLOW_ALERT_TRANSCRIPTS_ENABLED=0; the production monitor alone owns transcript-job, timer, and Mac-pool alerting.

The first parallel cohort exposed queue-index contention when each consumer ran the expired-lease recovery range update inside its claim transaction. The second consumer safely saw no claimable row; it did not duplicate a job. Expired leases are now recovered once before consumers start, while each claim transaction is limited to selecting and updating its single leased row.

Installed execution boundary

Both worship@10.200.21.35 and worship@10.200.5.5 have the same user-scoped worker, Python environment, pinned model snapshot, and private cache. A dedicated key at /etc/riveroaks-portal/service-transcript-worker is readable only by the portal worker account. Its matching Mac authorization uses OpenSSH restrict plus a forced command. Supported commands are exactly health, receive <uuid>, run <uuid>, and cleanup <uuid>; an arbitrary-command test was rejected on August 10.

The orchestrator pins both Mac ED25519 host keys in /etc/riveroaks-portal/service-transcript-known-hosts. Uploads use a framed manifest and raw audio stream rather than scp or a remote shell. The Mac accepts only sequential part-NN.m4a names, bounded sizes, known model IDs, valid UUIDs, and matching SHA-256 values. Results remain untrusted until the VM validates them under the still-active database lease.

Verified historical runs

The initial ten-service development cohort and a seven-service production extension completed on August 10, 2026. No transcript wording is reproduced in this runbook.

Service Execution host Audio Runtime RTF Segments Words Cost
August 9, 2026 10.200.21.35 2:40:34 4:32 0.028233 2,083 19,150 $0
August 2, 2026 10.200.21.35 2:35:13 3:59 0.025716 1,884 14,489 $0
July 26, 2026 10.200.5.5 2:35:02 4:59 0.032230 2,030 17,753 $0
July 19, 2026 10.200.21.35 2:39:09 4:49 0.030296 1,870 20,450 $0
July 12, 2026 10.200.5.5 3:01:47 5:00 0.027481 2,095 19,360 $0
July 5, 2026 10.200.21.35 2:38:32 4:19 0.027187 2,001 17,732 $0
June 28, 2026 10.200.5.5 2:44:00 4:12 0.025571 2,266 19,729 $0
June 21, 2026 10.200.21.35 2:39:34 3:52 0.024189 1,747 18,130 $0
May 24, 2026 10.200.5.5 2:38:45 4:24 0.027769 2,048 17,703 $0
March 29, 2026 10.200.21.35 2:35:52 4:25 0.028387 2,421 19,075 $0
March 22, 2026 10.200.5.5 2:40:23 4:20 0.026995 2,110 17,834 $0
March 15, 2026 10.200.5.5 2:32:36 4:43 0.030952 1,981 15,985 $0
February 22, 2026 10.200.21.35 2:34:05 3:46 0.024427 1,995 15,222 $0
February 15, 2026 10.200.5.5 2:39:10 4:14 0.026545 2,228 18,296 $0
February 8, 2026 10.200.21.35 2:37:20 4:03 0.025749 2,222 18,192 $0
January 25, 2026 10.200.5.5 2:46:54 4:24 0.026364 2,462 20,036 $0
January 18, 2026 10.200.21.35 2:45:06 4:49 0.029171 2,249 17,095 $0

For all seventeen available service transcripts, the job and transcript segment/word counts matched; timestamp order had zero overlaps and zero non-positive ranges; source and result identities passed validation; the lease owner, token, and expiry were cleared; and both Macs had zero retained UUID job directories after commit. Every job completed on its first attempt, average runtime was 4:24, measured throughput was about 31–41 times real time, and total inference cost was $0. Full-text search produced indexed matches across all seventeen services. These are operational integrity results, not a word-accuracy assessment; staff should still perform ordinary listening spot checks while using the production cohort.

Deterministic transcript-quality advisory

The portal performs a bounded advisory check after it validates and normalizes the worker result. This check is ordinary code, not AI inference. It never rewrites, removes, or corrects transcript wording and it does not turn an otherwise valid transcript job into an operational failure.

For English transcripts, detector version transcript-quality-1 records:

  • a run of at least four characters from a writing system that is unexpected in an English result;
  • at least eight consecutive copies of the same normalized word;
  • a single normalized token containing at least eighty characters; and
  • as secondary context, the count and first fifty sequence numbers whose model confidence is below 0.30.

The first three patterns are deliberately narrow signs of likely model artifacts. Low confidence is context only because music, audience response, silence boundaries, and difficult PA audio can legitimately produce it. The stored quality_json contains only a schema/detector version, finding codes, counts, bounded segment sequence numbers, and confidence summary. It does not duplicate transcript wording.

When a strong pattern is present, the transcript viewer shows one concise review banner and links to the affected timestamps. Flagged wording remains visible and immutable; an authorized administrator may add a reviewed correction through the existing append-only workflow. Transcripts without strong patterns do not receive a banner. Low-confidence counts remain under the collapsed processing details.

Future jobs receive this metadata during their normal atomic commit. Existing rows can be populated without retranscribing or changing text by running the installed worker once with --backfill-quality; the update is conditional on quality_json IS NULL and logs only transcript ID, quality status, and anomaly count.

The ten-service QA cohort includes bounded opening, middle, closing, proper-name, Scripture-reference, silence-boundary, and difficult-audio samples. Independent short-clip inference confirmed that gross artifacts are repeatable enough to justify the three deterministic checks, but it is not a human listening substitute. An authorized reviewer must still audition the opening, middle, and closing queue against source audio before the listening accuracy task can be closed. Temporary review clips must be deleted after that check; Git and Notion retain only aggregate findings and timestamp references.

Production rollout

Production rollout completed on August 10, 2026.

  • Portal release MR !436 and production-root hotfix MR !437 merged to main; pipelines 7619 and 7622 passed and deployed.
  • The additive schema and services_transcripts.manage permission are present.
  • Both restricted Macs passed the production preflight.
  • The database reports eighteen completed jobs, seventeen available transcripts, one superseded transcript, and zero failed, retrying, or stale-leased jobs.
  • A controlled run started two independent consumers, claimed no duplicate work, and exited successfully.
  • The production timer is enabled for 2:30 a.m. Eastern with up to twenty minutes of jitter.
  • Its first unattended run fired at 2:49 a.m. on August 11, passed both Mac preflights, recorded an empty eligible queue as healthy, and exited zero.
  • The development acceptance run also passed, after which its redundant timer was disabled. Production is the sole nightly queue owner.
  • The Services route remains authentication-gated; search is available to staff and corrections require the dedicated administrator permission.

Privacy, search, and retention

Service audio and transcripts may include prayer requests, names, or other sensitive speech. Production remains staff-only. Transcript text must never be sent to a general-purpose AI service merely to improve search.

Current enforced controls

  • Viewing and searching require an authenticated portal account with staff.view. The portal records transcript views and searches; search terms are represented only by SHA-256 and character count in the audit table.
  • Append-only corrections and service markers require services_transcripts.manage, which is assigned to portal administrators.
  • There is no bulk transcript export endpoint. A staff viewer can read and copy text already visible to them in the browser.
  • The source selector accepts only the deterministic Sunday-service MPR PA path. It does not select isolated microphones, confidence-room feeds, talkback, or rehearsal folders.
  • The execution Macs receive one bounded source bundle and no storage or database credentials. Normal completion and failure paths remove the UUID-scoped bundle immediately.
  • Worker health and receive operations remove abandoned UUID job and staging directories after 48 hours, only while the transcription lock is idle. Model files and unrelated paths are outside that cleanup boundary.
  • Temporary QA clips are deleted when the review session ends and must never be committed to Git or attached to a ticket or Notion task.

The following is the least-privilege recommendation. It is not an approved retention schedule until the River Oaks owner records approval.

Data Recommended retention Notes
Published Sunday-service transcript and word timestamps While the corresponding service recording remains in the staff archive Removing the recording should trigger transcript review or removal; a transcript must not become an orphaned searchable record.
Superseded machine-transcript versions 90 days Retain briefly for rollback, then delete unless a legal hold or active correction review applies.
Reviewed corrections Life of the transcript Corrections remain append-only. Removing a transcript removes its correction text as part of the same request.
Job metadata and source hashes 1 year after terminal completion Preserve processing accountability without retaining another audio copy.
Transcript access audit 1 year Audit rows contain user/action metadata and a search hash, never raw query text.
Worker journal entries 30 days, subject to the host-wide bounded journal policy Logs must contain identifiers, timings, counts, and bounded error codes only.
Failed or interrupted Mac audio bundles Immediate cleanup; 48-hour hard maximum The hard maximum covers VM, network, or process crashes.
Temporary listening/QA clips End of the review session; 24-hour hard maximum Store only aggregate findings and timestamp references afterward.
Portal Playback/MPR PA*.m4a Existing service-archive policy Transcription reuses this derivative and does not create a second archive copy.

Recommended scope is public Goshen Sunday services only. Espanol, special events, rehearsals, and private services should remain excluded until an owner explicitly approves their audience, source channel, and retention.

Recommended audience is staff-only viewing/searching, with correction and marker administration remaining portal-admin-only. Transcript text should not appear in pastor confirmation email, public search, an external AI prompt, or an unauthenticated link. The unresolved access decision is whether staff.view is sufficiently narrow or whether a new services_transcripts.view permission should be granted only to production staff, pastors, and portal administrators.

Correction, takedown, and privacy handling

  1. A wording correction adds a reviewed row; it never overwrites the immutable machine result. The reviewer records a concise reason.
  2. A privacy or takedown request is routed to the portal owner and technology owner. Do not paste the transcript passage into a general ticket; identify the service, timestamp, requester, and requested scope.
  3. An urgent approved takedown must first remove the transcript from search and viewing. Permanent removal then deletes the transcript, ordered segments, and correction text together. It does not silently delete the source service recording, which follows its own archive decision.
  4. Access-audit rows may retain the action record with a null transcript reference after permanent removal. Job metadata may retain hashes and bounded operational facts, but not transcript wording.
  5. A legal or pastoral-care hold overrides routine deletion. The owner records the hold, scope, approver, and release date outside transcript text.

There is not yet an approved takedown command or scheduled database-retention job. Until the policy is approved and those operations are implemented, do not perform ad-hoc transcript deletion in SQL.

Verification and rollout

  1. Completed: move the client to the Proxmox-host NFS boundary and verify the portal VM automount and client-side read-only behavior. Still required: narrow the host's broad parent export after inventorying current clients.
  2. Inventory at least ten recent services and record every candidate channel.
  3. Implement source selection in dry-run mode and review all ambiguous cases.
  4. Benchmark the pinned local model on both Macs; retain hashes and aggregate accuracy metrics, not private transcript excerpts in tickets.
  5. The additive schema, lease/commit validation, receiver installation, pinned host keys, Mac health checks, and arbitrary-shell denial test are complete. Seventeen real historical services now pass across both Macs.
  6. Completed: enable staff-only transcript search and time-offset links for the reviewed backfill cohort.
  7. Completed: backfill seventeen representative services and run the aggregate integrity checks. Continue ordinary listening spot checks in production.
  8. Completed: enable the production worker/search after schema, Mac-health, authorization, deduplication, and timer checks. Complete the tracked access and retention policy as governance hardening.

Secrets and operations

Private SSH keys, optional cloud credentials, and model-provider keys belong in protected runtime environment files or GitLab CI variables. Documentation may name variables and describe their scope, but must never contain their values. The Macs receive a dedicated public key and a forced command only. General interactive SSH credentials are not part of the production workflow.