Live-DB reconciliation before seeding CRQ revealed two divergences:
- tickets.crq did NOT exist: 01_tickets_schema.sql was applied 2026-06-15 from a
version predating its crq section, so the IF-NOT-EXISTS ledger guard has blocked
it ever since (fn_tickets_for_map + resolve_ticket_geoms already reference crq, so
they errored if called — masked because the live INC view uses fn_inc_dashboard).
- The live ledger carries un-versioned 13_inc_search_fn.sql / 14_inc_filter_options.sql
(applied 2026-06-19, absent from this repo).
So 13_crq_columns.sql (ALTER-only, number 13) is replaced by 15_crq_table.sql, which
CREATEs tickets.crq self-containedly (table + geom trigger + raw/typed indexes) and
adds the typed STORED generated columns. Deterministic + idempotent on both the live DB
(crq missing) and a fresh DB (crq minimal from 01). Numbered 15 to sit after the live
ledger's max. Docs/CLI references updated 13->15.
Applied + seeded on the live DB out-of-band (running container, INC image untouched):
39,240 crq rows, 99.99% geocoded (cluster + shared location cache), watermark current,
crq now renders on fn_tickets_for_map.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Split the INC-only loader into a dataset-agnostic engine (pipeline.py, renamed
from import_tickets.py) parameterized by a Dataset config, with thin per-type
entrypoints inc/import_inc.py and crq/import_crq.py. CRQ shares INC's identical
32-column source schema and CDC change stream, so the engine is fully shared.
- pipeline.py: Dataset config (name/table/prefixes/key_regex/post_apply); INC
keeps the capture_history post-apply hook, CRQ has none yet. geocode_locations
now unions tickets.crq (geocoding is cross-dataset: one gazetteer/budget).
- crq/import_crq.py: drains automations/crq/changes/ from isptickets into
tickets.crq (data layer + map; SLA/dashboard/history deferred).
- migrations/13_crq_columns.sql: CRQ mirror of 03 — typed STORED generated
columns + indexes on tickets.crq (reuses tickets.eat_ts()).
- Deployment: Dockerfile/run_ingest.sh run both via `python -m`; pyproject
packages inc/crq. Docs (README, implementation, deployment-and-operations,
n8n export ref, phase-1) updated for the split + the one-time CRQ seed runbook.
tickets.crq already exists (mig 01, LIKE tickets.inc) and is unioned into
reporting.fn_tickets_for_map + resolve_ticket_geoms, so CRQ appears on the
existing Tickets map once seeded. Verified locally: ruff-clean new files, engine
lists/parses both streams against live S3 (crq=52 files, inc unaffected).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Capture the operational knowledge from the isptickets cutover: Coolify app/container,
env management (encrypted — UI or artisan tinker), cron, the Forgejo->Coolify auto-deploy
webhook (config + recreate/verify; it was missing), manual deploy trigger, the
source-bucket cutover procedure, and verification queries. Link it from README; refresh
stale tickets-bucket/ETag references in implementation.md to the isptickets CDC model.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Was hourly at :15 (15 7-19 * * *); now */20 6-20 * * * for fresher ticket
data through the working day. Updates the documented schedule in the Coolify
Scheduled Task command, run_ingest.sh, Dockerfile, README, and implementation
notes (the live schedule is set in the Coolify UI).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The S3 source switched from full hourly snapshots at
automations/inc/<ts>.csv to an incremental CDC stream at
automations/inc/changes/<ts>.csv (first file = full baseline, each later
file = only the rows that changed, keyed by ticket_id; no deletions).
The loader still pointed at the old root path and only ingested the single
newest file, so after the switch it found nothing (no new tickets ingested)
and, even with the path fixed, would silently drop intermediate deltas.
Changes:
- point ingestion at automations/inc/changes/ (_CHANGE_KEY_RE)
- ingest EVERY not-yet-processed file in ascending timestamp order
(baseline first, then each delta), upserting each
- replace the single-ETag skip with a per-file timestamp watermark
(import_meta.metadata->>'source_max_key'); rows + watermark commit in one
txn per file, then archive to processed/ — so a mid-run failure leaves a
consistent, resumable state
- docs: rename n8n-hourly-s3-full-data-exports.md -> n8n-s3-ticket-exports.md
and rewrite it for the incremental stream; fix the reference in
docs/phase-1-ingestion.md
Verified live against prod: re-seeded baseline + 5 deltas (26,529 rows),
files archived to processed/, watermark advanced, re-run is a no-op.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- tickets.closure_events: append-only observed closures (PK ticket_id, closed_at;
observed_at = first sighting; survives row churn).
- tickets.inc_daily_snapshot: one row per EAT day — open backlog (+ SLA split, by
cluster/status) and created/closed flow; upserted each run.
- tickets.capture_history(): appends new closures + upserts today's snapshot.
- import_tickets calls it after each --apply run (ingest or skip); add
--capture-history CLI flag for standalone runs.
Verified: backfilled 21,282 closures; today's snapshot recorded (open_total 30).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GET /webhook/inc-dashboard wrapper spec: query params (cluster/status/window/from/to)
-> SQL passthrough, full response schema, field semantics (open=live vs closed=window,
mttr minutes, derived vs source SLA, map/metrics geocoding gap), examples, caching/auth.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One parameterized function returns {window, open GeoJSON, closed GeoJSON, metrics,
freshness} for the FleetOps live INC map:
- open = all is_actionable tickets (live), filtered by cluster/status, with
sla_state/hours_open (from tickets.inc_open_sla)
- closed= closed_at within the selected window (EAT calendar today/week/month or
custom [from,to)), filtered by cluster/status
- metrics= open/closed counts, SLA split (open derived, closed source), by status/
cluster, closure rate + daily series, avg mttr (minutes)
Filters combine with AND; grants to dashboard_ro/grafana_ro. Verified live
(today/month/cluster/status/custom; last-7d closed=913 matches raw).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
