The S3 source switched from full hourly snapshots at
automations/inc/<ts>.csv to an incremental CDC stream at
automations/inc/changes/<ts>.csv (first file = full baseline, each later
file = only the rows that changed, keyed by ticket_id; no deletions).
The loader still pointed at the old root path and only ingested the single
newest file, so after the switch it found nothing (no new tickets ingested)
and, even with the path fixed, would silently drop intermediate deltas.
Changes:
- point ingestion at automations/inc/changes/ (_CHANGE_KEY_RE)
- ingest EVERY not-yet-processed file in ascending timestamp order
(baseline first, then each delta), upserting each
- replace the single-ETag skip with a per-file timestamp watermark
(import_meta.metadata->>'source_max_key'); rows + watermark commit in one
txn per file, then archive to processed/ — so a mid-run failure leaves a
consistent, resumable state
- docs: rename n8n-hourly-s3-full-data-exports.md -> n8n-s3-ticket-exports.md
and rewrite it for the incremental stream; fix the reference in
docs/phase-1-ingestion.md
Verified live against prod: re-seeded baseline + 5 deltas (26,529 rows),
files archived to processed/, watermark advanced, re-run is a no-op.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- tickets.closure_events: append-only observed closures (PK ticket_id, closed_at;
observed_at = first sighting; survives row churn).
- tickets.inc_daily_snapshot: one row per EAT day — open backlog (+ SLA split, by
cluster/status) and created/closed flow; upserted each run.
- tickets.capture_history(): appends new closures + upserts today's snapshot.
- import_tickets calls it after each --apply run (ingest or skip); add
--capture-history CLI flag for standalone runs.
Verified: backfilled 21,282 closures; today's snapshot recorded (open_total 30).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GET /webhook/inc-dashboard wrapper spec: query params (cluster/status/window/from/to)
-> SQL passthrough, full response schema, field semantics (open=live vs closed=window,
mttr minutes, derived vs source SLA, map/metrics geocoding gap), examples, caching/auth.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
One parameterized function returns {window, open GeoJSON, closed GeoJSON, metrics,
freshness} for the FleetOps live INC map:
- open = all is_actionable tickets (live), filtered by cluster/status, with
sla_state/hours_open (from tickets.inc_open_sla)
- closed= closed_at within the selected window (EAT calendar today/week/month or
custom [from,to)), filtered by cluster/status
- metrics= open/closed counts, SLA split (open derived, closed source), by status/
cluster, closure rate + daily series, avg mttr (minutes)
Filters combine with AND; grants to dashboard_ro/grafana_ro. Verified live
(today/month/cluster/status/custom; last-7d closed=913 matches raw).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
