Verified each finding against the code (+ profiled the 31k-row CSV sample);
implemented only the genuinely valid fixes:
- import_tickets.py: fold _record_meta into the upsert transaction so rows +
snapshot meta commit atomically (BUG 2); guard _ts_from_key against
regex-matching-but-invalid dates so the sort can't crash (BUG 11);
extract_place now splits glued NW prefixes (~1.7k rows, e.g. NWKIAMBU→KIAMBU)
and only drops a trailing '-<seg>' when it's a unit/instruction code, keeping
real-word tails like '-MALL' (BUG 14). Scoped glued-split to NW only —
CO/NE/SE begin real words (COAST/NEW/SEASONS) per the data.
- Dockerfile + pyproject.toml: install from pyproject (single source of truth)
instead of mirroring deps; add build-system + py-modules so `pip install .`
works for the flat-module layout (BUG 9).
- migrations/03_inc_columns.sql: document the eat_ts IMMUTABLE/tzdata footgun
and the manual-recompute path (BUG 6).
- .gitignore: narrow *.json → *.local.json so real fixtures can be versioned;
ignore build/ and *.egg-info/ (BUG 10).
Reclassified/skipped as invalid or by-design: BUG 1, 3, 4, 5, 7, 8, 12, 13.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rework import_tickets.py from the retired JSON `latest.json` model to the new
hourly full-snapshot CSV export. Strictly INC (CRQ out of scope).
- Ingest the newest automations/inc/<EAT-timestamp>.csv; skip-if-unchanged by
comparing S3 ETag to tickets.import_meta.metadata.source_etag.
- Upsert on ticket_id (PK; no dups, never delete -> closure history accrues).
No truncate. On success, move processed files to automations/inc/processed/.
- Clean at ingest: drop is_alarm=true + the "EXPORT STOPPED..." sentinel; drop
week_*, source_s3_*/source_snapshot_id, department/source_type; lowercase
region, uppercase raw_status; keep service_type + bucket.
- Force path-style S3 addressing; --inc-csv for local dev; --from-bucket for cron.
- Add migrations/02 (import_meta + freshness); refresh README/.env.example/docs.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
