david kiania 4631cc6382 feat: fleettickets — INC/CRQ ticket ingestion, geocoding + read-schema

Standalone module extracted from the tracksolid repo (was migrations 21-23 +
tools/import_tickets.py). Owns the `tickets` schema in the shared tracksolid_db.

- migrations/01_tickets_schema.sql: consolidated final-state schema (tickets.inc/
  crq raw-jsonb-first, geo_clusters + geo_locations gazetteers, geom trigger,
  reporting.fn_tickets_for_map)
- import_tickets.py: rustfs bucket ingest + cluster/location geocoding
  (LocationIQ/OpenCage, viewbox-bounded + cluster-distance guard)
- run_migrations.py, shared.py (self-contained), pyproject, .env.example, README

The DB stays in tracksolid_db; dashboard_api keeps serving /webhook/tickets; the
Tickets map stays a FleetOps tab.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-06-11 20:13:50 +03:00

3.2 KiB

Raw Blame History

fleettickets

Field-ops INC / CRQ ticket ingestion, geocoding, and read-schema that powers the Tickets map in FleetOps. Extracted from the tracksolid repo into its own module (it previously lived there as migrations 21–23 + tools/import_tickets.py).

INC — incident / customer-fault tickets
CRQ — new-installation requests

What this owns

Piece	What
`migrations/01_tickets_schema.sql`	The `tickets` schema: `tickets.inc` / `tickets.crq` (raw-jsonb-first), `tickets.geo_clusters` + `tickets.geo_locations` gazetteers, geom-resolution trigger, and `reporting.fn_tickets_for_map` (the GeoJSON read function)
`import_tickets.py`	Pulls ticket snapshots from the rustfs `tickets` bucket and upserts them; geocodes clusters + INC locations
`run_migrations.py`	Applies `migrations/*.sql` in order (ledger: `tickets.schema_migrations`)
`shared.py`	Minimal DB/logging helpers (self-contained — no tracksolid dependency)

What this does NOT own (stays where it is)

The DB — the tickets schema lives in the shared tracksolid_db.
The read-API — dashboard_api (in the tracksolid stack) serves GET /webhook/tickets, which calls reporting.fn_tickets_for_map (defined here).
The frontend — the Tickets map is a tab in the FleetOps SPA (fleetops repo).

Data model (raw-first)

Each row is just ticket_id + raw (the full source record as jsonb) + a derived geom / geo_source. Everything reads from raw, so a change to the source schema needs no migration. geom is resolved: feed coords (raw lat/lng) → location (geocoded location_name) → cluster centroid → none.

Source coordinates are empty in the feed, so geocoding is required:

--geocode-clusters — one coordinate per cluster (coarse fallback).
--geocode-locations — precise per-location for actionable INC tickets: strips the network codes from location_name (e.g. NW_, ADR_MNT_, FDT<n>, SDUS), geocodes the real place via a keyed provider (LocationIQ / OpenCage), and **rejects any result

25 km from the cluster centroid** (wrong-city guard). Results cache in tickets.geo_locations.

Setup

uv sync
cp .env.example .env        # fill in DATABASE_URL, RUSTFS_*, GEOCODER_*
python run_migrations.py    # apply the schema (idempotent)

Run

# ingest the latest snapshots from the bucket
python import_tickets.py --from-bucket --apply

# geocode (needs GEOCODER_API_KEY)
python import_tickets.py --geocode-clusters  --apply   # coarse, once
python import_tickets.py --geocode-locations --apply   # precise, actionable INC

# from local files instead of the bucket
python import_tickets.py --inc-json inc.json --crq-json crq.json --apply

Dry-run is the default (omit --apply). import_tickets.py --from-bucket shells out to the aws CLI using the RUSTFS_* env (no boto3 dependency).

Notes

The changes/ subdirectory in the bucket holds full timestamped snapshots (not deltas) — ingest latest.json only; don't process changes/.
The curated/geocoded coordinates are written verified = false — review tickets.geo_clusters / tickets.geo_locations and flip verified once checked.

3.2 KiB Raw Blame History Unescape Escape