898fd25a5a
Phase 0 of the three-stakeholder analytics redesign: - 08_analytics_config.sql: ops.cost_rates + ops.kpi_targets with seed fuel rates (KES 195/L NBO+MBA, UGX 5200/L KLA) and 6 seed KPI targets (utilisation_pct, idle_pct global+osp-patrol, fuel_kes_per_100km, mttr_hours, alarms_per_100km). Granted SELECT to grafana_ro. Wired into run_migrations.py MIGRATIONS. - import_drivers_csv.py: full rewrite for the new Mitieng CSV (20260427_FSG_Vehicles_mitieng.csv). Snake_case columns, drops _infer_city() plate-prefix logic in favour of reading assigned_city directly. Adds cost_centre, assigned_route, vehicle_category, vehicle_brand, fuel_100km, depot_address. Treats the literal "NULL" string as missing. Reuses clean(), clean_num(), clean_ts(), get_conn(), get_logger() from ts_shared_rev. Special-cases numeric and timestamptz columns in the UPDATE clause. - audit_device_reconciliation.py: read-only audit comparing the CSV against tracksolid.devices. Reports per-account row counts, IMEIs on one side only, and devices on both sides whose metadata is still NULL. - 260427_device_reconciliation.md + 260427_audit_output.txt: Phase 0.2 reconciliation record. First run: DB has 172 devices, CSV has 162, delta +10 (10 IMEIs in DB-only, mostly fireside-account auto-syncs). Importer run with --only-null --apply filled 154 rows; coverage now assigned_city 152/172, cost_centre 150/172. Applied to stage on 2026-04-27 23:35 UTC. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
91 lines
3.3 KiB
Markdown
91 lines
3.3 KiB
Markdown
# Device Reconciliation — 162 vs 182 (2026-04-27)
|
||
|
||
Phase 0.2 of the Business Analytics redesign. Resolves the gap between
|
||
`20260427_FSG_Vehicles_mitieng.csv` (162 rows) and `tracksolid.devices`
|
||
(~182 rows at last check).
|
||
|
||
## How to populate this report
|
||
|
||
1. Pull this branch onto the Coolify host (or rebuild containers so the
|
||
ingest container has `audit_device_reconciliation.py`).
|
||
|
||
2. Run inside the ingest container so it has `DATABASE_URL`:
|
||
```bash
|
||
ING=$(docker ps --filter name=ingest_movement --format "{{.Names}}" | head -1)
|
||
docker cp 20260427_FSG_Vehicles_mitieng.csv "$ING":/app/
|
||
docker exec "$ING" python audit_device_reconciliation.py \
|
||
--csv 20260427_FSG_Vehicles_mitieng.csv \
|
||
--out /tmp/260427_audit_output.txt
|
||
docker cp "$ING":/tmp/260427_audit_output.txt ./
|
||
```
|
||
|
||
3. Paste the audit output into the **Audit output** section below.
|
||
|
||
4. Mark the chosen disposition for each IMEI in the **Disposition** section.
|
||
|
||
## Audit output
|
||
|
||
First run: 2026-04-27 23:35 UTC against `tracksolid_db` on stage. Full output in
|
||
`260427_audit_output.txt`. Headline numbers:
|
||
|
||
| Metric | Value |
|
||
|---|---|
|
||
| CSV rows | 162 |
|
||
| `tracksolid.devices` rows | 172 |
|
||
| Delta (DB − CSV) | +10 |
|
||
| In CSV but not in DB | 0 |
|
||
| In DB but not in CSV | 10 |
|
||
| Devices both sides, DB metadata still NULL on ≥1 field | 162 (resolved by importer run) |
|
||
|
||
After running `import_drivers_csv.py --only-null --apply` (2026-04-27): 154
|
||
devices updated, 8 already complete, 0 inserted. Coverage now: `assigned_city`
|
||
152/172, `cost_centre` 150/172, `vehicle_brand` 2/172, `fuel_100km` 3/172.
|
||
`assigned_route` / `vehicle_category` / `depot_address` remain 0/172 (CSV
|
||
provided no values for these — Phase 1 follow-up).
|
||
|
||
The 10 in-DB-not-in-CSV IMEIs are listed in `260427_audit_output.txt`. They
|
||
sit in `(blank)` or `fireside` accounts and surface in Grafana as
|
||
`assigned_city = 'unassigned'` thanks to the existing COALESCE in
|
||
`07_analytics_views.sql`.
|
||
|
||
## Disposition
|
||
|
||
For each IMEI in the "in DB but not in CSV" list, choose one and record why:
|
||
|
||
| IMEI | Account | Last seen | Disposition | Notes |
|
||
|---|---|---|---|---|
|
||
| | | | | |
|
||
|
||
**Disposition options:**
|
||
|
||
- **prune** — Delete from `tracksolid.devices`. Use when the unit is a stale
|
||
test/decommissioned device that should never have synced. Capture the SQL
|
||
before running:
|
||
```sql
|
||
DELETE FROM tracksolid.devices WHERE imei = '<imei>';
|
||
```
|
||
*Caveat:* foreign keys from `position_history`, `trips`, `alarms` must be
|
||
considered first — these will block the delete if there's any history.
|
||
Usually safer to leave-as-NULL.
|
||
|
||
- **leave-as-NULL** — Keep the row; metadata fields stay NULL. The device
|
||
was auto-synced from a Tracksolid account that the CSV doesn't cover
|
||
(likely `Fireside@HQ` rows that were left out of this Mitieng export).
|
||
Grafana views already use `COALESCE(d.assigned_city, d.city, 'unassigned')`
|
||
so these surface as "unassigned" but don't break dashboards.
|
||
|
||
- **addendum** — Add to a follow-up CSV and re-run the importer with
|
||
`--apply`. Use when the device is legitimate fleet metadata was just
|
||
missing from the export.
|
||
|
||
## Decision
|
||
|
||
<!-- Fill in once dispositions are chosen. -->
|
||
|
||
- Total devices reviewed: ___
|
||
- Pruned: ___
|
||
- Left-as-NULL: ___
|
||
- Added via addendum: ___
|
||
|
||
After action, re-run `audit_device_reconciliation.py` and confirm the
|
||
delta is what you expect.
|