feat: business analytics expansion, driver CSV import, live DB state docs #1

Merged

kianiadee merged 11 commits from quality-program-2026-04-12 into main 2026-04-18 06:04:11 +00:00

Conversation 0 Commits 11 Files changed 37 +3890 -301

kianiadee commented 2026-04-18 06:03:58 +00:00

Owner

Copy link

Summary

• 01_BusinessAnalytics.md: §0 usage tags, §2.4 cost-per-ticket, §3.6–3.8 alarm/drift/odometer, §4.4–4.5 dispatch log + SLA metrics, §9 fleet readiness scorecard, §10 service-interval forecaster, Appendix B threshold calibration (773 → 1500+ lines)
• 06_business_analytics_migration.sql: assigned_city, dispatch_log, ops schema, service_log, odometer_readings, tickets skeleton, vw_service_forecast view
• import_drivers_csv.py: one-shot script to populate 144 devices from CSV (dry-run by default, --apply to commit)
• 20260414_FS__Logistics - final_fixed.csv: source data committed for container exec workflow
• Analytics report updated with live DB state: 63 devices, pipeline stopped 6 Apr (401 token expiry), Step 0 deployment sequence added

🤖 Generated with Claude Code

## Summary - `01_BusinessAnalytics.md`: §0 usage tags, §2.4 cost-per-ticket, §3.6–3.8 alarm/drift/odometer, §4.4–4.5 dispatch log + SLA metrics, §9 fleet readiness scorecard, §10 service-interval forecaster, Appendix B threshold calibration (773 → 1500+ lines) - `06_business_analytics_migration.sql`: `assigned_city`, `dispatch_log`, `ops` schema, `service_log`, `odometer_readings`, `tickets` skeleton, `vw_service_forecast` view - `import_drivers_csv.py`: one-shot script to populate 144 devices from CSV (dry-run by default, `--apply` to commit) - `20260414_FS__Logistics - final_fixed.csv`: source data committed for container exec workflow - Analytics report updated with live DB state: 63 devices, pipeline stopped 6 Apr (401 token expiry), Step 0 deployment sequence added 🤖 Generated with [Claude Code](https://claude.com/claude-code)

kianiadee added 11 commits 2026-04-18 06:03:59 +00:00

Fix trip distance unit (metres→km) and full device sync on upsert ... 6a0ceb78dd

[FIX-M16] jimi.device.track.mileage returns distance in metres despite
docs claiming km. Confirmed: avgSpeed × runTimeSecond / 3600 = distance/1000.
poll_trips() now divides raw value by 1000 before storing as distance_km.
3 existing bad rows corrected in prod DB (distance_km / 1000).

[FIX-M17] sync_devices() ON CONFLICT clause was only updating 5 of 26
fields, silently dropping driver_phone, sim, iccid, vehicle_name, status
etc. on subsequent syncs. Expanded to update all device fields so driver
assignments made in Tracksolid Pro UI propagate to DB on next daily sync.

Add sync_driver_audit.py: one-shot script to compare API vs DB device
registry, report driver/IMEI gaps, and force a full field upsert.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add 260412 baseline report — first trip data, FIX-M16 confirmed ... 2131faf8c6

Post-deployment snapshot at ~00:15 EAT 2026-04-12. Key changes vs 260410:
- 3 trips recorded (FRED KMGW 538W HULETI, 6.94 km total) — pipeline validated
- FIX-M16 distance unit fix confirmed: implied speed matches API avgSpeed exactly
- 70 track_list fixes in 24h (was 13) — dense trail from active driving
- KDK 829A GP returned to primary depot from secondary Nairobi East cluster
- Uganda anomaly (X3-63282) persists — flagged for management
- Driver name root cause confirmed: not assigned in Tracksolid Pro UI

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Add CLAUDE.md and project docs for structured Claude project ... f9834564ab

CLAUDE.md: cached context file covering project identity, tech stack,
codebase map, schema quick-ref, API gotchas, fix history, working rules,
fleet state, and open items. Structured for maximum cache efficiency —
stable content first, dynamic state at the end.

docs/CONNECTIONS.md: connection parameter shapes (no secrets) for SSH,
DB, API, container resolution, Forgejo, Grafana, n8n.

docs/PROJECT_CONTEXT.md: client business context (telco field service,
3 cities, service types), data quality gaps, KPI framework by domain,
integration roadmap.

docs/KPI_FRAMEWORK.md: living KPI register with status tracking,
thresholds, client feedback log, and review checklist. To be co-developed
with client iteratively.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

docs: add quality program design spec 75d3417a2b

ci: add ruff + mypy static analysis config and Forgejo workflow 2ca3d2f021

test: add test suite - unit tests, webhook endpoint tests, and CI workflow ... 6ed4d3a1e2

57 unit tests covering clean helpers, API signing, and field mapping fixes
(FIX-E06, FIX-M16, BUG-01, BUG-03); integration tests for webhook endpoints
with mocked DB; Forgejo CI workflow with TimescaleDB service container.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat: add db_audit health checks, runner, and scheduled Forgejo workflow ... 20d3ddb841

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

chore: align .python-version to 3.12.0 (matches Docker image and pyproject.toml) f7cc48cc6a

perf+fix: SAVEPOINT-per-item pollers, batched GPS inserts, parallel detail fetch ... 8867be9d3d

Audit fixes across the ingestion stack:

Observability
- Move log_ingestion out of batch loops in poll_alarms and poll_parking
  (was emitting N cumulative log rows per run instead of one).
- Add missing log_ingestion + t0 to poll_trips.
- Count inserted via cur.rowcount instead of naive +=1 so ON CONFLICT
  DO NOTHING no longer inflates the metric.

Resilience
- SAVEPOINT-per-item added to poll_alarms, poll_live_positions,
  poll_trips, poll_parking so one bad row no longer aborts the batch
  (webhook handlers already had this; pollers were inconsistent).

Performance
- /pushgps and poll_track_list now use psycopg2.extras.execute_values
  with ON CONFLICT DO NOTHING — 10-50x write throughput on larger
  batches.
- sync_devices and sync_driver_audit fetch jimi.track.device.detail
  concurrently via ThreadPoolExecutor(max_workers=8), cutting the
  daily registry sync from ~24s to ~3s for an 80-device fleet.
- poll_track_list split into two phases: parallel API fetch (4 workers,
  no DB connection held) then one batched write. Previously the DB
  connection was held across every per-IMEI HTTP call, risking pool
  starvation.

Security
- _validate_token uses hmac.compare_digest for constant-time token
  comparison (closes timing side-channel).
- _parse_data_list caps incoming items at WEBHOOK_MAX_ITEMS (default
  5000) so a pathological push cannot blow memory.

Tests
- Fix test_null_alarm_type_skipped: its INSERT-count assertion was
  catching the ingestion_log insert written by log_ingestion. Filter
  that out so the test checks only data-table inserts.
- Full suite: 66 passed.

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

feat: business analytics expansion + driver CSV import ... cebcf74ba2

- 01_BusinessAnalytics.md: add §0 usage tags, §2.4 cost-per-ticket,
  §3.6–3.8 alarm/drift/odometer, §4.4–4.5 dispatch log + SLA metrics,
  §9 fleet readiness scorecard, §10 service-interval forecaster,
  Appendix B threshold calibration guide (773 → 1437 lines)

- 06_business_analytics_migration.sql: schema support for all new
  analytics sections — assigned_city column, dispatch_log table,
  ops schema, service_log, odometer_readings, tickets skeleton,
  vw_service_forecast view

- import_drivers_csv.py: one-shot script to populate driver_name,
  vehicle_number, vehicle_models, cost_centre, assigned_city, sim,
  iccid, imsi from 20260414_FS__Logistics - final_fixed.csv (144 rows);
  dry-run by default, --apply to commit, --only-null for safe additive mode

- 20260414_FS__Logistics - final_fixed.csv: source data committed for
  reproducibility and container exec workflow

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

docs: update analytics report with live DB state (18 Apr 2026) ... 274473c544

- §1: add current deployment state table — 63 devices, 0 driver names,
  5 trips, pipeline stopped 6 Apr (401 token expiry); note tracksolid_2
  vs tracksolid schema split
- §6: status column per question (Ready/Needs data/Blocked) reflecting
  actual DB state; add cost-per-ticket, city drift, odometer rows
- §8: add Step 0 full deployment sequence (git pull → migrations 01-06
  → container rebuild → sync_driver_audit → import_drivers_csv);
  Step 3 updated to reference import script; Step 5 collapsed to pointer
- Footer: db-state stamp and update date

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

kianiadee merged commit 5fa87a712e into main 2026-04-18 06:04:11 +00:00

kianiadee referenced this pull request from a commit 2026-04-18 06:04:11 +00:00

Merge pull request 'feat: business analytics expansion, driver CSV import, live DB state docs' (#1) from quality-program-2026-04-12 into main

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: kianiadee/tracksolid_timescale_grafana_prod#1

No description provided.