tracksolid_timescale_grafana_prod/CLAUDE.md

CLAUDE.md — Fireside Communications · Tracksolid Fleet Intelligence

1. What This Project Is

Fleet telematics ingestion and analytics stack for a telco first-line support client operating in Nairobi, Mombasa, and Kampala. The client dispatches field technicians to install, repair, and maintain home and business broadband, handle LOS signal faults, service migrations, and maintain outside plant infrastructure. The fleet is ~80 vehicles across three cities, all tracked via Tracksolid Pro (Jimi IoT API).

This repository ingests the Tracksolid Pro API into a TimescaleDB/PostGIS database and visualises fleet and operational KPIs in Grafana. The pipeline is deployed on Coolify at stage.rahamafresh.com.

Repository: https://repo.rahamafresh.com/kianiadee/tracksolid_timescale_grafana_prod.git

---

2. Tech Stack

Layer	Technology
Ingestion	Python 3.12 — ingest_movement_rev.py, ingest_events_rev.py, webhook_receiver_rev.py
Shared utils	ts_shared_rev.py — token cache, DB pool, API signing, clean helpers
Database	PostgreSQL 16 + TimescaleDB 2.15 + PostGIS 3 (tracksolid_db)
Orchestration	Docker Compose on Coolify
Visualisation	Grafana (provisioned via custom image)
Workflow automation	n8n
API source	Tracksolid Pro / Jimi IoT Open Platform (eu-open.tracksolidpro.com/route/rest)
Version control	Forgejo at repo.rahamafresh.com

---

3. Instance & Connection Parameters

See docs/CONNECTIONS.md for the full shape. Summary:

• SSH: ssh -i ~/.ssh/id_ed25519 kianiadee@stage.rahamafresh.com
• DB name: tracksolid_db · DB user: postgres (internal) · tracksolid_owner (app) · grafana_ro (read-only)
• DB schema: tracksolid_2 (current live data, legacy stack) · tracksolid (new stack target, currently empty) · infrastructure · dwh_gold (aggregates)
• ⚠ Schema split: new ingestion code targets tracksolid; all live rows are in tracksolid_2 until the new stack is deployed and sync_driver_audit.py has run.
• DB direct access: DATABASE_URL in .env points to 31.97.44.246:5888 — queries can be run locally via psql without SSH. Note: value has a leading space, strip it.
• Container naming: Coolify appends a random suffix. Always resolve with:

  docker ps --filter name=<service_name> --format "{{.Names}}" | head -1
  

  e.g. docker ps --filter name=timescale_db --format "{{.Names}}" | head -1
• Env vars: loaded from .env via env_file in docker-compose.yaml. See docs/CONNECTIONS.md for variable names. Never hardcode secrets.

---

4. Codebase Map

ts_shared_rev.py            # Shared: config, signing, DB pool, token cache, clean helpers
ingest_movement_rev.py      # GPS positions, trips, parking, track-list (high-res trail), device sync
ingest_events_rev.py        # Alarm events polling (fallback for webhook push)
webhook_receiver_rev.py     # FastAPI push receiver: /pushobd /pushevent /pushtripreport etc.
sync_driver_audit.py        # One-shot: API↔DB driver/IMEI gap report + full upsert
import_drivers_csv.py       # One-shot: populate 144 X3/JC400P devices from CSV (--apply to commit)
run_migrations.py           # Applies SQL migrations in order at container startup
docker-compose.yaml         # Services: timescale_db, ingest_movement, ingest_events,
                            #           webhook_receiver, grafana
grafana/                    # Grafana provisioning (baked into image)
n8n-workflows/              # n8n workflow exports
docs/                       # Reference docs (connections, API, KPIs, project context)
02_tracksolid_full_schema_rev.sql   # Full schema bootstrap
03..06_*.sql                # Incremental migrations (06 adds assigned_city, dispatch_log, ops.*)
01_BusinessAnalytics.md     # SQL analytics library — read before writing queries
20260414_FS__Logistics - final_fixed.csv  # 144-device driver/vehicle source data
tracksolidApiDocumentation.md       # API endpoint reference
260412_baseline_report.md   # Fleet state snapshot (Apr 2026)

---

5. Database Schema — Key Tables

tracksolid_2.devices         -- LIVE registry (63 AT4-series devices, 353549* IMEIs, 0 driver names)
                             -- NB: has `assigned_team` (not cost_centre), `city` (not assigned_city)
tracksolid_2.live_positions  -- LIVE positions (19 rows, all stale since 6 Apr 2026)
tracksolid_2.ingestion_log   -- LIVE pipeline audit trail
tracksolid.devices           -- Target registry (empty — new stack not yet deployed)
tracksolid.live_positions    -- Target positions (empty — new stack not yet deployed)
tracksolid.position_history  -- All GPS fixes (hypertable, partitioned by gps_time)
                             -- source: 'poll' (60s sweep) | 'track_list' (30m high-res)
tracksolid.trips             -- Trip summaries: distance_km, driving_time_s, avg/max speed
tracksolid.parking_events    -- Stop events with duration and address
tracksolid.alarms            -- Alarm events (alarm_type, alarm_name, alarm_time)
tracksolid.obd_readings      -- OBD diagnostics (push only, awaiting webhook registration)
tracksolid.device_events     -- Power on/off tamper events
tracksolid.ingestion_log     -- API call audit trail per endpoint
tracksolid.dispatch_log      -- Dispatch decisions for SLA tracking (migration 06)
dwh_gold.fact_daily_fleet_metrics  -- Nightly ETL aggregates per vehicle per day
ops.service_log              -- Workshop service history (migration 06)
ops.odometer_readings        -- Physical odometer captures (migration 06)
ops.tickets                  -- Ticket skeleton for ops integration (migration 06)

Full DDL: 02_tracksolid_full_schema_rev.sql + migrations 03–06.

---

6. API Critical Facts

Always read tracksolidApiDocumentation.md before adding a new endpoint call.

Fact	Detail
Auth	OAuth2 — token cached in tracksolid.api_token_cache, refreshed via jimi.oauth.token.refresh
Signing	MD5: secret + sorted(k+v pairs) + secret — see build_sign() in ts_shared_rev.py
Batch limit	Max 50 IMEIs per call for most endpoints
distance field	Returns METRES, not km despite docs. Always divide by 1000. (FIX-M16)
driverName/driverPhone	From jimi.user.device.list — will be NULL if not set in Tracksolid Pro UI
alarm_type field	API polling returns alertTypeId/alarmTypeName — NOT alarmType/alarmName (FIX-E06)
durSecond	Parking endpoint returns durSecond, not seconds (FIX-M13)
jimi.device.track.mileage	startMileage/endMileage are cumulative odometer in metres
Rate limit	Code 1006 — back off and retry with re-sign (handled in api_post())
OBD data	Push only via /pushobd webhook — no polling endpoint exists

---

7. Fix History (do not regress)

Fix ID	File	What it fixed
FIX-M11	ingest_movement_rev.py	Removed erroneous ×1000 on distance (was storing km as mm)
FIX-M13	ingest_movement_rev.py	Parking: added acc_type=0, account; mapped durSecond
FIX-M14	ingest_movement_rev.py	poll_track_list() — high-res GPS trail every 30m
FIX-M15	ingest_movement_rev.py	get_device_locations() — on-demand precision refresh
FIX-M16	ingest_movement_rev.py	distance from API is metres → divide by 1000 before storing
FIX-M17	ingest_movement_rev.py	sync_devices() ON CONFLICT now updates all 26 fields (was 5)
FIX-E06	ingest_events_rev.py	Alarm field mapping: alertTypeId/alarmTypeName/alertTime
BUG-02	Migration 04	Historical distance_m rows ÷1,000,000 → renamed to distance_km

---

8. Working Rules

1. No prod push without explicit user confirmation. Always state what you are about to push and wait.
2. Never rewrite a migration that is already applied. Check tracksolid.schema_migrations first. Add a new numbered migration file for any schema change.
3. Read before writing. Before suggesting any code change, read the relevant source file. Before writing a query, check 01_BusinessAnalytics.md for an existing pattern.
4. Reuse shared utilities. All DB access via get_conn(), all API calls via api_post(), all cleaning via clean() / clean_num() / clean_int() / clean_ts() in ts_shared_rev.py. Do not reinvent these.
5. Resolve container names dynamically. Never hardcode the Coolify suffix. Use docker ps --filter name=<service>.
6. SSH only when asked. Default workflow is local code → commit → push. SSH into the instance only when explicitly asked to test or run something live.
7. Secrets from env only. Connection strings, API keys, and passwords live in .env. Reference variable names from docs/CONNECTIONS.md, never values.
8. Two developers, one incoming. Write code and docs that a second developer (mixed technical/operations background) can follow without prior context.
9. Forgejo API auth: credentials stored in macOS keychain. Retrieve with git credential fill (host=repo.rahamafresh.com). Use basic auth against https://repo.rahamafresh.com/api/v1 directly — no tea or gh needed.
10. Check active schema before querying. Always verify which schema holds live data (tracksolid vs tracksolid_2) before running or writing queries. Until new stack deploys, live data is in tracksolid_2.

---

9. Fleet State (as of 2026-04-18)

Metric	Value
Registered devices (live DB)	63 AT4-series (353549* IMEIs) in tracksolid_2
Devices in CSV (not yet in DB)	144 X3/JC400P (865135*, 862798* IMEIs)
Driver names populated	0 — run import_drivers_csv.py --apply after new stack deployed
Live positions	19 (all stale — last fix 6 Apr 2026)
Trips recorded	5 (12.8 km total, 4–6 Apr 2026 only)
Pipeline status	Stopped 6 Apr 2026 — 401 token expiry (fixed in ts_shared_rev.py, deploy new stack)
Cities active	Nairobi (primary), Mombasa (deploying), Kampala (4 devices in CSV)
Service flags	KDK 829A GP (239,264 km), Belta KCU-647D (235,000 km)

Latest full snapshot: 260412_baseline_report.md

---

10. Open Items (update as resolved)

Priority	Item
HIGH	Deploy new ingestion stack (see §8 Step 0 in 01_BusinessAnalytics.md for full sequence)
HIGH	Run import_drivers_csv.py --apply after stack deployed (144 devices, names + plates ready)
HIGH	Register webhooks: /pushobd /pushoil /pushtem /pushlbs /pushevent
HIGH	Investigate X3-63282 in Kampala — legitimate or unauthorised?
MEDIUM	Set fuel_100km per vehicle type to activate fuel cost calculations
MEDIUM	Investigate 44 silent devices — SIM installed? Activated?
MEDIUM	Co-develop client KPI framework (see docs/KPI_FRAMEWORK.md)
LOW	Populate geofences — depot boundaries, city zones
LOW	Run nightly ETL: SELECT dwh_gold.refresh_daily_metrics(CURRENT_DATE - 1)