From 244112154ac9d62e4dd7bac5625f0769a9fe8c78 Mon Sep 17 00:00:00 2001 From: David Kiania Date: Sat, 18 Apr 2026 12:26:21 +0300 Subject: [PATCH 1/2] docs: update CLAUDE.md with session learnings (18 Apr 2026) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - §3: note tracksolid_2 as live schema, tracksolid as empty target; add DB direct access tip (31.97.44.246:5888, leading space in .env) - §4: add import_drivers_csv.py and migration 06 to codebase map - §5: document tracksolid_2 live tables with column differences (assigned_team vs cost_centre, city vs assigned_city); add ops.* - §8: add rule 9 (Forgejo API auth via keychain) and rule 10 (always check active schema before querying) - §9: update fleet state — pipeline stopped Apr 6, CSV fleet pending, 0 driver names, 19 stale positions - §10: replace driver-name manual item with deploy + CSV import tasks Co-Authored-By: Claude Opus 4.7 --- CLAUDE.md | 46 +++++++++++++++++++++++++++++++--------------- 1 file changed, 31 insertions(+), 15 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 20ebd66..bae46ed 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,7 +31,9 @@ See `docs/CONNECTIONS.md` for the full shape. Summary: - **SSH:** `ssh -i ~/.ssh/id_ed25519 kianiadee@stage.rahamafresh.com` - **DB name:** `tracksolid_db` · **DB user:** `postgres` (internal) · `tracksolid_owner` (app) · `grafana_ro` (read-only) -- **DB schema:** `tracksolid` (operational) · `infrastructure` · `dwh_gold` (aggregates) +- **DB schema:** `tracksolid_2` (current live data, legacy stack) · `tracksolid` (new stack target, currently empty) · `infrastructure` · `dwh_gold` (aggregates) +- **⚠ Schema split:** new ingestion code targets `tracksolid`; all live rows are in `tracksolid_2` until the new stack is deployed and `sync_driver_audit.py` has run. +- **DB direct access:** `DATABASE_URL` in `.env` points to `31.97.44.246:5888` — queries can be run locally via psql without SSH. Note: value has a leading space, strip it. - **Container naming:** Coolify appends a random suffix. Always resolve with: ```bash docker ps --filter name= --format "{{.Names}}" | head -1 @@ -49,6 +51,7 @@ ingest_movement_rev.py # GPS positions, trips, parking, track-list (high-re ingest_events_rev.py # Alarm events polling (fallback for webhook push) webhook_receiver_rev.py # FastAPI push receiver: /pushobd /pushevent /pushtripreport etc. sync_driver_audit.py # One-shot: API↔DB driver/IMEI gap report + full upsert +import_drivers_csv.py # One-shot: populate 144 X3/JC400P devices from CSV (--apply to commit) run_migrations.py # Applies SQL migrations in order at container startup docker-compose.yaml # Services: timescale_db, ingest_movement, ingest_events, # webhook_receiver, grafana @@ -56,10 +59,11 @@ grafana/ # Grafana provisioning (baked into image) n8n-workflows/ # n8n workflow exports docs/ # Reference docs (connections, API, KPIs, project context) 02_tracksolid_full_schema_rev.sql # Full schema bootstrap -03..05_*.sql # Incremental migrations -01_BusinessAnalytics.md # SQL analytics library (reference before writing queries) +03..06_*.sql # Incremental migrations (06 adds assigned_city, dispatch_log, ops.*) +01_BusinessAnalytics.md # SQL analytics library — read before writing queries +20260414_FS__Logistics - final_fixed.csv # 144-device driver/vehicle source data tracksolidApiDocumentation.md # API endpoint reference -260412_baseline_report.md # Latest fleet state snapshot +260412_baseline_report.md # Fleet state snapshot (Apr 2026) ``` --- @@ -67,8 +71,12 @@ tracksolidApiDocumentation.md # API endpoint reference ## 5. Database Schema — Key Tables ```sql -tracksolid.devices -- Master device registry (63 devices, imei PK) -tracksolid.live_positions -- Current position per device (1 row per IMEI, upserted) +tracksolid_2.devices -- LIVE registry (63 AT4-series devices, 353549* IMEIs, 0 driver names) + -- NB: has `assigned_team` (not cost_centre), `city` (not assigned_city) +tracksolid_2.live_positions -- LIVE positions (19 rows, all stale since 6 Apr 2026) +tracksolid_2.ingestion_log -- LIVE pipeline audit trail +tracksolid.devices -- Target registry (empty — new stack not yet deployed) +tracksolid.live_positions -- Target positions (empty — new stack not yet deployed) tracksolid.position_history -- All GPS fixes (hypertable, partitioned by gps_time) -- source: 'poll' (60s sweep) | 'track_list' (30m high-res) tracksolid.trips -- Trip summaries: distance_km, driving_time_s, avg/max speed @@ -77,10 +85,14 @@ tracksolid.alarms -- Alarm events (alarm_type, alarm_name, alarm_time tracksolid.obd_readings -- OBD diagnostics (push only, awaiting webhook registration) tracksolid.device_events -- Power on/off tamper events tracksolid.ingestion_log -- API call audit trail per endpoint +tracksolid.dispatch_log -- Dispatch decisions for SLA tracking (migration 06) dwh_gold.fact_daily_fleet_metrics -- Nightly ETL aggregates per vehicle per day +ops.service_log -- Workshop service history (migration 06) +ops.odometer_readings -- Physical odometer captures (migration 06) +ops.tickets -- Ticket skeleton for ops integration (migration 06) ``` -Full DDL: `02_tracksolid_full_schema_rev.sql` + migrations `03`–`05`. +Full DDL: `02_tracksolid_full_schema_rev.sql` + migrations `03`–`06`. --- @@ -128,19 +140,22 @@ Full DDL: `02_tracksolid_full_schema_rev.sql` + migrations `03`–`05`. 6. **SSH only when asked.** Default workflow is local code → commit → push. SSH into the instance only when explicitly asked to test or run something live. 7. **Secrets from env only.** Connection strings, API keys, and passwords live in `.env`. Reference variable names from `docs/CONNECTIONS.md`, never values. 8. **Two developers, one incoming.** Write code and docs that a second developer (mixed technical/operations background) can follow without prior context. +9. **Forgejo API auth:** credentials stored in macOS keychain. Retrieve with `git credential fill` (host=repo.rahamafresh.com). Use basic auth against `https://repo.rahamafresh.com/api/v1` directly — no `tea` or `gh` needed. +10. **Check active schema before querying.** Always verify which schema holds live data (`tracksolid` vs `tracksolid_2`) before running or writing queries. Until new stack deploys, live data is in `tracksolid_2`. --- -## 9. Fleet State (as of 2026-04-12 baseline) +## 9. Fleet State (as of 2026-04-18) | Metric | Value | |---|---| -| Total registered devices | 63 (growing to 80) | -| Devices with GPS fix < 2h | 2 | -| Devices never reported | 44 | -| Driver names populated | 0 — must be set in Tracksolid Pro UI first | -| Cities active | Nairobi (primary), Mombasa (deploying), Kampala (1 device confirmed) | -| Uganda anomaly | X3-63282 at 0.196, 32.540 — under investigation | +| Registered devices (live DB) | 63 AT4-series (`353549*` IMEIs) in `tracksolid_2` | +| Devices in CSV (not yet in DB) | 144 X3/JC400P (`865135*`, `862798*` IMEIs) | +| Driver names populated | 0 — run `import_drivers_csv.py --apply` after new stack deployed | +| Live positions | 19 (all stale — last fix 6 Apr 2026) | +| Trips recorded | 5 (12.8 km total, 4–6 Apr 2026 only) | +| Pipeline status | Stopped 6 Apr 2026 — 401 token expiry (fixed in `ts_shared_rev.py`, deploy new stack) | +| Cities active | Nairobi (primary), Mombasa (deploying), Kampala (4 devices in CSV) | | Service flags | KDK 829A GP (239,264 km), Belta KCU-647D (235,000 km) | Latest full snapshot: `260412_baseline_report.md` @@ -151,7 +166,8 @@ Latest full snapshot: `260412_baseline_report.md` | Priority | Item | |---|---| -| HIGH | Assign driver names + vehicle numbers in Tracksolid Pro UI | +| HIGH | Deploy new ingestion stack (see §8 Step 0 in `01_BusinessAnalytics.md` for full sequence) | +| HIGH | Run `import_drivers_csv.py --apply` after stack deployed (144 devices, names + plates ready) | | HIGH | Register webhooks: `/pushobd` `/pushoil` `/pushtem` `/pushlbs` `/pushevent` | | HIGH | Investigate X3-63282 in Kampala — legitimate or unauthorised? | | MEDIUM | Set `fuel_100km` per vehicle type to activate fuel cost calculations | From 160f477318e05a37f67479c4afef039256c5fea8 Mon Sep 17 00:00:00 2001 From: David Kiania Date: Sat, 18 Apr 2026 14:14:32 +0300 Subject: [PATCH 2/2] infra: expose timescale_db port 5888 for direct pgcli access MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Maps host port 5888 → container port 5432 so the DB can be reached directly from the MacBook (requires UFW allow 5888/tcp on the server). Co-Authored-By: Claude Opus 4.7 --- docker-compose.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docker-compose.yaml b/docker-compose.yaml index 8f0992b..3e88708 100644 --- a/docker-compose.yaml +++ b/docker-compose.yaml @@ -6,6 +6,8 @@ services: - POSTGRES_DB=${POSTGRES_DB} - POSTGRES_USER=${POSTGRES_USER} - POSTGRES_PASSWORD=${POSTGRES_PASSWORD} + ports: + - "5888:5432" volumes: - timescale-data:/var/lib/postgresql/data healthcheck: