Tracksolid deployment with timescale & grafana with backup
8867be9d3d
Audit fixes across the ingestion stack: Observability - Move log_ingestion out of batch loops in poll_alarms and poll_parking (was emitting N cumulative log rows per run instead of one). - Add missing log_ingestion + t0 to poll_trips. - Count inserted via cur.rowcount instead of naive +=1 so ON CONFLICT DO NOTHING no longer inflates the metric. Resilience - SAVEPOINT-per-item added to poll_alarms, poll_live_positions, poll_trips, poll_parking so one bad row no longer aborts the batch (webhook handlers already had this; pollers were inconsistent). Performance - /pushgps and poll_track_list now use psycopg2.extras.execute_values with ON CONFLICT DO NOTHING — 10-50x write throughput on larger batches. - sync_devices and sync_driver_audit fetch jimi.track.device.detail concurrently via ThreadPoolExecutor(max_workers=8), cutting the daily registry sync from ~24s to ~3s for an 80-device fleet. - poll_track_list split into two phases: parallel API fetch (4 workers, no DB connection held) then one batched write. Previously the DB connection was held across every per-IMEI HTTP call, risking pool starvation. Security - _validate_token uses hmac.compare_digest for constant-time token comparison (closes timing side-channel). - _parse_data_list caps incoming items at WEBHOOK_MAX_ITEMS (default 5000) so a pathological push cannot blow memory. Tests - Fix test_null_alarm_type_skipped: its INSERT-count assertion was catching the ingestion_log insert written by log_ingestion. Filter that out so the test checks only data-table inserts. - Full suite: 66 passed. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|---|---|---|
| .forgejo/workflows | ||
| administration | ||
| db_audit | ||
| docs | ||
| grafana | ||
| n8n-workflows | ||
| tests | ||
| .env | ||
| .gitignore | ||
| .python-version | ||
| 01_BusinessAnalytics.md | ||
| 02_tracksolid_docker_commands.md | ||
| 02_tracksolid_full_schema_rev.sql | ||
| 03_webhook_schema_migration.sql | ||
| 04_bug_fix_migration.sql | ||
| 05_enhancement_migration.sql | ||
| 260410_baseline_report.md | ||
| 260412_baseline_report.md | ||
| CLAUDE.md | ||
| docker-compose.yaml | ||
| Dockerfile | ||
| grafanaDeployment.md | ||
| grafanaOperationalManual.md | ||
| ingest_events_rev.py | ||
| ingest_movement_rev.py | ||
| OPERATIONS_MANUAL.md | ||
| pyproject.toml | ||
| README.md | ||
| run_migrations.py | ||
| run_migrations.sh | ||
| sync_driver_audit.py | ||
| tracksolid_DB_manual.md | ||
| tracksolidApiDocumentation.md | ||
| ts_shared_rev.py | ||
| webhook_receiver_rev.py | ||