34f5fa1b9c
DWH pipeline (new):
- dwh/261001_dwh_control.sql — watermarks + per-run audit log schema
- dwh/261002_bronze_constraints_audit.sql — ON CONFLICT key assertion
- dwh/261003_dwh_roles.sql — dwh_owner / grafana_ro contract assertion
- dwh/261004_dwh_observability_views.sql — v_table_freshness,
v_recent_failures, v_watermark_lag (readable by grafana_ro)
- docs/DWH_PIPELINE.md — operations runbook (setup, troubleshooting,
manual re-run, back-fill, rotation)
- DWH_Execution_Manual.md — reusable playbook for future data
projects (extract → blob → load pattern, 7 design principles,
snapshot-vs-incremental matrix, verification gates)
- docs/superpowers/{specs,plans}/2026-04-24-n8n-dwh-bronze-pipeline-*
— design spec + 27-task implementation plan
Security:
- dwh/260423_dwh_ddl_v1.sql — redacted plaintext role passwords to
'CHANGE_ME_BEFORE_APPLY' placeholders; added SECURITY header
documenting generation + rotation flow
Docs:
- CLAUDE.md — §3 adds tracksolid_dwh@31.97.44.246:5888 target,
§4 adds dwh/ + docs/DWH_PIPELINE.md to codebase map, §5 adds
bronze + dwh_control schema roll-up, §10 adds deploy task +
password rotation follow-up
Also includes miscellaneous in-progress files accumulated on this
branch (workspace, analytics notes, vehicle CSVs, extract helpers,
renamed markdown archives).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
56 lines
1.6 KiB
Text
56 lines
1.6 KiB
Text
### Pipeline health — last hour (key check)
|
|
|
|
SELECT
|
|
endpoint,
|
|
COUNT(*) AS calls,
|
|
SUM(rows_upserted) AS upserted,
|
|
SUM(rows_inserted) AS inserted,
|
|
ROUND(AVG(duration_ms)::numeric, 0) AS avg_ms,
|
|
COUNT(*) FILTER (WHERE success = false) AS failures,
|
|
MAX(run_at AT TIME ZONE 'Africa/Nairobi') AS last_call_eat
|
|
FROM tracksolid.ingestion_log
|
|
WHERE run_at > now() - interval '1 hour'
|
|
GROUP BY endpoint
|
|
ORDER BY calls DESC;
|
|
|
|
-------- Ingestion Pipeline Health
|
|
|
|
SELECT
|
|
endpoint,
|
|
COUNT(*) AS total_calls,
|
|
SUM(rows_upserted) AS total_upserted,
|
|
SUM(rows_inserted) AS total_inserted,
|
|
ROUND(AVG(duration_ms)::numeric, 0) AS avg_ms,
|
|
COUNT(*) FILTER (WHERE success = false) AS failures,
|
|
MIN(run_at AT TIME ZONE 'Africa/Nairobi') AS first_call,
|
|
MAX(run_at AT TIME ZONE 'Africa/Nairobi') AS last_call
|
|
FROM tracksolid.ingestion_log
|
|
GROUP BY endpoint
|
|
ORDER BY total_calls DESC;
|
|
|
|
### Recent calls — last 20 entries
|
|
|
|
SELECT
|
|
run_at AT TIME ZONE 'Africa/Nairobi' AS run_eat,
|
|
endpoint,
|
|
imei_count,
|
|
rows_upserted,
|
|
rows_inserted,
|
|
duration_ms,
|
|
success,
|
|
error_message
|
|
FROM tracksolid.ingestion_log
|
|
ORDER BY run_at DESC
|
|
LIMIT 20;
|
|
|
|
### Recent calls — FAILED CALLS entries
|
|
|
|
SELECT
|
|
run_at AT TIME ZONE 'Africa/Nairobi' AS run_eat,
|
|
endpoint,
|
|
error_code,
|
|
error_message
|
|
FROM tracksolid.ingestion_log
|
|
WHERE success = false
|
|
ORDER BY run_at DESC;
|
|
|