dashql p95 1.93 s vs a normal 1.00 s ± 110 ms for Sun 21:00 UTC (3 of the last 4 checks)
info
Latency anomaly
automation
resolved
impact 12
Owner: Vishal Karedla (backup: Taylor Futral)
opened 9d ago · last observed 2d ago · observed 5× · #16906 · 5a58dcfc115ff0694fcc
What happened template
dashql's p95 is 1.9 s on automation, above a normal of 1.0 s ± 110 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 8.4, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.
p95 far above its hour-of-week normal — and at least 1 s; under a second is noise.
The facts
| measure | api_p95_ms |
| unit | ms |
| value | 1,925 |
| expected_median | 1,000 |
| expected_mad | 109.65 |
| z | 8.44 |
| rel | 0.93 |
| buckets_hit | 3 |
| n | 34 |
| weeks | 4 |
| slot | Sun 21:00 UTC |
| bucket | 2026-11-15 21:35:00 UTC (2d ago) |
| floor_ms | 1,000 |
| discriminator | api_p95_ms |
| window | 2026-11-15 21:25:00 UTC → 2026-11-15 21:45:00 UTC |
History
- 2026-11-15 21:55:27 UTCreopened x-ray · severity=info was=info title=dashql p95 1.93 s vs a normal 1.00 s ± 110 ms for Sun 21:00 UTC (3 of the last 4 checks) level=low was_level=low finding={'id': 16906, 'fingerprint': '5a58dcfc115ff0694fcc', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'automation', 'subject': 'dashql', 'title': 'dashql p95 1.93 s vs a normal 1.00 s ± 110 ms for Sun 21:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 1925.0, 'expected_median': 1000.0, 'expected_mad': 109.65063, 'z': 8.435884043712289, 'rel': 0.925, 'buckets_hit': 3, 'n': 34, 'weeks': 4, 'slot': 'Sun 21:00 UTC', 'bucket': 1794778500, 'window': [1794777900, 1794779100], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/dashql?cluster=automation&from=1794756300&to=1794779100'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794756300&to=1794779100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794779548, 'age_seconds': 186, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 6.8, 'status': 'open', 'first_seen': 1794177232, 'last_seen': 1794779727, 'resolved_at': None, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 5, 'narrative': "dashql's p95 is 1.5 s on automation, above a normal of 967 ms ± 62 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 8.6, 2 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794779727, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 602503, 'stale_s': 8}
- 2026-11-15 22:33:05 UTCresolved x-ray · reason=not observed grace_s=900 finding={'id': 16906, 'fingerprint': '5a58dcfc115ff0694fcc', 'kind': 'anomaly.latency', 'severity': 'info', 'cluster': 'automation', 'subject': 'dashql', 'title': 'dashql p95 1.93 s vs a normal 1.00 s ± 110 ms for Sun 21:00 UTC (3 of the last 4 checks)', 'facts': {'measure': 'api_p95_ms', 'unit': 'ms', 'value': 1925.0, 'expected_median': 1000.0, 'expected_mad': 109.65063, 'z': 8.435884043712289, 'rel': 0.925, 'buckets_hit': 3, 'n': 34, 'weeks': 4, 'slot': 'Sun 21:00 UTC', 'bucket': 1794778500, 'window': [1794777900, 1794779100], 'floor_ms': 1000, 'recent_deploys': [], 'discriminator': 'api_p95_ms'}, 'evidence': [{'label': 'Service', 'href': '/service/dashql?cluster=automation&from=1794756300&to=1794779100'}, {'label': 'Fleet vs normal', 'href': '/fleet?cluster=automation&from=1794756300&to=1794779100'}], 'provenance': {'source': 'rollup', 'collected_at': 1794779548, 'age_seconds': 186, 'stale': False, 'shown': None, 'total': None, 'truncated': False, 'coverage_from': None, 'retain_days': None, 'note': None}, 'impact': 11.8, 'status': 'resolved', 'first_seen': 1794177232, 'last_seen': 1794779727, 'resolved_at': 1794781985, 'acked_by': None, 'acked_at': None, 'snoozed_until': None, 'occurrences': 5, 'narrative': "dashql's p95 is 1.9 s on automation, above a normal of 1.0 s ± 110 ms for this hour of the week (3 of the last 4 five-minute buckets, z = 8.4, 4 weeks of history). Whether users noticed depends on which endpoints carried it — the service page shows the split. A p95 that stays under a second is never reported — below that nobody notices, whatever the deviation.", 'narrative_kind': 'template', 'discriminator': 'api_p95_ms', 'level': 'low', 'told_level': None, 'episode_at': 1794779727, 'chat_thread': None, 'kind_label': 'Latency anomaly', 'group': 'Anomalies', 'age_s': 604761, 'stale_s': 2266}
Check it yourself
Every link carries the finding's own time window, so it shows tomorrow what it shows now.
Actions
Sign in (IAP) to acknowledge, snooze or resolve.
Where the numbers come from
| source | rollup |
| collected_at | 1794779548 |
| age_seconds | 186 |
| stale | False |
| shown | None |
| total | None |
| truncated | False |
| coverage_from | None |
| retain_days | None |
| note | None |
| detector | seasonal |