Sample data. This database was generated by scripts/seed_dev_db.py —
every service, deploy, person and figure on this page is synthetic. Names are
sample.*@seed.invalid and image tags are prefixed SEED-.
Nothing here happened.
Ask views — 60 cases
Every Ask answer can now carry a view: charts, tables, timelines and buttons drawn from a small program the model writes (OpenUI Lang).
Below is each test case — the question, the tools and components a good answer reaches for, and a sample answer drawn exactly as Ask draws one.
Opening this page calls no model. Ask it live ↗ sends the question to Ask on the case's cluster.
Two things are broken on master right now, and both are Neo4j: the shared database master is failing 218 Cypher queries in 15 min (finding #7603, open since Oct 27 15:34 UTC), and matrix_a is failing 9 in 15 min (#16175, since Nov 4 14:36 UTC) 1. Nothing was deployed in the last 2 h, and first look found no linked change for either 1.
✗ 2 things broken on master, both Neo4j query failures 1 — No deploy in the last 2 h. First look: cause not clear from logs or recent changes, for both 1.
Broken now on master (UTC) 1 - Neo4j · shared database master — 218 Cypher queries failed in 15 min · critical · first look: no linked change (since Oct 27 15:34) - Neo4j · shared database matrix_a — 9 Cypher queries failed in 15 min · warning · first look: cause not clear (since Nov 4 14:36)
User journeys on master, last 60 min 1 - Sign-in & profile: ok — working - Trial execution & work orders: ok — working - Documents: idle — no calls - AI document processing: idle — no calls - Notifications: idle — no calls
Ask next: Which services send the failing Cypher on the shared database master? · Show the Neo4j errors for database master in the last hour · Is matrix_a on master failing for the same reason as database master?
Neither row has a CODEOWNERS owner; the leads listed are ojas.kale and samba.bandarupalli 1. First look counted 53 failures in its own 15-min sample (onset Nov 6 08:47 UTC), so the per-15-min count moves between reads 1.
Not quite. One journey is failing: Trial execution & work orders, 7 of 7 requests failed in the last hour, every one an execsvc 401 on POST /api/trialexec/v2/study/:id/activity/status1. Nothing was deployed in the last 2 h 1, and over 24 h production served 84,085 requests with only 211 5xx 2.
✗ 1 thing broken: Trial execution & work orders, 7 of 7 failed in the last hour 1 — execsvc 401 on POST /api/trialexec/v2/study/:id/activity/status (7). Owner Madhava Mukkamala, backup Shashank Agrawal 1.
Production now: journeys, last 60 min 1 · services, last 24 h 2 - Trial execution & work orders: bad — 7 of 7 failed, last hour 1 - execsvc: bad — 89.1% of 13,010 calls failed 2 - onboardingsvc: warn — 201 5xx · 1.3% errors 2 - notifsvc: warn — 6.1% of 1,231 streams failed 2 - virtual-assistant: warn — p95 5.9 s 2 - docsvc: ok — 0.15% errors · p95 526 ms 2 - docproc: ok — 0.04% errors · p95 981 ms 2 - enrollsvc: ok — 1.1% errors · p95 422 ms 2 - Sign-in & profile: ok — working 1 - Documents: idle — no calls in 60 min 1 - AI document processing: idle — no calls in 60 min 1 - Notifications: idle — no calls in 60 min 1
Ask next: Show the failing execsvc 401 calls on production in the last hour · Which caller is getting execsvc's 401s on production, and why? · Why is onboardingsvc returning 5xx on production?
The 14.3% error rate is mostly execsvc 4xx rather than outages. The 24 h totals miss the last 5 min, which is a coverage gap 2.
A03Cluster overview, 24 hproductionreal dataAsk it live ↗
Production handled 84,085 requests in the last 24 h at a 14.3% error rate, but almost all of that is execsvc 4xx: there were only 211 5xx, and 201 of them came from onboardingsvc 1. onboardingsvc is the one service low on error budget (8.5% left), and the other twelve are healthy 2.
#### Error budgets ⚠ onboardingsvc has 8.5% of its error budget left 2 — Every other service is healthy; execsvc's 4xx do not spend budget.
All 13 services, budget detail 2Error budgets, last 24 h, all 13 services; objectives derived 2
Service
Calls
Objective
Achieved
Budget left
Burn 6 h
Verdict
onboardingsvc
21,978
99%
99.09%
8.5%
0.65×
budget low
docsvc
31,596
99.9%
99.98%
84.2%
0.0×
healthy
execsvc
13,010
99.5%
99.96%
92.3%
0.0×
healthy
docproc
12,524
99.9%
100.00%
100%
0.0×
healthy
enrollsvc
1,836
99.9%
100.00%
100%
0.0×
healthy
notifsvc
1,231
99.9%
100.00%
100%
0.0×
healthy
tusker
671
99.9%
100.00%
100%
—
healthy
dashql
449
99.9%
100.00%
100%
—
healthy
virtual-assistant
413
99.9%
100.00%
100%
—
healthy
studiosvc
241
99.9%
100.00%
100%
—
healthy
schedsvc
84
99.5%
100.00%
100%
—
healthy
regsvc
44
99.9%
100.00%
100%
—
healthy
patientsvc
8
99.9%
100.00%
100%
—
healthy
Ask next: Why is onboardingsvc burning its error budget on production? · Show example failing execsvc calls on production in the last 6 hours · Why is virtual-assistant's p95 5.9 s on production?
Every objective is derived from the service's own last week, not a declared SLO. Budgets count only 5xx, which is why execsvc's 4xx don't spend budget 2. The last 5 min of the window are a coverage gap 1.
A04Open findings this weekmasterreal dataAsk it live ↗
What is wrong on master this week? Summarise the open findings, most important first.
410 findings were active on master in the last 7 days. 302 of them are error.novel (new error shapes), and 22 are high level. The two high-level ones on the latest page come first: Cypher failures on the shared database master (#7603, 218 in 15 min) and the workplan.summarization-sweep job failing (#5483, 28 failed runs in 6 h) 1.
Findings by level, last 7 days 1 - high: 22 - medium: 331 - low: 57
AI spend projects to $1,538 this month, 1.4× last month's $1,067
42×
Ask next: Why are Cypher queries failing on the shared database master? · Why is workplan.summarization-sweep failing on master? · Which error.novel findings on master appeared in the last 24 hours?
The table ranks only the latest page (25 of 410, page 1 of 17). 20 more high-level findings exist that this page doesn't show 1. I ranked by X-ray's level rather than raw severity, which marks 315 of the 410 as critical 1.
A05What changed in 6 hproductionreal dataAsk it live ↗
Nothing was deployed, hand-edited or rolled out on production in the last 6 hours: the change feed is empty 1. The change was in behaviour. 18 findings were active, two of them new tonight: the litellm HPA losing its scaling metric at 05:20 UTC (#16811) and a read-only filesystem on a node at 01:24 (#16786). Neo4j load ran at 99 q/min against a normal 32 for this hour 2.
No deploys, hand changes or rollouts on production in the last 6 h 1 — So none of the findings below can be tied to a change.
Production, last 6 h: findings by when X-ray saw them (UTC) 2 - 01:24 — Read-only filesystem on node gk3-production-nap-1sqli2in-94849021-hh7s: #16786 · first seen tonight, 2× - 01:39 — onboardingsvc blindedScopes: 12 of 12 calls failed: #4570, with budget burn 9.3× over 1 h (#1530) - 04:47 — Traffic at a normally idle hour: whole cluster 94.0 req/min vs 0.0 normal (#155) · docsvc Neo4j 326 q/min vs 0 (#649) - 05:02 — Whole-cluster Neo4j load 99 q/min vs 32 normal: #49 · 3 of the last 4 checks - 05:20 — litellm HPA cannot read its scaling metric: #16811 · first seen tonight, 4× in 2 h - 06:38 — exportsvc.activity_update still failing: #7927 · 18 failed runs in 6 h
Ask next: What drove the extra Neo4j load on production after 04:00 UTC? · Why can't the litellm HPA read its scaling metric on production? · What changed on production in the last 24 hours?
Times are when X-ray last saw each finding, not exact onsets. The "normal 0.0 req/min" baselines are ± 5 req/min wide at Sat 04:00, so those comparisons are weak 2.
Compare master and production over the last 24 hours: traffic, errors, latency, Neo4j load. Where do they differ most?
tools: overview
· components: Compare, BarChart
The biggest difference is Neo4j. Production ran 1,174,881 queries against master's 474,130, mostly from docsvc (816,705), yet master's Cypher is the slower one, at p95 51.1 ms against 9.3 ms 12. Production also carries far more traffic (84,085 vs 21,560 requests) and a 14.3% error rate, nearly all execsvc 4xx, against master's 2.2% 12.
Ask next: Why is studiosvc's p95 120 s on master? · Why does workplansvc run 270,268 Neo4j queries a day on master? · Why is master's Neo4j p95 51.1 ms when production's is 9.3 ms?
I left studiosvc (p95 120 s on master) and workplansvc (49.1 s) off the latency chart so it stays readable. studiosvc's 12,034,199 ms max is what drags master's average up to 32.5 s 1. Both windows end in a 5-min coverage gap 12.
One journey is failing on production: Trial execution & work orders, 7 of 7 requests failed in the last hour, every one an execsvc 401 on POST /api/trialexec/v2/study/:id/activity/status1. Three journeys are silent (Documents, AI document processing and Notifications had no calls in 60 min), and only Sign-in & profile is working 1. The last 2 h had just 5 user sessions, all signed out, with 0 errors 2.
✗ Trial execution & work orders: 7 of 7 failed in the last hour 1 — All execsvc 401 on POST /api/trialexec/v2/study/:id/activity/status. Owner Madhava Mukkamala, backup Shashank Agrawal 1.
Journeys on production, last 60 min 1 - Trial execution & work orders: bad — 7 of 7 failed · execsvc 401 - Documents: idle — no calls - AI document processing: idle — no calls - Notifications: idle — no calls - Sign-in & profile: ok — working
Signed-in users: none (all 5 sessions signed out 2)
Silent is not the same as broken — With 5 signed-out sessions in 2 h 2, the three idle journeys may just have no users at this hour. No calls means nothing to judge 1.
Ask next: Which caller sends execsvc POST /api/trialexec/v2/study/:id/activity/status on production? · When did the execsvc 401s on production start? · Is exportsvc.activity_update behind the execsvc 401s on production?
7 calls is a small sample, and the 2 h session read can't say which user or service sent them 2. Who calls the endpoint isn't in these results.
Performance
B01Is docsvc slower than usualproductionreal dataAsk it live ↗
Is docsvc slower than normal for this time of week?
There's no sign of it, though this exact slot can't be judged. docsvc has no baseline for Sat 06:00 UTC (the slot is normally idle) and has served no calls since 05:00 12. Its last busy hour, Sat 04:00 UTC, ran at p95 416 ms, at the low end of its weekly normal range of 340–3,125 ms 12.
p95, last busy hour: 416 ms (Sat 04:00 UTC, 1,056 calls 1)
Baseline for Sat 06:00: none (slot normally idle · 139 slots have history 2)
Weekly normal range: 340–3,125 ms (median p95 across the week's slots 2)
Calls since 05:00: 0 (nothing to judge right now 1)
docsvc p95 latency by hour, last 24 h (UTC) 1 (ms)
docsvc: Friday's busiest hour vs Saturday's last busy hour 1
Fri 08:00
Sat 04:00
p95 latency
465 ms
416 ms
Calls
3,923
1,056
Error rate
0.05%
0.00%
Neo4j load
1,936 q/min
163 q/min
Ask next: Is docsvc slower than normal on production on weekday mornings this week? · Which docsvc endpoints drove the 949 ms p95 at Fri 12:00 UTC? · Show docsvc p99 on production for the last 24 hours
Empty hours are gaps, not zeros, and the 06:00 hour is still filling (measured=false) 1. Without a baseline for this slot, "normal for this time of week" can't be stated. The next busy weekday hour can be 2.
Rank every production service by p95 latency over the last 6 hours.
tools: overview
· components: BarChart, Table
Over the last 6 h docsvc is the slowest production service at p95 479 ms (1,912 calls), followed by schedsvc at 375 ms and enrollsvc at 360 ms 1. Only 7 services served API calls in the window, and it was a quiet one: 4,481 requests from 6 users 1.
p95 latency by service, production, last 6 h 1 (ms)
p95
docsvc
479
schedsvc
375
enrollsvc
360
docproc
169
tusker
121
onboardingsvc
49
execsvc
1.9
⚠ schedsvc and tusker rank on 5 and 6 calls 1 — A p95 over a handful of calls is one slow request, not a trend. execsvc looks fast only because 97.7% of its 43 calls were rejected 1.
Every production service with API calls, last 6 h 1
Service
Calls
p95
Avg
Max
Errors
5xx
Neo4j q/req
docsvc
1,912
479 ms
165 ms
4,627 ms
0.00%
0
10.5
schedsvc
5
375 ms
237 ms
375 ms
0.00%
0
6.6
enrollsvc
169
360 ms
99 ms
951 ms
0.00%
0
6.9
docproc
298
169 ms
70 ms
251 ms
0.00%
0
4.3
tusker
6
121 ms
62 ms
121 ms
0.00%
0
7.7
onboardingsvc
1,845
49 ms
16 ms
381 ms
0.81%
12
1.5
execsvc
43
1.9 ms
2.0 ms
82 ms
97.67%
0
0.05
Ask next: Which docsvc endpoints have the highest p95 on production in the last 6 hours? · Rank production services by p95 over the last 24 hours · Why are 97.7% of execsvc calls failing on production?
virtual-assistant, regsvc and the background workers had no API calls in the window, so they aren't ranked 1. The last 5 min are a coverage gap.
B03Why was this request slowproductionreal dataAsk it live ↗
Why was this docsvc request slow? Break down where the time went.
Neo4j wasn't the cause. The request took 686 ms, but its 3 Cypher queries add up to only 12 ms, and the first one didn't start until 570 ms in 2. Most of the time passed inside docsvc before it touched the database, and none of the trace's log lines explain that gap 2.
⚠ 570 ms before the first query, and it isn't Neo4j 2 — GET /api/documents/collections/:id · 200 · Sat 04:36:09 UTC · [email protected] (tildainc). The 3 queries took 12 ms in total. The trace's 4 log lines are all INFO, and none of them falls in the gap 2.
Trace 8d7312f7…, GET /api/documents/collections/:id, 686 ms 2 - whole request: 686 ms (starts at 0 ms) - inside docsvc, before any Cypher: 570 ms (starts at 0 ms) - user and org access check: 3 ms (starts at 570 ms) - app time between queries: 84 ms (starts at 573 ms) - collection folder and its documents: 8 ms (starts at 657 ms) - actor lookup: 1 ms (starts at 680 ms)
MATCH (c:Folder{tildaid:$collectionFolderId, deltimestamp:0})
OPTIONAL MATCH (root:Folder{level:0, name:$rootName, deltimestamp:0})-[:INCLUDES]->(c)
WITH c, root
WHERE c.folderType = $ft AND c.level = 1 AND c.scope = $scope AND root IS NOT NULL
MATCH (o:Organization{tildaid:$orgId})<-[:STUDY_TO_ORG]-(st:Study{tildaid:c.scopeId})
OPTIONAL MATCH (c)-[:CREATED_BY]->(u:User)
Ask next: Is there a 500 ms gap before the first query in other docsvc collections requests today? · Show the slowest docsvc requests on production in the last 6 hours · What does docsvc do before its first query in GET /api/documents/collections/:id?
I picked the slowest of the 25 newest docsvc calls (686 ms). The window held 1,912, so this isn't the slowest request of the 6 h 1. What docsvc did during the 570 ms (auth, a downstream call, GC) isn't in the logs, so the code would have to say.
B04N+1 / chatty endpointsproductionreal dataAsk it live ↗
Which endpoints are chatty with Neo4j (N+1 patterns) on production?
Only one service stands out: virtual-assistant runs 103.5 Neo4j queries per request, against a peer median of 5.7 (robust z 16.1) 1. It is also the p95 (5.9 s) and p99 (12.6 s) outlier, which is what a per-request fan-out looks like. outliers compares services, not endpoints, though, so it doesn't name the endpoint 1.
✗ virtual-assistant is the only N+1 outlier on production, last 24 h 1 — 103.5 Neo4j queries per request vs a peer median of 5.7 across 9 peers. No other service crossed the bar.
Find the chatty endpoint - [ ] Pick a slow virtual-assistant session — journeys on production, then journey_detail to see Cypher counts per screen - [ ] Find the repeated shape — one plan hash run once per document or folder inside a single request is the N+1 - [ ] Batch it — replace the per-item query with one UNWIND $ids query - [ ] Re-check outliers after the fix ships — queries per request should fall towards the peer median of 5.7
Ask next: Which Cypher shapes does virtual-assistant run most per request on production? · Show one slow virtual-assistant session on production with its Cypher · Is virtual-assistant also an N+1 outlier on master?
outliers judges a service against its peers, not its own history, and skips services with under 20 calls 1. The fan-out is the likely reading, but no trace has confirmed it yet.
B05Slowest hours by serviceproductionreal dataAsk it live ↗
When during the day is production slowest? Show it by hour and service for the last 24 hours.
tools: timeseries
· components: Heatmap, Callout
Production is slowest in the early afternoon UTC. On Friday the cluster p95 peaked at 858 ms at 14:00 and 832 ms at 12:00 1, the same hours docproc reached 6.2 s (12:00) and virtual-assistant 8.2–9.2 s (13:00–14:00) 35. docsvc and onboardingsvc stay fairly flat all day 24.
⚠ The slow stretch is Fri 12:00–14:00 UTC 1 — docproc hit p95 6.2 s at 12:00 3, and virtual-assistant 8.2 s and 9.2 s at 13:00 and 14:00 on 95 calls each 5. Overnight the cluster is close to idle 1.
p95 latency by hour and service, last 24 h (UTC) 1–5
Fri 06
07
08
09
10
11
12
13
14
15
16
17
18
19
20
21
22
23
Sat 00
01
02
03
04
05
06
whole cluster
1
487
462
497
486
647
832
473
858
454
472
457
485
479
430
613
643
471
155
466
2
417
271
3
2
virtual-assistant
195
910
775
1,550
800
433
455
8,214
9,150
650
1,050
1,650
1,650
875
185
—
900
2,650
—
—
—
—
—
—
—
docproc
—
843
754
3,300
1,324
1,942
6,241
535
2,650
451
627
200
618
786
536
1,700
190
1,200
—
99
—
188
171
—
—
docsvc
1,163
479
465
561
479
462
949
500
886
603
516
805
795
651
462
625
883
635
—
767
—
464
416
—
—
onboardingsvc
98
94
92
96
92
88
73
68
93
80
85
87
88
93
65
84
86
82
135
49
2
88
48
2
2
Ask next: What made docproc's p95 6.2 s at 12:00 UTC Friday on production? · Why did virtual-assistant reach 9.2 s p95 at 14:00 UTC Friday? · Is the 12:00–14:00 UTC slowdown on production there every weekday?
This is one day, so it shows when Friday was slow, not a proven daily pattern. Blank cells are hours with no calls, and the near-zero cluster values overnight come from a few rejected calls 1. I read four services, not every one.
B06Did the deploy slow it downproductionreal dataAsk it live ↗
No. The last docsvc deploy, v2.10.43-86c75b5 at Oct 30 14:21 UTC, was followed by a faster hour: p95 554 → 454 ms and p99 3,607 → 499 ms 1. Daily p95 has stayed between 474 and 512 ms on every weekday since 2.
docsvc, 1 h before vs 1 h after the Oct 30 14:21 UTC deploy 1
before
after
p50
163 ms
141 ms
p95
554 ms
454 ms
p99
3,607 ms
499 ms
Error rate
0.31%
0.12%
Calls
325
808
docsvc on production (UTC) 12 - Oct 30 06:15 — v2.10.42-0f0f52e synced: revision 9dc97ef16 · manual sync by samba.bandarupalli · p95 4,421 → 629 ms, but only 27 calls before 1 - Oct 30 14:21 — v2.10.43-86c75b5 synced, the last deploy: revision 4273d9ebb · manual sync by samba.bandarupalli 1 - Oct 30 15:21 — One hour after: p95 454 ms, p99 499 ms: 808 calls, 0.12% errors 1 - Nov 7 — No docsvc deploy since · p95 479 ms so far today: 12
Threshold: 554 p95 in the hour before the deploy 1
Ask next: What changed in docsvc between v2.10.42 and v2.10.43? · Why was docsvc's p95 838 ms on Sunday Nov 1 on production? · Did the Oct 30 06:15 docsvc deploy change its error rate on production?
The comparison covers one hour on each side, and calls rose from 325 to 808, so time of day is mixed in 1. Oct 30 and Nov 7 are partial days, and the weekend's 657–838 ms ran on only 633 and 405 calls 2.
B07virtual-assistant p95productionreal dataAsk it live ↗
Two busy hours explain it. On Friday at 13:00 and 14:00 UTC virtual-assistant served 95 calls each at p95 8.2 s and 9.2 s, while most other hours stayed under 1.7 s 1. It also runs 103.5 Neo4j queries per request against a peer median of 5.7, so a per-request fan-out is the first suspect 3. Nothing reached it in the last 6 h, so there was no request to trace 2.
Confirm the cause - [ ] Pull the slow calls from Fri 13:00–15:00 UTC — api_calls for virtual-assistant in that window, slowest first - [ ] Trace two of them with Cypher — trace_detail with cypher=true: count the queries per request and see which shape repeats - [ ] Batch the repeated query — if one shape runs per document or folder, fold it into one UNWIND query - [ ] Re-check against the 478 ms peer median — outliers on production after the fix ships
Ask next: Show the slowest virtual-assistant calls on production Fri 13:00–15:00 UTC · Which Cypher shapes does virtual-assistant run per request on production? · Is virtual-assistant's p95 also over 5 seconds on master?
The fan-out is a lead, not a proven cause. outliers compares with peers, not with history 3, and the latency spikes still need a trace from Friday afternoon to tie them to the queries.
Neo4j & Cypher
C01Cypher that got slowerproductionreal dataAsk it live ↗
Which Cypher queries got slower on production in the last 24 hours?
Only one shape really got slower: virtual-assistant's folder-tree query 32395444 went from p95 4.6 ms to 977 ms (210× drift) across the day, on 77 executions 1. The rest of the list roughly doubled from a low base. enrollsvc's 9D3836B5 (17.8 → 51.2 ms over 388 runs) is the only other one worth watching 1.
✗ 32395444 is the one real regression, p95 4.6 ms to 977 ms 1 — virtual-assistant · 77 runs · max 1,095 ms. The next biggest, enrollsvc 9D3836B5, went from 17.8 ms to 51.2 ms over 388 runs 1.
p95 early vs late in the window, top 10 degrading shapes, last 24 h 1 (ms)
p95 early
p95 late
32395444 · virtual-assistant
4.6
977
81C0B9EE · tusker
185
365
856B8DA1 · regsvc
48.5
97.5
9D3836B5 · enrollsvc
17.8
51.2
017A8E7A · dashql
9.7
18.8
30AA9AC1 · virtual-assistant
2
3.9
CBF2C7D7 · docsvc
2
3.9
A3BB82EA · docproc
1.95
3.7
6E20C5D2 · dashrsvc
2
3.5
C5AEF889 · (internal)
0.95
1.95
Cypher shapes that got slower on production, last 24 h, top 10 of 808 1
MATCH (st:Study {tildaid: $study_id})-[:USES]->(:FolderManager)-[:INCLUDES]->(tmf:Folder {name:?}) WHERE tmf.level = ? MATCH (tmf)-[:INCLUDES*?..?]->(d:Document) RETURN DISTINCT d.tildaid AS doc_id
Ask next: Why did Cypher shape 32395444 get 210× slower on production? · Which virtual-assistant requests run shape 32395444 on production? · Is enrollsvc shape 9D3836B5 also slower on master?
Six of the ten went from about 2 ms to about 4 ms, which is noise at that scale. Drift compares the early and late parts of the window, so it doesn't say when the change happened 1.
C02Busiest Cypher shapesmasterreal dataAsk it live ↗
What are the busiest Cypher shapes on master, and who runs them?
The busiest Cypher shapes on master are cheap user lookups by execsvc: 308B13B1 and 046B6411 ran 13,759 and 13,074 times in 24 h at p95 under 2 ms 1. The shapes that cost database time are workplansvc's background polls. Five of the top ten belong to it, led by DCDC23FF at 77.7 ms average over 6,830 runs 1.
Executions by shape, master, last 24 h, top 10 of 1,473 1
executions
308B13B1 · execsvc
13,759
046B6411 · execsvc
13,074
E083AA2B · enrollsvc
10,254
9BF74CB7 · workplansvc
6,842
DCDC23FF · workplansvc
6,830
38102FE8 · workplansvc
6,446
6F0FF668 · workplansvc
6,032
ECCD8C56 · enrollsvc
5,840
1A26C00F · enrollsvc
5,631
0A4540E4 · workplansvc
5,549
Who runs the top 10: executions by service 1 - workplansvc: 31,699 - execsvc: 26,833 - enrollsvc: 21,725
MATCH (wp:WorkPlanInstance)-[:WORKPLAN_INSTANCE_TO_MILESTONE]->(:WorkPlanMilestone)-[:REL_WORKPLAN_INSTANCE_MILESTONE_TO_ITEM]->(item:WorkplanInstanceMilestoneItem) WHERE wp.state <> ? AND coalesce(item.skippedAt, ?) = ? AND coalesce(item.notApplicableAt, ?) = ? AND (coalesce(item.workOrderCreateUnresolvableAt, ?) = ? OR $nowMs - item.workOrderCreateUnresolvableAt >= $windowMs) AND ( (item.itemType = ? AND (coalesce(item.readyAt, ?) > ? OR coalesce(item.vaWoInstanceFetchRequestSentAt, ?) > ?) AND coalesce(item.workOrderId, ?) = ? AND coalesce(item.expectedAttachmentUnresolvedAt, ?) = ? AND (coalesce(item.vaWoInstanceFetchRequestSentAt, ?) = ? OR $nowMs - item.vaWoInstanceFetchRequestSentAt >= $windowMs)) OR (item.itemType = ? AND coalesce(item.readyAt, ?) > ? AND coalesce(item.edlEventScopeId, ?) = ? AND coalesce(item.edlScopeCreateFailedAt, ?) = ? AND (coalesce(item.edlScopeCreateRequestSentAt, ?) = ? OR $nowMs - item.edlScopeCreateRequestSentAt >= $windowMs)) ) RETURN item.tildaid AS id LIMIT $limit
Ask next: Why does workplansvc poll shape DCDC23FF 6,830 times a day on master? · Why does execsvc's user-by-email lookup run under two plans on master? · Which Cypher shapes on master use the most total database time?
The donut sums only these ten shapes out of 1,473 1. 308B13B1 and 046B6411 are the same user-by-email statement under two plans. 308B13B1 has no late-window p95, so it stopped running part-way through the day, possibly when 046B6411 took over 1.
C03Explain one shapeproductionreal dataAsk it live ↗
Explain the slowest Cypher shape on production: why is it slow and what would fix it?
Strictly, the top of the p95 list is tusker's 522F6906 at 4.85 s, but it ran once: a one-off MERGE of Inbox nodes across every Organization 1. The slowest shape that runs repeatedly is virtual-assistant's 32395444, a folder-tree walk to every Document under a study's TMF folder, at p95 966 ms over 77 runs 1. It isn't a plan change. The same plan ran at p50 1.9 ms at 15:00 and 471 ms at 23:00 on Friday, so its cost follows what it walks 2.
Shape 32395444 by bucket, Fri (UTC), its only two buckets 2 (ms)
p50
p95
15:00
1.9
4.6
23:00
471
977
Why it's slow — The [:INCLUDES*?..?] walk visits every folder below the TMF folder before DISTINCT collapses the documents. The plan is the same at both times, so the jump in p50 points at a bigger tree being walked, not a worse plan 2.
MATCH (st:Study {tildaid: $study_id})-[:USES]->(:FolderManager)-[:INCLUDES]->(tmf:Folder {name:?}) WHERE tmf.level = ? MATCH (tmf)-[:INCLUDES*?..?]->(d:Document) RETURN DISTINCT d.tildaid AS doc_id
What would fix it - [ ] PROFILE it for a slow study — take a study_id from a 23:00 run and PROFILE the statement: count db hits on the INCLUDES expansion - [ ] Stop walking the tree — if Documents carry their study id, match them by an indexed property and keep only the folder filter - [ ] Cache per study in virtual-assistant — 65 runs in one hour suggests the same document list is fetched repeatedly - [ ] Confirm with shape_detail — p95 should fall back toward the 4.6 ms seen at 15:00
Ask next: Which virtual-assistant requests ran shape 32395444 at 23:00 UTC Friday? · Show Cypher shape 32395444 on production over the last 7 days · Is tusker's Inbox MERGE 522F6906 a startup job on production?
The shape has only two buckets of data, since it first ran Fri 14:50 UTC 2. "A bigger tree" is the likely reading, not something the tools measured, and PROFILE will confirm or rule it out.
Healthy. All 3 of 3 production Neo4j servers are up with 0 restarts, the page cache serves 100% of reads, and CPU is near idle at about 0.04 cores each over the last 6 h 1. I couldn't read transactions, because that chart was trimmed from the result 1.
✓ Neo4j on production is healthy, last 6 h (00:50–06:50 UTC) 1 — 3 of 3 servers up, 0 restarts, page cache hit ratio 100%, page faults at most 0.012/s, around 01:42 UTC.
⚠ The disk reading looks wrong 1 — The volume metric reports 4,096 bytes used of 101 GB on every server. Treat disk as unknown, not as empty.
Ask next: Show Neo4j transactions and lock waits on production for the last 6 hours · Was the page-fault blip on production Neo4j at 01:42 UTC tied to a slow query? · How healthy is the shared Neo4j that master uses?
Memory is the container's non-evictable memory, not a heap vs page-cache split 1. 93% page-cache usage is fine. Being pinned at 100% would mean the hot set had outgrown the cache.
C05Cypher failures on mastermasterreal dataAsk it live ↗
Why are Cypher queries failing on the shared database master, and should we do something?
It's mostly one bug. enrollsvc's study–SMO binding handler tries to create a ChartReviewStudy node that already exists, Neo4j rejects it with ConstraintValidationFailed, and the Pub/Sub message is retried: 874 such failures in the last hour2, with one trace failing 4 times between 06:46 and 06:50 UTC 3. Yes, it's worth fixing. The database itself is fine, but those bindings never complete, and the retries keep finding #7603 critical at 218 failures in 15 min 1.
✗ enrollsvc keeps re-creating ChartReviewStudy nodes that already exist 23 — The handler logs that the user is already a member, then fails to CREATE the chart-review study node, returns an error, and the same message comes back. First look on #7603 found no cause, and this is the app-side evidence it lacked 1.
ChartReviewStudy constraint failures: 874 (enrollsvc, last hour 2)
Error groups on master: 295 (last hour, page 1 of 12 read 2)
Cypher failures on the shared database master (UTC) - Oct 27 15:34 — Finding #7603 opens: Cypher failures on database master: critical · leads ojas.kale, samba.bandarupalli 1 - Nov 6 08:47 — Onset of the current run, per first look: no linked change found 1 - Nov 7 05:51 — Already failing when the 1-hour read starts: ConstraintValidationFailed from internal/study.go:214 and internal/pubsub.go:115 2 - 06:46 — Trace 72d887ff…: binding message received, CREATE fails, handler returns error: 4 attempts between 06:46 and 06:50 3 - 06:50 — Still failing: last error 06:50:51 23
study smo binding msg received
User is already a member. Updating the same actor
could not add chart review study node while binding study to org — Neo4jError: Neo.ClientError.Schema.ConstraintValidationFailed …
pubsub message handler returned error
Shall I file a Linear ticket to make enrollsvc's ChartReviewStudy create idempotent (MERGE, or treat already-exists as done)? — Yes, file it for the enrollsvc owners / Raise a PR that MERGEs instead of CREATE / No, keep watching
The full error goes on: (Node(10378619) already exists with label ChartReviewStudy and properties orgid = '7374e427-32…3. I read only page 1 of the 12 error groups 2, and matrix_a's 9 failures (#16175) weren't examined 1. The 874 in an hour lines up with #7603's 218 per 15 min, but the tools don't link the two directly.
Errors & logs
D01What keeps failingproductionreal dataAsk it live ↗
What keeps failing on production, with an example?
The one app failure that keeps going is exportsvc's activity updates: 43 errors in 6 h, steady from 00:52 to 06:46 UTC. Each one is a 401 from execsvc, because the token exportsvc sends has expired 12. The bigger counts are platform noise (cilium-agent 1,563, fluentbit) or one-off bursts: docproc's pipeline failures all landed at 01:15, and onboardingsvc's twelve 500s at 01:11–01:13 1.
Error lines by group, production, last 6 h, top 10 of 173 1
platform
app
cilium-agent · foreign IP
1,563
—
cilium metrics collector
196
—
fluentbit-gke · parse
154
—
docproc · processing log
—
116
fluentbit-gke · parse (2)
105
—
docproc · slots in use
—
58
exportsvc · 401 activity update
—
43
docproc · pipeline failed
—
29
docproc · no agent token
—
29
metrics-server · scrape timeout
26
—
What keeps failing on production, last 6 h (UTC) 1
Service
Error
Count
First
Last
Pattern
exportsvc
Error processing activity update: 401 Unauthorized from /api/trialexec/v2/study/:id/activity/status
500 · failed to get study: get with empty tildaId (blinded-scope lookup)
12
01:11
01:13
one burst
metrics-server
Failed to scrape node, timeout to access kubelet
26
01:12
06:13
platform
cilium-agent
Found a foreign IP address with the ID of the current node
1,563
01:13
06:13
platform
⚠ exportsvc is calling execsvc with an expired token 2 — On the same trace, execsvc logs token validation failed and Unauthorized: token expired, and exportsvc logs the 401. It has repeated all night 1.
06:46:12 WARNING execsvc token validation failed, sending 401
06:46:12 INFO execsvc Request completed — 401 · Unauthorized: token expired
06:46:12 WARNING exportsvc 401 Unauthorized - token may be expired or invalid
06:46:12 ERROR exportsvc Error processing activity update: API request failed: Client error '401 Unauthorized' for url 'api.tilda.bio/api/trialexec/v2/study/… ↗
Ask next: Where does exportsvc get the token it sends to execsvc on production? · What made docproc's pipelines fail at 01:15 UTC on production? · Show onboardingsvc's blindedScopes 500s on production at 01:11 UTC
This is page 1 of 7 (25 of 173 groups) 1. Counts are log lines, not failed requests, and docproc writes several lines per failed pipeline.
D02execsvc error rateproductionreal dataAsk it live ↗
Why does execsvc have an 89% error rate on production?
execsvc's 89.1% error rate on production comes almost entirely from 4xx rejections. The service isn't failing: of 13,010 calls in the last 24 h only 5 were 5xx 1, and its log has no ERROR-level groups in the last 6 h 2. Something is sending execsvc requests it refuses.
⚠ execsvc is refusing requests, not crashing 1 — About 11,593 of 13,010 calls in 24 h were 4xx; only 5 were 5xx, and execsvc wrote no ERROR log groups in the last 6 h 2.
execsvc calls by outcome, last 24 h 1 - 4xx (refused): 11,593 - succeeded: 1,412 - 5xx: 5
No ERROR log groups for execsvc, last 6 h 2 — 4xx responses are logged below ERROR, so the errors tool has nothing to group. To see the status codes and endpoints, read api_calls.
Ask next: Which execsvc endpoints return 4xx on production in the last hour? · Which orgs or callers hit execsvc's 4xx on production today? · Did execsvc's error rate change after its last production deploy?
The 4xx and success counts are derived (calls × error rate, minus the 5 5xx). I couldn't tell which status codes (401/403/404) or which caller from these tools. The overview also has a 5-minute coverage gap at 06:45 UTC 1.
D03onboardingsvc 5xxproductionreal dataAsk it live ↗
Show the onboardingsvc 5xx errors over the last 24 hours and what they are.
onboardingsvc returned 201 5xx in the last 24 h. 145 of them came in one burst on Nov 6 between 12:15 and 12:59 UTC 1. In that burst, every failure is the same bug: a study lookup with an empty tildaId, which the data layer refuses 2.
5xx, last 24 h: 201 (Nov 6 06:00 – Nov 7 06:00 UTC 1)
Worst hour: 145 (12:00 UTC Nov 6, 9.2% of calls 1)
p95 in that hour: 73 ms (the requests failed fast 1)
onboardingsvc 5xx per hour on production, Nov 6 06:00 – Nov 7 06:00 UTC 1
Request completed — 500 · failed to get study: … get with empty tildaId
145
12:15
12:59
handlers/errors.go:68
request failed — 500
145
12:15
12:59
health/actor.go:49
could not get study for blinded-scope lookup — … empty tildaId
145
12:15
12:59
Request completed — 500 · failed to get study: objLabel Study: get with empty tildaId requires a struct with at least one non-inherited dbkey field; a {deltimestamp} filter would match a
Ask next: What were the other 56 onboardingsvc 5xx on production (16:00, 17:00 and 01:00 UTC)? · Did a deploy to onboardingsvc on production precede 12:15 UTC Nov 6? · Which orgs hit the empty-tildaId study error on production?
The three groups share one trace, so they are most likely one failure logged three times: 145 failing requests, not 435. I read error logs only for 12:00–13:00 UTC, so the other 56 5xx (16:00, 17:00 and 01:00 UTC hours) are not explained here.
D04Log patterns for a serviceproductionreal dataAsk it live ↗
Summarise docproc's error log patterns in the last hour.
tools: logs
· components: Table, BarChart, Code
docproc wrote no WARNING or ERROR lines on production in the last hour 1. Over the last 6 h there was one burst, at 01:15 UTC on Nov 7: 380 lines in a single minute, which included 29 failed document pipelines and repeated bot-token refresh failures 2.
✓ Quiet for the last hour 1 — 0 WARNING+ lines from 05:52 to 06:52 UTC. The figures below cover the last 6 h 2.
ServiceTokenManager: failed to get bot token … in backoff
WARNING
28
01:15:05–01:15:08
same traces as the failures
docsvc checksum update skipped: no agent token
ERROR
29
01:15:05–01:15:08
asks for a TildaSense bot token
[…] Context …s -> Download FAILED
ERROR
18
01:15:05–01:15:08
7 variants of one line
[…] ═══ and N/20 slots in use
ERROR
174
01:15:05–01:15:08
context dump printed at ERROR
Connection pool is full, discarding connection: storage.googleapis.com
WARNING
2
01:15:07
pool size 10
Failed to fetch Pub/Sub num_unacked: 504 Deadline expired
WARNING
1
02:30:13
separate, one-off
ServiceTokenManager: failed to get bot token: ServiceTokenManager: token refresh in backoff after a recent failure and no usable cached token yet
Ask next: Why did docproc's bot-token refresh fail on production at 01:15 UTC Nov 7? · Show trace dcfb72e61b11e600d67cc8d448983a20 on production · Has docproc's token-backoff failure happened on production before this week?
The ERROR count overstates the failures: 174 of the 349 ERROR lines are the pipeline's context dump (banners and slot counts) logged at ERROR. The pattern list is the top patterns only, covering 283 of the 381 lines.
D05When did the spike startmasterreal dataAsk it live ↗
When did the error spike on master start, and what was deployed around then?
The 5xx spike on master began in the 11:00 UTC hour on Nov 6 with 29 5xx. It peaked at 12:00 with 95 5xx (6.2% of calls) and was gone by 13:00 1. The nearest deploy was docproc master-713dc89, which ArgoCD synced at 11:08 UTC. Four more services followed at 11:18–11:19 2. That timing makes both adjacent to the spike, not proven causes.
master error rate per hour, Nov 6 06:00 – Nov 7 05:00 UTC 1 (%)
all errors
5xx
06:00
—
—
07:00
0.98
0.05
08:00
2.18
0.25
09:00
0.2
0
10:00
0.49
0.03
11:00
2.93
1.03
12:00
6.33
6.2
13:00
1.75
0
14:00
16.17
0
15:00
0.29
0
16:00
1.76
0
17:00
0.58
0
18:00
3.54
0
19:00
0.84
0
20:00
0
0
21:00
0
0
22:00
0
0
23:00
0
0
00:00
0
0
01:00
0
0
02:00
1.19
0
03:00
2.54
0
04:00
0
0
05:00
0
0
What changed on master around the spike, Nov 6 (UTC) 12 - 06:55 — Fleet sync to master-11306eb: ArgoCD rev 882d8d8dc: docsvc, execsvc, onboardingsvc, enrollsvc, notifsvc and more 2 - 09:07 — docproc → master-3d14e1b: ArgoCD rev 256ee762e 2 - 09:36 — x-ray 3.9.16 rolled out by hand: [email protected] via kubectl 2 - 09:53 — x-ray 3.9.17 rolled out by hand: [email protected] via kubectl 2 - 11:00 — 5xx begin: 29 in the hour (1.0%): 1 - 11:08 — docproc → master-713dc89: ArgoCD sync, rev 0476ae4b4 2 - 11:18 — rag-agent, exportsvc, feedsvc, studioaisvc → master-792bf41: ArgoCD rev 85850840a 2 - 12:00 — Peak: 95 5xx, 6.2% of calls: p95 latency 28.4 s in that hour 1 - 13:00 — 5xx back to 0: 1 - 14:00 — Separate 4xx rise: 16.2% errors, 0 5xx: 1 - 14:10 — Fleet sync to master-fe8a323: ArgoCD rev 580db29e1 2
⚠ Adjacent, not proven — docproc's deploy landed inside the first spike hour, but hourly buckets can't show whether the 5xx began before or after 11:08, and the cluster series doesn't name the failing service.
Ask next: Which service returned the 5xx on master between 11:00 and 13:00 UTC Nov 6? · What changed in docproc between master-3d14e1b and master-713dc89? · What caused the 4xx rise on master at 14:00 UTC Nov 6?
Traffic on master is low (about 25–60 requests/min in those hours), so a few dozen failures move the percentage a lot. The 06:00 bucket was not measured.
Pods & Kubernetes
E01Restarts and OOMsproductionreal dataAsk it live ↗
Which pods restarted or were OOMKilled on production this week?
Only 2 containers restarted on production in the last 7 days: one chat-agent pod and one docproc pod, once each 1. I can't confirm OOMKills either way. The Kubernetes event scan returned 0 events for the week, so an empty OOM list here means missing data, not proof that none happened 1.
⚠ OOM status unknown for this week 1 — The Kubernetes events read covered 0 events, so OOM, eviction and scheduling counts are missing. Each workload's pod page reads about 2 days of events and can show the reason.
Ask next: Why did docproc-9d58f6f65-dt8xh restart on production? · Did any production pods get OOMKilled in the last 48 h? · Which production services threw 5xx after the Nov 7 01:12 UTC spot preemption?
The node-event list is paged (25 of 44 rows shown), but the counts above (22 + 22) are the tool's totals.
E02Why does a pod keep restartingproductionreal dataAsk it live ↗
docproc isn't restarting repeatedly: it restarted once in the last 7 days, on pod docproc-9d58f6f65-dt8xh2, and there are no restart events since Nov 5 06:50 UTC 1. Over the last 48 h, memory peaked at 2.09 GiB against a 5 GiB limit and CPU peaked at 0.78 of 4 cores. Neither suggests an OOM kill or throttling 1.
✓ One restart in 7 days, none in the last 48 h 12 — That restart is older than X-ray's ~2-day event history (from Nov 5 06:50 UTC), so its reason — OOM, crash or probe — is not on record.
docproc on production (UTC) 12 - Oct 31 – Nov 5 — 1 container restart: pod docproc-9d58f6f65-dt8xh, inside the 7-day window 2 - Nov 5 06:50 — Restart-reason history begins: no restart events since 1 - Nov 6 17:56 — Spot node preempted on production: gk3-production-nap-1sqli2in-94849021-lblh; docproc did not restart 2 - Nov 7 01:12 — Spot node preempted on production: gk3-production-nap-1sqli2in-94849021-sxc4; docproc did not restart 2 - Nov 7 06:02 — Now: 1 pod, 0.87 GiB, 0.03 cores: 1
docproc resources against limits, last 48 h 1 - Memory now: 0.87 GiB of 5 (request 4 GiB)
Memory peak: 2.09 GiB of 5 (mean 1.26 GiB)
CPU now: 0.03 cores of 4 (request 1 core)
CPU peak: 0.78 cores of 4 (mean 0.16 cores)
To find that one restart's reason - [ ] Open docproc's pod page — docproc pods: restarts and last state per container - [ ] Read the restarted container's last state — kubectl -n tilda describe pod docproc-9d58f6f65-dt8xh: Last State shows OOMKilled, Error or a probe failure - [ ] Watch for a repeat — If it restarts again, the event falls inside the 2-day window and the reason will show here
Ask next: Show docproc's memory on production over the last 7 days · Did docproc log errors on production before its restart? · Which production workloads restarted in the last 48 h?
The spot preemptions are cluster-wide node events. Nothing ties them to docproc's restart.
Yes: master had 8 node incidents this week, 3 spot preemptions (each followed by an auto-repair) and 5 live migrations for host maintenance 1. None was followed by 5xx on services sharing the node. All three preemptions hit the same spot pool, z5asznva-3730cfb41.
Node incidents on master, last 7 d (UTC) 1 - Nov 3 09:22 — Live-migrated for host maintenance: gk3-master-nap-wcc7s7td-1469370b-7pvb - Nov 4 21:43 — Live-migrated for host maintenance: gk3-master-nap-wcc7s7td-1469370b-77g4 - Nov 4 22:03 — Spot node preempted, then auto-repaired: gk3-master-nap-z5asznva-3730cfb4-ndcm - Nov 4 23:59 — Spot node preempted, then auto-repaired: gk3-master-nap-z5asznva-3730cfb4-bd4q - Nov 5 15:04 — Live-migrated for host maintenance: gk3-master-nap-wcc7s7td-ed06d1b1-qgjq - Nov 5 16:48 — Spot node preempted, then auto-repaired: gk3-master-nap-z5asznva-3730cfb4-4nmf - Nov 5 22:19 — Live-migrated for host maintenance: gk3-master-nap-wcc7s7td-56b443e2-krmp - Nov 6 19:47 — Live-migrated for host maintenance: gk3-master-nap-wcc7s7td-228a94b5-v5cl
Pods on these nodes were not mapped 1 — Each incident lists 0 pods because X-ray could not map the node's pods (mapped = false), not because the node was empty. Watched namespaces: tilda, neo4j-system, weaviate.
Ask next: Which master pods ran on the gk3-master-nap-z5asznva-3730cfb4 nodes this week? · Any node incidents or preemptions on production this week? · Did master's error rate rise after the Nov 5 16:48 UTC preemption?
"0 5xx after" holds only for services X-ray could place on each node. With no pod mapping, a disruption elsewhere wouldn't show up here.
Yes: production matches git. ArgoCD reports it clean, with 0 objects changed by hand in the last 24 hours 1. This check runs per environment, not per app. It tells you nothing was changed outside git, but not whether every app's last sync succeeded.
ArgoCD drift, last 24 h 1 - production: ok — clean: matches git, 0 hand changes
What "in sync" covers here 1 — A scale change counts as capacity, not drift, because git doesn't set replica counts. Restarts and service-account changes are excluded, so every hand change listed would be a person.
Ask next: Was anything on production changed by hand in the last 7 days? · Is master in sync with git right now? · Which ArgoCD syncs ran on production in the last 24 h?
The window is 24 h by default. Ask for up to 168 h to cover the week.
E05Where is docsvc deployedproductionreal dataAsk it live ↗
Which clusters deploy docsvc, and which image tags do they pin?
tools: k8s_map
· components: Table, Tags
6 overlays in ash-nazg deploy docsvc: production, delta, nexus, master, automation and dev 1. They pin 4 different tags. Production runs v2.10.46-c571bfa, delta v2.10.38-beta-dafae8a and nexus production-d716e5c; master, automation and dev all pin master-c3412ef2.
Tags pinned for docsvc 2: v2.10.46-c571bfa, v2.10.38-beta-dafae8a, production-d716e5c, master-c3412ef
Rendered from a Helm chart 1 — The Deployment comes from the tilda-service chart (release docsvc). The overlays only patch it and pin the tag.
Ask next: What changed in docsvc between v2.10.38-beta-dafae8a (delta) and v2.10.46-c571bfa (production)? · Is docsvc on production actually running v2.10.46-c571bfa right now? · Which overlays deploy docproc, and which tags do they pin?
This is what git pins on the default branch. What each cluster is running right now could differ if a sync is pending. The map doesn't say which cluster the dev overlay targets.
Budgets & SLOs
F01Closest to its budgetproductionreal dataAsk it live ↗
Which service is closest to exhausting its error budget, and when?
onboardingsvc is closest: it has spent 92% of its 24-hour error budget and has 8.1% left, after 201 5xx against an allowance of 219 1. I can't say when it would run out. Every production budget forecast reports incomplete rolling-window history, so none has an exhaustion date 2.
Error budget left on production, last 24 h 1 - onboardingsvc: 8.1% of 100 (201 of 218.8 bad events spent)
docsvc: 84.1% of 100 (5 of 31.4 spent)
execsvc: 92.3% of 100 (5 of 64.6 spent)
docproc: 100% of 100 (0 of 12.5 spent)
enrollsvc: 100% of 100 (0 of 1.8 spent)
notifsvc: 100% of 100 (0 of 1.2 spent)
⚠ No exhaustion date available 2 — All 22 budget:production forecasts have eta = null (reason: incomplete rolling-window history). Over the last 6 h onboardingsvc burned at 0.65×, below the 1× pace that empties a budget on schedule 1.
Ask next: What were onboardingsvc's 201 5xx on production in the last 24 h? · Show onboardingsvc's error-budget burn on production over the last 7 days · Which production services have a declared rather than derived objective?
Every objective here is derived from the service's own last week, not a declared SLO. The budget is a rolling 24 h window, so onboardingsvc recovers as its Nov 6 12:00 UTC burst ages out.
F02Burn rate for one serviceproductionreal dataAsk it live ↗
What is onboardingsvc's error-budget burn rate and how long until it runs out?
onboardingsvc is burning at 0.65× over the last 6 h, slower than the pace that would empty its budget. The last 24 h have still used 92% of it, leaving 8.1%1. I can't give a run-out date: its forecast has no ETA because the rolling-window history is incomplete 2.
onboardingsvc error budget left, last 24 h: 8.1% of 100 (201 of 218.8 allowed bad events spent 1)
Burn rate, 6 h: 0.65× (1× empties the budget on schedule 1)
onboardingsvc 5xx per hour on production, Nov 6 06:00 – Nov 7 06:00 UTC 3
5xx
06:00
0
07:00
0
08:00
0
09:00
0
10:00
0
11:00
0
12:00
145
13:00
0
14:00
0
15:00
0
16:00
6
17:00
38
18:00
0
19:00
0
20:00
0
21:00
0
22:00
0
23:00
0
00:00
0
01:00
12
02:00
0
03:00
0
04:00
0
05:00
0
Threshold: 9.1 even burn ≈ 9.1/h (derived)
One burst spent most of it 3 — 145 of the 201 bad events came in the 12:00 UTC hour on Nov 6. Only two other hours went over the even-burn line: 17:00 (38) and 01:00 (12).
Ask next: What were onboardingsvc's 145 5xx at 12:00 UTC Nov 6 on production? · What is onboardingsvc's error budget on production over the last 7 days? · Did a deploy precede onboardingsvc's 5xx burst on production?
The even-burn line is derived (the 24 h allowance ÷ 24). The objective is derived from last week. Because the window rolls, the 12:00 burst stops counting about 24 h later, and the budget recovers then unless new errors come in.
F03Budget forecastproductionillustrative dataAsk it live ↗
Forecast error-budget exhaustion for every production service.
Only onboardingsvc is on course to exhaust its 30-day error budget on production, around Nov 22 (band Nov 18–27) 1. notifsvc follows around Dec 17. Every other production service is flat or not trending toward its threshold 1.
⚠ onboardingsvc runs out around Nov 22 1 — It has spent 52% of a 3,088-bad-event budget and is adding about 21 points a week. Its objective (99%) is derived from its own last week 2.
Budget spent, 30-day window: actual to Nov 7, Theil–Sen projection after 1 (%)
onboardingsvc
notifsvc
docsvc
Oct 24
12
5
3
Oct 31
31
14
6
Nov 7
52
27
9
Nov 14
74
40
11
Nov 21
96
54
13
Nov 28
117
67
15
Dec 5
—
81
17
Threshold: 100 budget exhausted
Error-budget forecasts, production — 10 of 22 services with traffic 12
Service
Budget (30 d)
Spent
Trend / week
ETA
Band
onboardingsvc
3,088 bad events
52%
+21 pts
Nov 22
Nov 18 – Nov 27
notifsvc
68 bad events
27%
+13 pts
Dec 17
Dec 6 – Jan 2
docsvc
309 bad events
9%
+2.5 pts
not trending
—
execsvc
617 bad events
9%
flat
not trending
—
docproc
118 bad events
0%
flat
not trending
—
enrollsvc
25 bad events
4%
flat
not trending
—
schedsvc
11 bad events
0%
flat
not trending
—
virtual-assistant
10 bad events
0%
flat
not trending
—
dashql
9 bad events
0%
flat
not trending
—
tusker
7 bad events
0%
flat
not trending
—
Ask next: What is driving onboardingsvc's error-budget burn on production this week? · Why is notifsvc's budget burn on production rising since Oct 31? · Show onboardingsvc's 5xx per day on production for the last 14 days
Forecasts are a straight-line fit with a band, not a promise. A single burst or a fix moves the date. "Not trending" means the fit doesn't reach the threshold within 90 days.
F04Reliability scorecardproductionreal dataAsk it live ↗
Give me a reliability scorecard for production services.
Production is healthy apart from onboardingsvc. Its error budget is down to 8.1% after 201 5xx in the last 24 h (verdict: "budget low"). The other 12 services with traffic are healthy, and 10 of them had no bad events at all 1.
Production reliability, last 24 h 1 - onboardingsvc: warn — 8.1% budget left · 201 bad - docsvc: ok — 84.1% left · 5 bad - execsvc: ok — 92.3% left · 5 bad - docproc: ok — 100% · 12,468 calls - enrollsvc: ok — 100% · 1,824 calls - notifsvc: ok — 100% · 1,228 calls - tusker: ok — 100% · 671 calls - dashql: ok — 100% · 449 calls - virtual-assistant: ok — 100% · 413 calls - studiosvc: ok — 100% · 240 calls - schedsvc: ok — 100% · 84 calls - regsvc: ok — 100% · 43 calls - patientsvc: ok — 100% · 8 calls
Error budgets on production, last 24 h, availability SLI 1
Service
Objective
Achieved
Calls
Bad
Budget left
p95
p99
Verdict
onboardingsvc
99%
99.08%
21,883
201
8.1%
86 ms
163 ms
budget low
docsvc
99.9%
99.98%
31,409
5
84.1%
524 ms
1,876 ms
healthy
execsvc
99.5%
99.96%
12,923
5
92.3%
141 ms
552 ms
healthy
docproc
99.9%
100%
12,468
0
100%
980 ms
5,292 ms
healthy
enrollsvc
99.9%
100%
1,824
0
100%
419 ms
490 ms
healthy
notifsvc
99.9%
100%
1,228
0
100%
—
—
healthy
tusker
99.9%
100%
671
0
100%
426 ms
1,911 ms
healthy
dashql
99.9%
100%
449
0
100%
1,955 ms
4,253 ms
healthy
virtual-assistant
99.9%
100%
413
0
100%
5,903 ms
12,625 ms
healthy
studiosvc
99.9%
100%
240
0
100%
956 ms
1,733 ms
healthy
schedsvc
99.5%
100%
84
0
100%
478 ms
580 ms
healthy
regsvc
99.9%
100%
43
0
100%
454 ms
491 ms
healthy
patientsvc
99.9%
100%
8
0
100%
1,600 ms
1,920 ms
healthy
Availability only: no latency targets set 1 — Every objective is derived from the service's own last week, and no service has a latency target. virtual-assistant's 5.9 s p95 and dashql's 2.0 s p95 therefore don't count against any budget.
Ask next: What were onboardingsvc's 201 5xx on production in the last 24 h? · Is virtual-assistant's 5.9 s p95 on production normal for it? · How does master's reliability scorecard compare with production's?
patientsvc (8 calls) and regsvc (43) have too little traffic for 100% to mean much.
AI & LLM spend
G01AI spend, 24 hproductionillustrative dataAsk it live ↗
How much did we spend on AI in the last 24 hours, and which org drove it?
Production spent $118.40 on AI in the last 24 h across 4,812 calls 1. sandboxThermofisher drove the most: $46.10, 39% of the total, mostly Tmf Document Language Detection 1.
Ask next: What did sandboxThermofisher's AI calls cost on production over the last 7 days? · Which operations drove gpt-5.6-terra's spend on production today? · Which AI calls failed on production in the last 24 h?
Cost is a floor: 37 calls had no price in the proxy log. These are LiteLLM's own figures, not the OpenAI or GCP bill.
G02Costliest AI operationsproductionillustrative dataAsk it live ↗
Which AI operations cost the most on production this week?
tools: ai_usage
· components: BarChart, Table
This week's costliest AI operation on production is Data Monitor Review, at $212.40 of $642.80 (33%). virtual-assistant chat ($148.75) and Tmf Document Language Detection ($96.20) come next, and the top three are 71% of the week's spend 1.
AI cost by operation on production, last 7 days — top 8 of 23 1 (usd)
cost
Data Monitor Review
212.4
virtual-assistant chat
148.75
Tmf Document Language Detection
96.2
Work order reminder — draft prose
61.3
rag-agent document Q&A
44.85
Wo Reply Agent
18.6
tilda-weaviate:RagDocs:write-embedding
4.4
Work order — status summary
2.15
Data Monitor Review is the expensive one per call 1 — $0.18 a call, about 5× virtual-assistant chat. Each review sends ~150k tokens to gpt-5.6-terra.
Ask next: Which orgs triggered Data Monitor Review on production this week? · How much of Data Monitor Review's input on production is cached? · What did virtual-assistant chat cost per day on production this week?
Cost is a floor where calls lack a price (52 this week). The remaining 15 operations add up to $54.15.
G03OpenAI vs LiteLLMproductionreal dataAsk it live ↗
Why don't OpenAI's bill and LiteLLM's log match this month?
For production the two match. OpenAI billed $630.75 for Nov 1–6 and LiteLLM logged $630.78, a $0.03 difference, and no day is off by more than $0.15 2. The gap you're seeing is most likely scope. OpenAI's org-wide bill for the same days is $1,245.14, and the other $614.39 belongs to other OpenAI projects (eval, automation, master, nexus, test27, delta, test26), which production's proxy log doesn't cover 1.
OpenAI bill vs LiteLLM log per UTC day, Nov 1–6 12 (usd)
OpenAI, whole org
OpenAI, production
LiteLLM, production
Nov 1
21.5
5.66
5.66
Nov 2
169.61
110.16
110.16
Nov 3
229.43
164.16
164.04
Nov 4
242.93
106.6
106.75
Nov 5
389.64
149.91
149.91
Nov 6
192.03
94.26
94.26
✓ Production reconciles to within cents 2 — Requests match model by model: gpt-5.6-terra 10,350 vs 10,350. 8 LiteLLM calls carried no cost (7 on gpt-5.4, 1 on gpt-5.2).
By OpenAI project OpenAI's bill by project, Nov 1–6 1
OpenAI project
Billed Nov 1–6
In production's LiteLLM log?
production
$630.75
yes: logged $630.78
eval
$197.49
no
automation
$157.20
no
master
$90.58
no
nexus
$69.39
no
test27
$56.44
no
delta
$25.76
no
test26
$17.52
no
Production by model production, OpenAI vs LiteLLM by model, Nov 1–6 2
Model
OpenAI billed
LiteLLM logged
Billed − logged
OpenAI requests
LiteLLM calls
gpt-5.6-terra
$448.29
$448.29
$0.00
10,350
10,350
gpt-5.2
$109.67
$109.70
−$0.03
9,578
9,581
gpt-5.4
$68.34
$68.34
$0.00
3,373
3,380
text-embedding-3-small
$4.44
$4.44
$0.00
0
11,656
Ask next: Reconcile OpenAI's automation project with the automation LiteLLM log for Nov 1–6 · Reconcile OpenAI's master project with master's LiteLLM log for Nov 1–6 · Which production operations drove gpt-5.6-terra's $448 in Nov 1–6?
Nov 6 may still be filling in, since OpenAI's copy refreshes about every 12 h 1. I didn't reconcile the other projects, so whether eval, automation or master match their own logs is still open.
Barely. 1 of 502 LLM calls on master failed in the last 6 h: an anthropic/claude-opus-5-5 call from X-ray's xray_optimizer, which ended in *Server disconnected* 12. No other model or operation had a failure 2.
LLM calls on master by status, last 6 h 2 - success: 501 - failure: 1
litellm.InternalServerError: AnthropicException - Server disconnected. Handle with `litellm.InternalServerError`.
Ask next: Are LLM calls failing on production in the last 6 h? · Show xray_optimizer's calls on master in the last 24 h · What did the Test agent run operation cost on master today?
One disconnect from the provider looks transient, not a pattern. A wider window would show whether it repeats.
GCP cost
H01Where the money goesproductionreal dataAsk it live ↗
November's net GCP spend is $2,086.71 for Nov 1–5, on course for about $13,527 (range $12,571–$14,484) against October's $12,337.77. That's 12% ahead of the same point last month 1. Kubernetes Engine is 40% of it ($828.41), followed by Compute Engine ($318.97) and Pub/Sub ($138.85, up $39.75 on the previous five days) 2.
Net by project, Nov 1–5 1 - tilda-trial-production: 678.76 - tilda-trial-dev: 602.32 - tilda-trial-automation: 458.51 - tilda-control-plane: 144.8 - tilda-dev-325721: 120.14 - other: 82.18
Net by GKE cluster, Nov 1–5 1 - not cluster-attributed: 972.49 - production: 279.54 - shared-neo4j-5-26: 237.8 - master: 108.26 - automation: 102.28 - test27: 101.37 - other clusters: 284.97
Net by service, Nov 1–5 vs Oct 27–31 — top 8 of 32 2 (usd)
Nov 1–5
Oct 27–31
Kubernetes Engine
828.41
935.31
Compute Engine
318.97
288.22
Cloud Pub/Sub
138.85
99.1
Vertex AI
121.92
113.22
Artifact Registry
117.35
90.32
Cloud Monitoring
83.19
125.21
Anthos Service Mesh
67.76
86.72
Security Command Center
66.68
68.34
Average net per day by window, UTC (derived from window totals) 1 (usd)
net per day
Oct 9–22
383
Oct 23–29
451
Oct 30–Nov 5
418
Threshold: 414.61 run rate $415/day
Ask next: What is Kubernetes Engine costing per cluster this month? · Why did Cloud Pub/Sub rise $39.75 in Nov 1–5 vs Oct 27–31? · What does the production cluster cost per namespace this month?
The figures are complete through Nov 5, and the export lags about 4 h. The projection is a Theil–Sen estimate, so it is derived, not billed. 47% of spend is project-level, so it can't be attributed to a cluster.
H02Biggest cost spikeproductionreal dataAsk it live ↗
Explain the biggest cost spike in the last 14 days.
The biggest spike was on Oct 27: Cloud Translation on tilda-trial-production billed $188.23 against a normal $0.14 a day 1. That was 93.7% of the project's $200.66 jump that day. Almost all of it was the Translation V3 Document Translation SKU, and almost all of it landed in the 13:00 UTC hour 2. It wasn't a one-off: Translate also spiked on Oct 26 ($31.36) and Nov 2 ($46.72) 1.
Cost anomalies, last 14 days (UTC days) 1 - Oct 26 — Translate $31.36 (normal $0.14): tilda-trial-production, project-level · z 151 - Oct 27 — Translate $188.23 (normal $0.14): tilda-trial-production · z 911, the biggest - Oct 27 — Vertex AI $28.40 (normal $1.65): tilda-trial-dev · z 12 - Nov 2 — Translate $46.72 (normal $0.08): tilda-trial-production · z 393
What moved on Oct 27, tilda-trial-production vs the previous 7-day median 2 (usd)
change vs normal
Translate
188.04
Vertex AI
6.68
Cloud Vision API
4.7
Cloud Pub/Sub
1.64
Kubernetes Engine
1.12
Compute Engine
-1.57
tilda-trial-production net per hour, Oct 27 (UTC) 2 (usd)
net
00:00
4.4
01:00
4.48
02:00
4.3
03:00
4.24
04:00
4.63
05:00
4.33
06:00
6.34
07:00
4.69
08:00
4.54
09:00
4.64
10:00
4.65
11:00
7.1
12:00
7.06
13:00
192.78
14:00
6.08
15:00
6.57
16:00
8.83
17:00
7.01
18:00
4.81
19:00
5.35
20:00
4.7
21:00
4.7
22:00
4.72
23:00
4.39
Neural Translation Model Online Prediction In Translation V3 Document Translation.
⚠ The bill can't name the caller 2 — The Translate cost is project-level, with no cluster, namespace or workload. On the same day, production's API calls ran 1.35× normal and Neo4j queries 2.4×, and rag-agent.upload and exportsvc.activity_update were failing. Those signals are adjacent, not shown to be the cause.
Ask next: Which production service calls Cloud Translation document translation? · Explain the Nov 2 Translate spike on tilda-trial-production · What has Translate cost per day on tilda-trial-production over the last 30 days?
The day's total for the project was $315.32 against a $114.66 median 2. This kind of usage-billed API spike comes from whoever submitted the documents, and nothing in the bill identifies them.
H03Where could we saveproductionreal dataAsk it live ↗
The 10 cost moves listed add up to about $257 a month, and the best are AI caching and Kubernetes requests, not commitments 1. Turn on Gemini context caching for gemini-3.8-flash ($70.12/mo, an X-ray estimate) and gemini-3.0 ($38.54/mo), check x-ray's growth on master ($51.74/mo), and right-size two over-requested deployments ($39.00 and $31.83/mo, from Google) 1. Commitments leave nothing to reclaim: $6.07 of $4,272.77 went unused in 30 days 2.
Cost moves ranked by monthly saving — 10 cost items of 23 open recommendations 1
#
Move
Source
Saving / month
Evidence
1
Context caching for gemini-3.8-flash
X-ray (estimate)
$70.12
$277 of input in 30 d, $0 cached 1; $323.49 billed 3
Gemini input served from cache, last 30 d 3 - gemini-3.8-flash: 0% of 100 ($323.49 billed)
gemini-3.0: 0% of 100 ($234.52 billed)
Order to do them in - [ ] Enable context caching on gemini-3.8-flash and gemini-3.0 — Find the operations that resend the same system prompt or documents. About $109/mo combined (estimates) 1 - [ ] Look at x-ray on master — X-ray ($51.74) and Google ($31.83) both flag it, so the two savings overlap; don't add them. Compare requests with use on x-ray pods - [ ] Right-size argocd-repo-server on control-center-dev — Google's CPU-request recommendation, $39.00/mo 1 - [ ] Snapshot and delete the 5 idle disks — $25.55/mo together. Confirm nothing mounts them first 1
⚠ Google's list is partial 1 — Cloud Run cost recommendations were denied (403) on tilda-trial-production, tilda-trial-dev and tilda-control-plane, one recommender failed on tilda-trial-automation (invalid location), and commitment recommendations were denied at the billing account.
Ask next: Which operations resend the same context to gemini-3.8-flash on production? · Why did x-ray's cost on master grow 151% in 14 days? · What does Translate cost per day on tilda-trial-production?
X-ray's AI-caching savings are estimates, while Google's figures are Google's own. The $257 counts x-ray on master twice. The list shows 12 of 23 open recommendations, so more may exist, and two of the 12 (PodDisruptionBudgets on master and automation) are reliability items with no dollar figure.
H04Cost per cluster per dayproductionreal dataAsk it live ↗
What does each GKE cluster cost per day over the last 14 days?
Over the 14 days Oct 23 – Nov 5, production is the most expensive GKE cluster at $795.28 net, about $56.81 a day1, followed by shared-neo4j-5-26 at $684.68 (~$48.91 a day) 1. Production's own daily bill ran between $52.24 and $64.09 2; the biggest riser is test27, up $122.31 on the previous 14 days 1.
Per-day figures for clusters other than production are 14-day averages — Only production's day-by-day series was read 2; the others need one cost_search per cluster. 46.7% of the bill has no cluster at all (project-level logging, networking, storage) 1.
Ask next: Why did test27's cost rise $122 over the previous 14 days? · Break production's cost down by namespace for the last 14 days · What is in the $2,843.78 that is not attributed to any cluster?
Cost is complete through Nov 5 (UTC usage days, net of credits); the export lags ~4 h 1. The per-day bars are derived (net ÷ 14), not read day by day.
H05What does Neo4j costproductionreal dataAsk it live ↗
Everything named Neo4j cost $2,084.76 over the last 30 days (Oct 7 – Nov 5), 17.1% of the $12,182.11 bill 1. Most of it is the shared-neo4j-5-26 GKE cluster ($1,402.03); the daily run-rate stepped up from about $65–68 to $72–74 from Oct 23 1.
⚠ Don't add these rows up — The dimensions overlap (the neo4j-system namespace is inside the cluster), and the per-resource ledger ($1,103.86) is already inside the $2,084.76 — each row of the bill is counted once 1.
Ask next: Break shared-neo4j-5-26's cost down by node pool for the last 30 days · What changed on shared-neo4j-5-26 around Oct 23, when its daily cost rose? · What do the logs-raw-neo4j Pub/Sub topic and subscriptions cost per day?
Complete through Nov 5 1; the daily line stops at Oct 31 because Nov 1–5 are on the result's second page, which I did not read. Named resources are kept 35 days, and this is a word match — Neo4j cost billed under rows that don't say "neo4j" is not in it.
Yes. Over the last 30 days (Oct 7 – Nov 5) the three commitments were 99.9% used: $4,266.70 of $4,272.77 in fees funded real usage and only $6.07 went unused 1. They also put $3,576.94 of usage below list price, most of it on Kubernetes Engine 1.
✓ Commitments are fully used: $6.07 unused in 30 days 1 — The GKE CUD funded every dollar of its fee; Compute Flexible left $5.10 and Cloud SQL $0.97 unused.
Utilisation, all commitments: 99.86% of 100 ($4,266.70 of $4,272.77 funded usage 1)
Ask next: Is there steady GKE usage on production above what the CUD covers? · What are Google's cost recommendations for tilda-trial-production? · What did Kubernetes Engine cost at list price vs what we paid over the last 30 days?
Complete through Nov 5 1. Full use says the commitments are not too big; whether a larger one would pay off is not in this result — cost_recommendations would show it.
H07Cost per customer orgproductionreal dataAsk it live ↗
Over the last 30 days production's directly attributed $1,637.00 splits most heavily to thermofisher ($273.79), then calls with no org ($257.29) and perceiveBio ($172.99) 1. These are estimates: the cost is apportioned by each org's per-day share of 936,913 API calls, not billed per customer 1.
⚠ Apportioned, not billed 1 — The bill knows workloads, not customers. Production's $1,637.00 direct net cost is split by each org's share of API calls per day; project-level shared costs (logging, networking, storage) are not included.
Production direct cost: $1,637.00 (Oct 7 – Nov 5 1)
Apportioned cost by org, production, last 30 days (top 12 of 30) 1 (usd)
apportioned cost
thermofisher
273.79
(no org)
257.29
perceiveBio
172.99
sandboxIconPlc
130.61
ollin
124.44
mintraHealth
104.63
sandboxThermofisher
72.25
tildainc
66.59
surrozen
56.3
aiTeammatesDemo
46.18
uPennSANDBOXByDT
40.68
sandboxOllin
30.88
Same cost by product area 1 - TMF: 704.26 - Account & Org Settings: 424.29 - EDC: 160.6 - Studies: 142.02 - Home: 79.82 - Studio: 45.5 - 15 other areas: 80.5
Cost per org, production, Oct 7 – Nov 5 (25 of 30 orgs read) 1
Org
API calls
Share of calls
Apportioned cost
thermofisher
172,722
18.4%
$273.79
(no org)
184,201
19.7%
$257.29
perceiveBio
81,688
8.7%
$172.99
sandboxIconPlc
67,043
7.2%
$130.61
ollin
95,198
10.2%
$124.44
mintraHealth
56,688
6.1%
$104.63
sandboxThermofisher
42,116
4.5%
$72.25
tildainc
32,392
3.5%
$66.59
surrozen
38,825
4.1%
$56.30
aiTeammatesDemo
8,372
0.9%
$46.18
uPennSANDBOXByDT
33,828
3.6%
$40.68
sandboxOllin
14,447
1.5%
$30.88
inlandPsychiatricMedicalGroupInc
14,863
1.6%
$27.33
retinaMaculaInstituteOfArizona
5,107
0.6%
$24.63
uPenn
4,468
0.5%
$23.91
ophthalmicConsultantsOfBoston
5,171
0.6%
$22.15
jade
3,991
0.4%
$22.14
sandboxDemoOrg
16,302
1.7%
$20.22
vedaAawc
3,870
0.4%
$13.35
tildaScale
5,936
0.6%
$12.17
1SouthernCollegeOfOptometry
5,847
0.6%
$11.13
sandboxIuvo
5,446
0.6%
$8.97
vedaCrcsi
5,098
0.5%
$8.94
vedaAaecq
3,242
0.3%
$5.88
sandboxEmerald
3,324
0.4%
$5.68
Ask next: What does thermofisher cost on production per day over the last 30 days? · Which services do the (no org) calls on production go to? · What does each org cost on delta over the last 30 days?
Because the split is per day, an org's cost isn't simply proportional to its 30-day call count (ollin has more calls than perceiveBio but a smaller share) 1. Five small orgs on the result's second page were not read.
Users & journeys
I01Logins and lockoutsproductionreal dataAsk it live ↗
No lockouts, and nothing that looks like an attack. In the last 6 h (about 01:00–06:55 UTC) Auth0 recorded 2 failed logins out of 11, and no event had a block or lockout outcome 1. One failure was the [email protected] service login, which succeeded in the same second; the other was a "couldn't find your session" error a minute after akhil.mekala logged in successfully 2.
✓ No lockouts or blocks in the last 6 h 1 — 6 of 44 tenant events failed: 2 logins and 4 refresh-token exchanges. Every user who failed also logged in successfully.
Ask next: Show every Auth0 event from 35.226.74.193 in the last 6 hours · Any login failures or lockouts on the master tenant in the last 6 hours? · What did [email protected] do on production after logging in at 04:30?
"Today" here is the last 6 h; I kept the read small, so 00:00–00:55 UTC was not read. 17 of the 44 events are machine-to-machine tokens (Tilda Sense Bot, Vanta), all successful 1.
I02One user's sessionproductionreal dataAsk it live ↗
What did the most active user on production do in their last session?
The most active user on production in the last 6 h was [email protected] (tildainc): 33 sessions and 1,001 requests 1. His last full working session (01:10–01:16 UTC, 120 requests) went sign-in → Home → Studies → Study › Files, where he spent 262 s and hit 12 failed requests, 6 of them 500s on GET /api/myprofile/visibility/study/:id/blindedScopes2.
[email protected] on production, Nov 7 (UTC) 2 - 01:10:17 — Signed in with SSO: POST /register/discover-method · onboardingsvc - 01:10:30 — Home: app ready in 4.9 s, main content at 9.0 s · 18 requests - 01:10:42 — Pressed Studies: 7 requests, slowest 3,256 ms - 01:10:46 — Opened Study › Files: study 4a5c6740-6896-4c3d-ab01-7d52a2f4485f · 262 s, 76 requests - 01:10:49 — GET /api/documents took 4.0 s: 5 times on this screen - 01:11:12 — PUT storage.upload took 6.1 s: 4 uploads - 01:11:19 — GET …/blindedScopes failed with 500: 6 times, after pressing a grid cell - 01:16:20 — Session ends: Files › Collections was the last area visited
Ask next: Why did trace ef8d7018f29718b0e3063cdd3fcbfd8a return 500 on production? · Is blindedScopes failing for other users on production in the last 6 hours? · Why did GET /api/documents take 4 s for [email protected] on production?
"Most active" is by requests; by session count [email protected] leads with 38 1. Harish's very last session (01:49 UTC) was a 2-second view of one TMF collection, so I showed the last one with real work in it 1. The Neo4j log was not joined, so Cypher per call is unknown 2.
Only one org had sessions on production this morning: tildainc, with 124 of 129 sessions in the last 6 h, and every one of its users is @tilda.bio 2. Production is quiet overall, with 4,510 requests from 8 users in 4 orgs over 7 h 1.
No customer org opened a session in the last 6 h 2 — 124 of 129 sessions were tildainc, Tilda's own staff; the other 5 were signed out. Overview counts 4 orgs over 7 h 1; the other 3 had no sessions in the 6 h journeys read.
Ask next: Which orgs made the most API calls on production over the last 30 days? · Why is execsvc failing 98% of its 50 calls on production? · What did [email protected] do on Study › Files on production?
"Today" is 00:00–06:55 UTC on a Saturday; overview covers 7 h and journeys 6 h, and overview has a 10-minute coverage gap at 06:45–06:55 1. Per-org request counts are not in overview, so the 3 orgs without sessions are not named.
Security & perimeter
J01Security findingsproductionreal dataAsk it live ↗
What are the open Security Command Center findings in production? What should we fix first?
tilda-trial-production has 6,907 active, unmuted Security Command Center findings: 460 critical and 4,077 high 1. Fix first the critical CVEs in images running on the production cluster that already have a fixed version: regsvc's openssl, rag-agent-worker's pillow, and X-ray's own x-ray-weaviate-stats CronJob (Go 1.22.6). All three score CVSS 9.8 1.
What to fix first - [ ] Rebuild regsvc and rag-agent on patched packages — openssl ≥ 3.5.8-r0 and pillow 13; both images are v2.10.43, running on production 1 - [ ] Rebuild X-ray's weaviate-stats image on Go ≥ 1.25.9 — x-ray:3.7.1-5aab6ea ships Go 1.22.6 1 - [ ] Roll delta forward from v2.10.38-beta — its docsvc (Go) and rag-agent-worker (curl) carry critical CVEs with fixes available 1 - [ ] Triage the 2,116 RBAC privilege-escalation threats — find the principal making the RBAC changes; if it is the deploy pipeline, mute by rule rather than one by one - [ ] Close the one-offs — PUBLIC_BUCKET_ACL (1), USER_MANAGED_SERVICE_ACCOUNT_KEY (3), SERVICE_ACCOUNT_KEY_NOT_ROTATED (3) 1
Ask next: List every CRITICAL OS_VULNERABILITY finding on the production cluster · Which principal made the RBAC changes behind the 2,116 privilege-escalation findings? · Which bucket has the PUBLIC_BUCKET_ACL finding in tilda-trial-production?
Counts are exact; only 50 findings were listed, so the table shows 5 of the 460 critical ones 1. I did not count findings per image, so how many a single rebuild clears is not known.
J02What is exposedproductionillustrative dataAsk it live ↗
In tilda-trial-production, 3 firewall rules accept traffic from 0.0.0.0/0 and 2 external load balancers face the internet 1. The one to fix is default-allow-ssh: tcp:22 open to the whole internet on the default network 1. Both GKE control planes (production, delta) have public endpoints, limited by authorized networks 1.
✗ SSH is open to the internet on the default network 1 — default-allow-ssh allows tcp:22 from 0.0.0.0/0 to every instance on default; no load balancer or GKE node pool in this read uses that network.
Rules open to 0.0.0.0/0: 3 (of 41 ingress rules 1)
External load balancers: 2 (ingress-nginx on production and delta 1)
Public GKE endpoints: 2 (authorized networks on 1)
GKE control planes 1 - production: warn — public endpoint · 3 authorized CIDRs - delta: warn — public endpoint · 3 authorized CIDRs
Ask next: Who created default-allow-ssh in tilda-trial-production, and does anything run on the default network? · What did Cloudflare block in front of production.tilda.bio today? · What is open to the internet in tilda-trial-dev?
Logging is off on two of the three open rules 1, so hits on them can't be counted from GCP. The HTTP side is behind Cloudflare, whose firewall log shows what it blocked.
J03Is anyone attacking usproductionreal dataAsk it live ↗
Is anyone attacking us? Show today's Cloudflare firewall events.
Yes, but it is routine automated probing, and Cloudflare blocked all of it: 7,383 firewall events in the last 24 h (Nov 6 07:00 – Nov 7 07:00 UTC), every one a block1234. Most were Bot Fight Mode (4,608) and the managed rule for the React RCE CVE-2025-55182 (2,187). That rule's events peaked at 2,056 between 07:00 and 13:00 on Nov 6 1.
Firewall events: 7,383 (last 24 h, by 6-hour block 1234)
Scanning, not a targeted attack 5 — The newest events are bursts of 3–4 POST / requests from a handful of cloud IPs, aimed at every hostname alike (tilda.pizza, tildasense.ai, production.tilda.bio). That spread looks like an automated CVE scanner.
Ask next: Which IPs and countries triggered Bot Fight Mode in the last 24 hours? · Did any request from 34.116.77.178 reach automation.nexus.tilda.pizza today? · How many threats did Cloudflare count per day this week?
This log holds only what Cloudflare blocked or challenged; requests it let through are in the traffic totals, not here. The event sample is capped at the newest 30 5.
J04GCP inventoryproductionillustrative dataAsk it live ↗
How many buckets, VMs and service accounts do we have in production?
tools: gcp_assets
· components: BarChart, Table
tilda-trial-production holds 31 buckets, 27 VM instances and 44 service accounts1. All 27 VMs are GKE nodes of the production and delta clusters; there are no standalone VMs 2.
Ask next: List the 31 buckets in tilda-trial-production with their location and public access · Which of the 44 service accounts in tilda-trial-production have user-managed keys? · How many buckets, VMs and service accounts are in tilda-trial-dev?
Counts are complete (412 assets, under the 5,000 cap) 1. A count is not a cost: what the buckets cost comes from cost_resources.
J05Edge traffic and blocksproductionreal dataAsk it live ↗
How much traffic did Cloudflare serve this week, and what share was blocked?
Cloudflare served 12.8 million requests (83.3 GB) in the 7 days Oct 31 – Nov 6, and counted 123,542 of them as threats, 0.97%1. Nov 6 stands out: 66,935 threats, 3.9% of that day's requests and about 7× the average of the other six days 1.
⚠ Nov 6: 66,935 threats, about 7× the rest of the week 1 — The other six days averaged about 9,400. Requests that day were not unusual (1.7 M), so it was a burst of hostile traffic, not a traffic surge.
Ask next: What were the 66,935 threats on Nov 6: which rules, IPs and hosts? · Which zone served the most requests this week: tilda.pizza, tilda.bio or tildasense.ai? · Show Cloudflare requests and threats per day for the last 30 days
Nov 7 is partial (454,929 requests so far) and left out 1. "Threats" is Cloudflare's own count of blocked or challenged requests; the per-rule breakdown is in the firewall event log, which I didn't read here.
Of the 25 most recently updated open ENG issues, 11 have no assignee, all of them [Coverage] tickets from Test Agent Automation 1. Among assigned issues, Monika Madan has the most: 3, all Urgent ICON QC bugs in Testing 1. Linear returned more than 100 open issues, so treat these counts as a floor 1.
Platform team read as Linear team ENG; 25 of 100+ open issues counted 1 — Open = started + unstarted. Linear returned 100 and said there are more; this view counts the 25 most recently updated. Name a project or label to narrow it.
Open issues counted: 25 of 100+ (most recently updated 1)
When we clear the answer data from a question and mark it as Not Done by providing a reason, the "Marked as Not Done" text is not displayed on the question.
Ask next: Show every open ENG issue assigned to Monika Madan · List every open ENG issue labelled Release Blocker or Prod Blocker · Which open ENG issues in Zero Tolerance Document QC - Q4 are still In Development?
The other 75+ open ENG issues were not counted, so the assignee ranking can change once all of them are read 1.
K02File a ticket (approval)masterreal dataAsk it live ↗
File a Linear ticket for the Cypher failures on the shared database master.
I've drafted the ticket, and it is waiting for your approval. Nothing has been filed yet 2. The evidence: the shared Neo4j database master has been failing Cypher queries since Oct 27 15:34 UTC, with 224 failures in 15 min at the last check (finding #7603) and no cause clear from logs or recent changes 1.
Action #318: File a Linear ticket: Cypher failures on shared Neo4j database master — approve or reject it on its card.
⚠ Not filed yet: waiting for your approval 2 — Open the gate card to approve or reject this exact draft. It is filed only if you approve.
224 Cypher queries failed in 15 min (whats_broken, Nov 7 06:38 UTC). First seen Oct 27 15:34 UTC. No deploy linked. First look: cause not clear from logs and recent changes. Leads: [email protected], [email protected].
Shared Neo4j failures seen from master (UTC) 1 - Oct 27 15:34 — Cypher failures start on database master: finding #7603 · no deploy linked - Nov 4 14:36 — Failures start on database matrix_a too: finding #16175 · not in this ticket - Nov 6 08:47 — Current burst on master begins: first look: 53 failed in 15 min, no linked change - Nov 7 06:38 — Still failing: 224 in 15 min: latest whats_broken check
The draft covers database master only. matrix_a (#16175, failing since Nov 4) is a separate finding I left out; tell me if it belongs in the same ticket. The root cause is not determined 1.
K03Raise a PR (approval)masterillustrative dataAsk it live ↗
Raise a PR that makes the docs sweep log which repos it skipped.
I've drafted the PR in tildabio/tools, and it is waiting for your approval; no PR exists until you approve it 3. Today sweep() in obs_dashboard/docsweep.py adds every skipped pull request to one total. The draft also counts them per repo and reason, logs one line per repo at the end of a pass, and keeps the per-repo counts in the run's stats 12.
Action #319: Open PR in tildabio/tools: docs sweep logs skipped pull requests per repo — approve or reject it on its card.
⚠ Not opened yet: waiting for your approval 3 — The gate card shows the full diff; the PR is opened only if you approve it.
+from collections import Counter
...
+ skipped_by: dict[str, Counter] = {repo: Counter() for repo, _ in repos()}
for repo, base in repos():
for pr in merged(repo, base, since, until):
...
if why:
stats["skipped"] += 1
+ skipped_by[repo][why] += 1
_record(conn, pr, "skipped", why, 0, slot, dry_run)
continue
...
if not shown.strip():
stats["skipped"] += 1
+ skipped_by[repo]["no product code"] += 1
...
+ for repo, why in skipped_by.items():
+ log(f"[docs-sweep] {repo}: skipped {sum(why.values())} ({', '.join(f'{k} {n}' for k, n in why.most_common()) or 'none'})")
+ stats["skipped_by_repo"] = {r: dict(w) for r, w in skipped_by.items()}
What the PR does, and how to check it - [x] Found where sweep() skips a pull request — two places: skip_reason() (bot, no product code) and an empty product diff 1 - [x] Edited obs_dashboard/docsweep.py — Counter import, a per-repo tally in both skip branches, one log line per repo, stats["skipped_by_repo"] 2 - [x] Added a test in tests/test_docsweep.py — a bot PR and a docs-only PR in two repos give two log lines with the right counts 2 - [ ] Approve or reject the draft on the gate card — action 319 3 - [ ] After merge, read the next pass's log — expect one [docs-sweep] <repo>: skipped N (…) line per repo in docs_sweep_repos
A repo with nothing skipped still gets a line ("skipped 0 (none)"), so a quiet repo is easy to tell apart from one that was never read. Say if you'd rather log only repos with skips.
K04Is it in production yetproductionillustrative dataAsk it live ↗
Yes, ENG-4826 is in production. Its fix (PR #16655, commit 4be91c2) shipped in v2.10.43, which production has run since Oct 30 17:42 UTC1. delta is still on v2.10.38-beta and does not have it 1.
✓ ENG-4826 is in production 1 — PR #16655 was merged, is in release v2.10.43, and has been running on production since Oct 30 17:42 UTC.
Does each cluster run it? 1 - delta: bad — v2.10.38-beta-dafae8a: does not include 4be91c2 - nexus: unknown — no running image recorded - production: ok — v2.10.43-86c75b5: includes it - test27: ok — v2.10.44-6023bae: includes it - automation: ok — v2.10.44-6023bae: includes it - master: ok — master builds since Oct 22
ENG-4826 from merge to production (UTC) 1 - Oct 21 14:05 — PR #16655 merged into main: commit 4be91c2 · regsvc - Oct 22 09:30 — On master: first master build after the merge - Oct 29 11:12 — Tagged v2.10.43: the release includes 4be91c2 - Oct 30 17:42 — v2.10.43 rolled out to production: regsvc:v2.10.43-86c75b5
Ask next: When will delta get v2.10.43 or later? · What else shipped to production in v2.10.43? · Did regsvc's error rate on production change after the Oct 30 rollout?
"In production" means the commit is in the image production runs; whether the fix behaves as the ticket expects needs a look at regsvc's traffic since Oct 30.
K05What shipped this weekproductionreal dataAsk it live ↗
What shipped to production this week?
tools: change_feed → release_notes
· components: Timeline, List
One product version went to production this week: studiosvc and studioaisvc v2.10.44-6023bae (Nov 2–3), plus a docproc deploy on Nov 6 1. The rest of the 20 changes were infrastructure: platform add-ons and per-service infra overlays on Nov 4 1. No release notes were written this week; the latest approved is Release 2.11, from Oct 23 2.
Release notes this week: 0 (latest approved Oct 23 2)
What changed on production, Nov 2 – Nov 6 (UTC) 1 - Nov 2 15:53 — studioaisvc → v2.10.44-6023bae: [email protected] · ash-nazg f162f90b6 - Nov 3 08:37 — studiosvc → v2.10.44-6023bae: ArgoCD sync - Nov 3 11:05 — metrics-server v0.8.0-gke.23: by hand · system:metrics-server-nanny (GKE) - Nov 3 17:11 — studioaisvc and studiosvc re-synced: [email protected] · ash-nazg 3c385e39f - Nov 3 18:08 — academy infra: [email protected] · ash-nazg 46cc997d5 - Nov 4 06:33 — cert-manager, external-secrets, litellm, chat-agent-postgres: [email protected] · ash-nazg 7982661b5 · 4 apps in 34 min - Nov 4 08:35 — Infra overlays for 7 services, plus asm: dashrsvc, enrollsvc, docsvc, regsvc, studioaisvc, execsvc, virtual-assistant · ash-nazg 6fa9ddbb2 - Nov 6 18:12 — docproc: [email protected] · ash-nazg 66d47f48b · image not recorded
Release notes in X-ray 2 - Release 2.11 — approved v26 by [email protected] · v2.10.0 → main 8a8e91e4c (Oct 23) - Release v2.10.39 — approved v7 by [email protected] · 5 open threads (Oct 23) - Release v2.10 U2 — approved v9 · v2.10.16 → v2.10.31 (Oct 12)
Ask next: What changed in studiosvc between v2.10.43 and v2.10.44? · Did studioaisvc's latency or errors on production change after the Nov 3 deploy? · What did the Nov 6 docproc deploy change in ash-nazg?
Deploys with no image recorded are config syncs from ash-nazg; what changed inside them needs the commit diff 1. I can't tell from the release list whether Release 2.11 (ending at main 8a8e91e4c) already covers v2.10.44 2.
How did the latest eval run score compared with the previous one?
tools: evals
· components: Compare, BarChart
The latest eval run (Quality Control, gpt-5.6-terra on qc_classification_v1_test, Nov 5 19:11 UTC) scored 92.38% (303/330), +0.96 pp over the previous run's 91.42% (299/330) at 18:17 1. That is inside the 91.42–93.08% band of the last seven QC runs, so it is not a real change 1.
Latest accuracy: 92.38% (303 / 330 · Nov 5 19:11 UTC 1)
Quality Control accuracy, last 7 runs, Nov 4 17:48 – Nov 5 19:11 UTC 1 (%)
accuracy
144dpi
93.08
72dpi
91.84
v86
91.63
v86 run2
92.23
8491 rules
92.38
v88
91.42
v89
92.38
All seven runs are [email protected]'s, from a laptop. The best was the Nov 4 144 dpi run at 93.08% (305/330), and the 72 dpi run on the same day scored 91.84% 1.
Ask next: Which 27 documents did the v89 run get wrong on qc_classification_v1_test? · How does claude-haiku-5-5 compare with gemini-3.8-flash on TMF Artifact across datasets? · Why did TMF Metadata on claude-haiku-5-5 drop 2.46 pp on dataset17?
"Previous" means the previous full run of the same category, dataset and model 1. With 330 documents, one document is 0.3 pp, so the +0.96 pp is 4 documents.
Which API tests are failing on master, and since when?
tools: tests
· components: Table, Donut
Only 2 of 808 API tests failed in the latest master run, api-tests-master #588 (Nov 7 00:22 UTC) 1. The one chronic failure is GatedFacilityChangeFlow, which failed in all 12 runs of the last 14 days, so it has been failing since at least #573 on Oct 25 1. EDLCrud also failed in #588, but it fails only about one run in three 1.
api-tests-master pass rate by build, last 14 days 1 (%)
pass rate
#573
98.36
#574
98.36
#575
—
#576
96.37
#577
0.37
#578
55.54
#579
98.63
#580
94.4
#581
94.4
#582
—
#583
—
#584
—
#585
—
#586
—
#587
—
#588
98.64
Threshold: 98.36 14-day median
Failing suites, api-tests-master, last 14 days (9 of 14 shown) 1
Suite
Failed runs
Fail rate
Class
Last failed
Failing since
GatedFacilityChangeFlow
12 of 12
100%
chronic
#588
every run in 14 days (since at least Oct 25)
WorkOrderTemplateCopy
8 of 12
66.7%
flaky
#586
intermittent
GatedTemplatePreview
8 of 12
66.7%
flaky
#585
intermittent
WorkPlanInstance
5 of 12
41.7%
flaky
#581
intermittent
EDLTracking
4 of 12
33.3%
flaky
#585
intermittent
EDLCrud
4 of 12
33.3%
flaky
#588
intermittent
TwoStepUpload
3 of 12
25%
flaky
#581
intermittent
BulkDocumentUpload
3 of 12
25%
flaky
#576
intermittent
WorkOrderLanguage
3 of 12
25%
flaky
#581
intermittent
⚠ Three runs were collapses, not test failures 1 — #575 (Oct 27) ran no tests; #577 (Oct 29) failed 794 of 803 in 2.8 min; #578 (Oct 30) failed 345. They are counted apart from suite failures.
Ask next: Show the failing GatedFacilityChangeFlow test in api-tests-master #588 and its error · Why did api-tests-master #577 collapse on Oct 29? · Which API tests are failing on production (api-tests-production)?
Builds #582–#587 were on a page of the result I did not read, so their points are blank 1. 14 days is the tool's window, so I can't see when GatedFacilityChangeFlow started failing, only that it was before Oct 25.
Yes. Master's Weaviate (the shared cluster on shared-neo4j-5-26) is healthy: 3 of 3 nodes up, 0 restarts, no stalled indexing queues and no app errors in the last 6 h 1. The node to watch is weaviate-1, which holds most of the data: 55% disk free and 9.0 GB of memory, against 4.1 GB and 0.36 GB on the other two 1.
✓ Weaviate for master is healthy 1 — 3 of 3 servers up, no restarts, async indexing queues at 0, no app errors in the last 6 h.
Ask next: Which Weaviate classes and shards sit on weaviate-1 vs weaviate-2? · Is Weaviate healthy on production? · How fast has RagDocs_master grown over the last 7 days?
This Weaviate is shared by master, automation, nexus and three more app clusters, so its load is not master's alone 1. The result has no memory limit, so I can't say how close weaviate-1 is to one.