Xray
1h 6h 24h 48h 3d 5d 7d 14d 30d ‹
Last 1 hour▾
Ends now. Anything up to this cluster's 30-day retention.
Start and end times
Cancel
Interpreted in UTC (change it at the right of this bar). Earliest available: 2026-10-19 10:10:56.
› →|
updated 23h ago
Sample data. This database was generated by scripts/seed_dev_db.py — every service, deploy, person and figure on this page is synthetic. Names are sample.*@seed.invalid and image tags are prefixed SEED-. Nothing here happened.

Neo4j schema test26

Every Go service creates the constraints and indexes for the labels it owns when it boots, by reading the live schema (main #16243). A constraint the data violates does not fail the boot: a standby _idx index stands in and the object reads DEGRADED until the rows are fixed — often by a data migration that needs a person. This page shows what each service found on its latest boot, what is blocking each degraded object, and how the clusters compare. UNEXPECTED objects (in the database, not in the model) are informational — usually an older key waiting on its two-release retirement — and never make a service unhealthy.

Last refreshed never · read every 6 h — only the minutes around each service start
Services healthy
12 / 14
2 carry a degraded, drifted or failed object
Objects not in place
4
4 degraded · 0 drift · 0 failed
Migrations waiting on a person
0
0 failed · 0 blocked · 0 pending
Objects in place
764
7 created on the latest boots · 22 undeclared (informational)

Newest boot 38 h ago. Convergence runs only at boot, so a report is what the service found when it last started — data fixed since then shows on its next boot.

Services 14

One row per owning service, unhealthy first. The bar is every object the service owns, by what convergence found.

presentcreatedretireddegradeddriftfailedunexpected
servicestateobjectsnot in place data migrationsreportedearlier boots
authz unhealthy
76 objects
3 degraded 2 undeclared none 53 days ago
boot · pod 722ls
execsvc unhealthy
194 objects
1 degraded 7 undeclared
8 / 8 applied · counted
8 applied on earlier boots (counted, not listed)
38 h ago
boot · pod 68kvn
advpsvc healthy
45 objects
— none declared 3 days ago
boot · pod 6p8xs
docsvc healthy
82 objects
1 undeclared
5 / 5 applied · counted
5 applied on earlier boots (counted, not listed)
40 h ago
boot · pod gf26g
enrollsvc healthy
32 objects
— none declared 44 h ago
boot · pod dsv64
finsvc healthy
27 objects
— none 13 days ago
boot · pod zrlbf
notifsvc healthy
23 objects
— none declared 7 days ago
boot · pod 8fc6r
onboardingsvc healthy
80 objects
3 undeclared
3 / 3 applied · counted
3 applied on earlier boots (counted, not listed)
44 h ago
boot · pod gwxjq
patientsvc healthy
47 objects
— none declared 3 days ago
boot · pod dmmg7
regsvc healthy
22 objects
— none declared 5 days ago
boot · pod mjwsv
schedsvc healthy
15 objects
— none declared 40 h ago
boot · pod rtctq
studiosvc healthy
98 objects
9 undeclared
8 / 8 applied · counted
8 applied on earlier boots (counted, not listed)
44 h ago
boot · pod tbxft
tusker healthy
18 objects
—
1 / 1 applied · counted
1 applied on earlier boots (counted, not listed)
5 days ago
boot · pod qv7nv
workplansvc healthy
31 objects
—
2 / 2 applied · counted
2 applied on earlier boots (counted, not listed)
7 days ago
boot · pod tqjlm

Objects not in place 6

Each one, with how much of its data violates the key, the migration that would heal it, and whether other clusters have the same problem — degraded on production as well usually means the declared key is wrong, not the data.

User_email_phoneNumber_subject_deltimestamp

DEGRADEDauthz
:User emailphoneNumbersubjectdeltimestamp
9,543 of 17,883 rows have a null key property — 53.36% of the label
standby index "User_email_phoneNumber_subject_deltimestamp_idx" still stands in for the constraint: 9543 rows with a null key property (of 17883), 0 duplicate keys over 0 nodes
Next: 53.36% of the label violates it — the declared key is more likely wrong than the data. Not degraded on any other cluster with a report.

StudyNotification_email_notificationType_studyId_deltimestamp

DEGRADEDexecsvc
:StudyNotification emailnotificationTypestudyIddeltimestamp
1 duplicate key over 2 nodes — 16.67% of the label
standby index "StudyNotification_email_notificationType_studyId_deltimestamp_idx" still stands in for the constraint: 0 rows with a null key property (of 12), 1 duplicate keys over 2 nodes
Next: no migration needing attention targets it; 8 applied one(s) are not listed by the boot log — check whether one of those was meant to fix it. Also on: master · 32.26%

UserPreference_key_userid_deltimestamp

DEGRADEDauthz
:UserPreference keyuseriddeltimestamp
10 duplicate keys over 20 nodes — 3.51% of the label
standby index "UserPreference_key_userid_deltimestamp_idx" still stands in for the constraint: 0 rows with a null key property (of 569), 10 duplicate keys over 20 nodes
Next: no migration is declared to fix it — write one (or change the key), then reboot the service. Not degraded on any other cluster with a report.

Staff_email_deltimestamp

DEGRADEDauthz
:Staff emaildeltimestamp
4 duplicate keys over 8 nodes — 0.1% of the label
standby index "Staff_email_deltimestamp_idx" still stands in for the constraint: 0 rows with a null key property (of 7834), 4 duplicate keys over 8 nodes
Next: no migration is declared to fix it — write one (or change the key), then reboot the service. Not degraded on any other cluster with a report.

constraint_89eb5a0d

UNEXPECTEDauthz
:Site siteKey
present in the database but absent from the data model; not dropped automatically
Next: no migration is declared to fix it — write one (or change the key), then reboot the service. Not degraded on any other cluster with a report.

User_email_phoneNumber_deltimestamp

UNEXPECTEDauthz
:User emailphoneNumberdeltimestamp
present in the database but absent from the data model; not dropped automatically
Next: no migration is declared to fix it — write one (or change the key), then reboot the service. Not degraded on any other cluster with a report.

Created and retired on the latest boots 7

What convergence changed. A created constraint that replaced a standby index is a degraded object that healed — its data was fixed and the boot made the real constraint. Retired objects were declared obsolete by their owner and dropped.

objectchangeservicelabelpropertieswhen
Filter_orgid created enrollsvcFilter orgid 44 h ago
ChartReviewList_studyid created enrollsvcChartReviewList studyid 44 h ago
Report_orgid created enrollsvcReport orgid 44 h ago
ChartReviewKeyword_orgid created enrollsvcChartReviewKeyword orgid 44 h ago
ChartReviewDecision_orgid created enrollsvcChartReviewDecision orgid 44 h ago
ChartReviewDecision_studyid created enrollsvcChartReviewDecision studyid 44 h ago
ChartReviewList_orgid created enrollsvcChartReviewList orgid 44 h ago

Every cluster 7

Objects not in place per service and cluster (green = healthy). A problem on one dev cluster is stale data; the same one on production is the model. Click a cell for that cluster.

servicemasterautomationnexusproductiondeltatest27test26
advpsvc · ✓ · · · ✓ ✓
authz · · · · · · 3
budgetsvc · · · · · ✓ ·
docsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
enrollsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
execsvc 1 ✓ ✓ ✓ ✓ 1 1
finsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
notifsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
onboardingsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
patientsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
regsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
schedsvc ✓ ✓ ✓ ✓ ✓ ✓ ✓
studiosvc ✓ ✓ ✓ ✓ ✓ 2 ✓
tusker ✓ ✓ ✓ ✓ ✓ ✓ ✓
workplansvc ✓ ✓ ✓ ✓ ✓ 3 ✓

Undeclared objects 20

In the database, absent from the data model. Never dropped automatically: each is a pending retirement (declare it in schema_retired.go with its ReplacedBy) or something created outside the model. Not unhealthy.

docsvc — 1
objectlabelproperties
Document_status_archivedDocument statusarchived
execsvc — 7
objectlabelproperties
DocOverlay_docId_name_nodeId_treeId_deltimestampDocOverlay docIdnamenodeIdtreeIddeltimestamp
Question_name_nodeId_treeId_deltimestampQuestion namenodeIdtreeIddeltimestamp
Questionnaire_name_nodeId_treeId_deltimestampQuestionnaire namenodeIdtreeIddeltimestamp
DocumentSet_name_nodeId_treeId_deltimestampDocumentSet namenodeIdtreeIddeltimestamp
ParticipantDocUpload_carePlanId_docUploadId_nodeId_deltimestampParticipantDocUpload carePlanIddocUploadIdnodeIddeltimestamp
Visit_name_nodeId_treeId_deltimestampVisit namenodeIdtreeIddeltimestamp
Activity_name_nodeId_treeId_deltimestampActivity namenodeIdtreeIddeltimestamp
onboardingsvc — 3
objectlabelproperties
constraint_89eb5a0dSite siteKey
MemberLabelGroupScopeIdScopeTypeIndexMemberLabelGroup scopeIdscopeType
MemberLabelScopeIdScopeTypeIndexMemberLabel scopeIdscopeType
studiosvc — 9
objectlabelproperties
FormV2_delpendingFormV2 delpending
DocOverlayV2_orgid_scopeid_state_title_version_deltimestampDocOverlayV2 orgidscopeidstatetitleversiondeltimestamp
ResponseSet_archivedtimestamp_name_orgid_scopeid_smoversion_shared_state_variablename_versionResponseSet archivedtimestampnameorgidscopeidsmoversionsharedstatevariablenameversion
Trigger_state_version_deltimestamp_scope_operation_orgidTrigger stateversiondeltimestampscopeoperationorgid
DocReview_orgid_scopeid_state_title_version_deltimestampDocReview orgidscopeidstatetitleversiondeltimestamp
TaskConfig_orgid_scopeid_state_title_version_deltimestampTaskConfig orgidscopeidstatetitleversiondeltimestamp
DocUpload_orgid_scopeid_state_title_version_deltimestampDocUpload orgidscopeidstatetitleversiondeltimestamp
Statement_orgid_scopeid_state_title_variablename_version_deltimestampStatement orgidscopeidstatetitlevariablenameversiondeltimestamp
TodoV2_state_title_version_deltimestampTodoV2 statetitleversiondeltimestamp