Skip to content

Add an internal ateapi statusz page - #1596

Open
Adhita Selvaraj (swiftdiaries) wants to merge 6 commits into
agent-substrate:mainfrom
swiftdiaries:codex/bug-bash-1458-statusz-base
Open

Adhita Selvaraj (swiftdiaries) wants to merge 6 commits into
agent-substrate:mainfrom
swiftdiaries:codex/bug-bash-1458-statusz-base

Conversation

@swiftdiaries

@swiftdiaries Adhita Selvaraj (swiftdiaries) commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Description

  • Added a statusz page for the ateapi
  • Used the existing ate-router page as reference

Addresses #1458

Review Notes

Things I'm not sure about:

  • I'm not aggregating metrics across replicas, is this needed?
  • On the metrics themselves, could I please get a review on what's needed/how they're represented?

Screenshot

Screenshot 2026-09-14 at 17 00 42

AI-generated PR description

Adds an internal ateapi /statusz page for inspecting the serving replica. HTML and JSON expose build/version, uptime, live readiness, listener/drain settings, and safely redacted startup configuration, alongside known workers grouped by namespace/pool, PostgreSQL client-pool occupancy, and recent failed authenticated unary Control RPCs.

The dedicated --status-port listener defaults to 4040, can be disabled with a nonpositive value, and stays available through gRPC drain. It is exposed through the internal API Service. Configuration uses a closed display policy to redact credentials, environment-resolved values, and sensitive paths.

Each request reads diagnostics directly from existing in-memory readers; there is no diagnostic response cache or background sampler. The browser offers Manual / 5s / 10s default / 30s full-page refresh, matching the router. The selection resets to 10 seconds on each page load.

Worker counts reflect this replica's cached view of shared inventory and must not be summed across replicas; the page displays at most 100 groups with totals and truncation information. Pool statistics perform no SQL, connection acquisition, or connectivity probe. Failure history retains the newest 100 matching completions with bounded method/principal text, status, and elapsed time, without payloads, credentials, or raw errors. It resets on restart and is not an error-rate metric.

Includes both the endpoint foundation and runtime diagnostics.

Validation: focused ateapi/statusz race tests passed on the combined head a1b96d9f302766f4f744693e06bd442a2c57da3f; diff checks passed. Earlier validation covered PostgreSQL-backed pool tests, browser refresh behavior, GolangCI-Lint, kube-api-linter, and applicable static checks. The full make verify run encountered inherited color/telemetry environment failures; those affected packages passed when rerun without the conflicting variables. Some static wrappers required a clean worktree and were not run during the uncommitted refactor. No single full-verification pass, live-cluster deployment, or E2E validation is claimed.

@swiftdiaries Adhita Selvaraj (swiftdiaries) changed the title Add an internal ateapi status page Add an internal ateapi statusz page Sep 10, 2026
@swiftdiaries

Adhita Selvaraj (swiftdiaries) commented Sep 11, 2026

Copy link
Copy Markdown
Contributor Author

cc Da Huang (@git286)

@bowei

Copy link
Copy Markdown
Collaborator

+1 I love statusz pages :-)

Expose build, readiness, listener settings, projected configuration, and bounded worker-pool counts through one status model and embedded dashboard. Keep cache failures distinct from an empty fleet and cover rendering and snapshot behavior with tests.
Add a configurable status port and project resolved flags through a closed display policy before handing them to the dashboard. Share existing readiness and worker-cache state, keep status available through the gRPC drain, and bound HTTP shutdown.
Add the named status port to the deployment and headless API Service. Document replica-local data, redaction, HTML/JSON access, and disabling the listener so operators can inspect an API replica through port-forwarding.
@swiftdiaries
Adhita Selvaraj (swiftdiaries) marked this pull request as ready for review September 14, 2026 11:33
@swiftdiaries

Copy link
Copy Markdown
Contributor Author

Bowei Du (@bowei) haiyanmeng added a statusz page based on the issue description. Generated a screenshot with some synthetic seed data to visualize what it'd look like.

Could I please a get review on the representation of the metrics and if replica-wide aggregation is needed here? Tried to keep it simple for the first pass.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants