CNCF published a guide to safe self-service access to their GPU metrics for Kubernetes tenants. The design needs no new metrics stack and gives every team safe self-service access to its metrics.

The design puts a proxy in front of the shared Prometheus that identifies the tenant, restricts every query to its namespace, and returns permitted results. The proxy can copy selected metrics into a tenant’s separate Prometheus for its dashboards and alerts.

To restrict queries, the authors use prom-label-proxy, a component that adds a namespace restriction to every query.

Claim check:

  • CNCF published a guide to safe self-service access to their GPU metrics for Kubernetes tenants. (confirmed by the publication itself: evidence; «Whose GPUs are these, anyway? Secure, self-service metrics for multi-tenant Kubernetes»)
  • The design needs no new metrics stack and gives every team safe self-service access to its metrics. (confirmed by the publication itself: evidence; «how we gave every team a safe, self-service view into their own metrics without handing them the keys to everyone else’s. No new metrics stack»)
  • The design puts a proxy in front of the shared Prometheus. (confirmed by the publication itself: evidence; «We needed a thin, tenant-aware layer in front of the one we already had.»)
  • The proxy identifies the tenant, restricts every query to its namespace, and returns permitted results. (confirmed by the publication itself: evidence; «Identify: Authenticate the caller and establish which tenant they are. Isolate: Restrict every query to that tenant’s namespace, enforced below the query language so it can’t be bypassed. Deliver: Optionally copy a curated slice of each tenant’s metrics into their own small Prometheus»)
  • The proxy can copy selected metrics into a tenant’s separate Prometheus for its dashboards and alerts. (confirmed by the publication itself: evidence; «Optionally copy a curated slice of each tenant’s metrics into their own small Prometheus, so their dashboards and alerts run against a store they own.»)
  • prom-label-proxy adds a namespace restriction to every query. (confirmed by the publication itself: evidence; «it rewrites every incoming query to inject a namespace matcher»)

Primary sources:

score 67.5 · kind guide · revision 1 · stories st-1fin977