Workflow: TSDB FinOps & Optimization
Step-by-step guide for analyzing Prometheus storage costs, identifying cardinality hotspots, and generating optimization configurations.
When to Use
Use this workflow when:
- Prometheus storage costs are growing and you need to identify why
- You suspect high-cardinality metrics are inflating your TSDB
- You need to set up recording rules for pre-aggregating expensive queries
- You want to configure remote-write to Thanos, Mimir, or Cortex
Journey
Step-by-Step
| Step | Action | Tool / Resource | Key Parameters |
|---|---|---|---|
| 1 | Get cardinality overview | Resource: prom://tsdb/cardinality | Total series + top-N metrics |
| 2 | Analyze hotspots | prom_optimize_cardinality(top_n=10) | Returns recommendations with severity |
| 3 | Investigate labels | prom_explore_labels(metric_name="<hot_metric>") | Find high-cardinality dimensions |
| 4 | Generate relabel config | prom_plan_relabel(metric_name="<metric>", labels_to_drop=["pod_id"]) | Returns YAML for metric_relabel_configs |
| 5 | Create recording rule | prom_create_recording_rule(rule_name="job:http_requests:rate5m", rule_expr="sum by (job) (rate(http_requests_total[5m]))") | Returns rule group YAML |
| 6 | Configure remote-write | prom_configure_remote_write(remote_url="http://thanos-receive:19291/api/v1/receive") | Returns remote_write YAML with queue config |
info
All FinOps tools generate configuration YAML — they do not apply changes. Users must manually add the YAML to their Prometheus configuration files.
Natural Language Prompts
"How many active series does our Prometheus have?"
"Analyze cardinality hotspots and recommend optimization strategies."
"Generate a relabel config to drop the pod_id label from http_requests_total."
"Create a recording rule that pre-aggregates request rates by service."
"Configure remote-write to our Thanos receive endpoint."
Next Steps
- Rule Management — Rule authoring, testing, and simulation
- Troubleshooting — Diagnosing failed scrape targets
- App Onboarding — Instrumenting applications