Guides

Observability quietly eats into cloud budgets

Log storage, trace sampling and metric labels affect observability costs. Where teams can take action.

By Benedikt Langer August 3, 2026 4 min read

Observability quietly eats into cloud budgets

Logs, metrics and traces help teams operate distributed cloud applications. Collecting and storing them consumes resources and, depending on the service, can generate usage-based costs. Logs stored multiple times and metrics with many different label values are particularly hard to spot. An inventory of data paths shows where teams can limit the expense.

Key takeaways

Costs depend on the billing model. Data volume, retention and additional storage destinations therefore all need to be factored in together.
Trace sampling requires a deliberate selection rule. Head sampling makes a decision early, while tail sampling allows selection based on further information from the trace.
Metric labels can greatly increase the number of time series. Value ranges that grow with every request or user are particularly problematic.

Where log data generates costs more than once

The first step is to map sources, data flows and storage destinations: which application generates which logs? Where are they stored and which teams use them? This also includes forwarding them to analytics and security systems. Only with this overview can teams assess whether a copy serves a distinct purpose.

The specific pricing model determines the potential savings. With Google Cloud Logging, storage in certain log buckets incurs a volume-based charge. If the same entry is stored in multiple billable buckets, each copy is counted for billing. Longer retention may incur additional charges. Other services and contracts may use different billing models.

Teams should therefore record the volume and cost of each data path and assign responsibility for it. A budget with an alert makes growth visible. Before setting a hard limit, teams need to establish which signals must be retained for operations and security.

Use trace sampling selectively

Sampling reduces the number of stored traces by selecting a subset. With head sampling, the decision is made early. At that point, the full sequence is not yet known. Head sampling alone therefore cannot account for whether an error occurs later in every trace.

Tail sampling in OpenTelemetry can select traces based on information from multiple spans, such as error status or latency. The sampling component must first collect and process the data. This increases operational overhead. An earlier head sampling rule can also discard traces before the later selection takes effect.

This does not establish a fixed sampling rate for all services. A test for each service is useful: which errors and slow processes remain detectable? How much data reaches the backend? The rule is then adjusted based on the results.

Assess retention and labels against their purpose

Every log class needs a documented purpose and an appropriate retention period. Debug data may have different requirements from security or audit logs. Before shortening retention or deleting data, the responsible team checks applicable legal and contractual obligations as well as the needs of incident investigations. An archive may be an option if access and costs are appropriate for the intended purpose.

For metrics, it is worth examining cardinality. According to Prometheus, each unique combination of label names and values creates a separate time series. User identifiers or other values without a fixed upper limit can cause data volumes to grow considerably. Bounded dimensions are usually more suitable for aggregated operational metrics; individual cases can potentially be investigated using logs or traces.

Review responsibility and impact regularly

After the inventory, the team prioritises the largest data paths and establishes a baseline for changes: data volume, costs and the operational purpose of the signals. Sampling and retention are documented and measured again after a change. This includes testing whether alerts and incident queries still provide the information required.

A regular review brings together platform operations, service owners and budget owners. This allows a new data source to be assessed alongside its benefits and ongoing costs.

Frequently Asked Questions

Can sampling retain all error traces?

Head sampling alone cannot ensure this because the decision is made early. Tail sampling can use errors as a selection criterion, but it requires access to the relevant trace data and creates its own operational overhead.

How long should logs be stored?

From a technical perspective, there is no single retention period for all logs. The team should assess the purpose, incident investigation needs and applicable legal and contractual requirements for each data class.

Why can user IDs in metrics become expensive?

Each new label combination can create an additional time series. Many different user IDs therefore increase the data volume and resource requirements of the metrics system.