Metrics Instrumentation Best Practices: 2026 Guide
Introduction: What Metrics Instrumentation Actually Means
Metrics instrumentation happens in two distinct phases that are worth separating clearly.
Instrumentation is the code that takes measurements and associates them with a metric name. You write this code inside your application, your libraries, or your middleware. Collection is when the application configures those named metrics to be transmitted externally for storage and analysis.
Most confusion around metrics instrumentation best practices stems from conflating these two phases. You can instrument perfectly but lose data because collection fails. You can collect everything but instrument poorly, filling your backend with noise. Both phases need deliberate attention.
This glossary covers the core concepts, frameworks, and practical rules that make instrumentation effective. It pays special attention to serverless environments, where short-lived handlers, missing agents, and flush timing constraints break assumptions baked into traditional monitoring guidance. For foundational context on telemetry and MELT, that overview complements what follows here.
If you want to skip straight to recording counters and histograms with no infrastructure setup, Distlang Metrics is an API-first serverless metrics service built for exactly this workflow.
Metric Types: Counter, Gauge, Histogram, Summary
Choosing the correct metric type is the single most impactful instrumentation decision you’ll make. It directly affects how metrics behave over time, how they’re queried, and how accurately they reflect system state.
Counter
A cumulative metric that represents a single monotonically increasing value. It can only go up or reset to zero on restart. Use counters for the number of requests served, tasks completed, or errors occurred.
When to use: Anything that only accumulates. Total HTTP requests. Total bytes sent. Error counts.
Common mistake: Using a counter for a value that can decrease, like active connections or queue depth. That’s a gauge.
In serverless environments: Counters work well, but you must flush before the function freezes. A counter increment that never reaches the backend is a counter increment that never happened.
Gauge
A metric that represents a single value that can go up or down. Gauges are typically used for measured values like current memory usage, temperature, or the number of concurrent requests.
When to use: Current state measurements. Active connections. Queue size. CPU utilization.
Common mistake: Gauges are snapshots. If your scrape interval is 60 seconds and a spike lasts 5 seconds, you’ll miss it entirely. Gauges are great for “what’s happening right now” but unreliable for trends unless sampled consistently.
Histogram
Histograms record the distribution of values across configurable buckets. This lets you compute percentiles like p50, p95, and p99, which is essential for latency measurements where averages hide the worst-case experience.
When to use: Request duration. Response sizes. Anything where the distribution matters more than the average.
Common mistake: Bad bucket boundaries. If your latency ranges from 1ms to 500ms but your buckets are [100, 200, 500, 1000], you’ve made the bottom 100ms of your distribution invisible. Tune buckets to match expected value ranges. For a hands-on walkthrough, see how to instrument counters and histograms in JavaScript.
Summary
Similar to histograms but pre-computes quantiles on the client side. Less commonly used because pre-computed quantiles cannot be aggregated across instances. If you have 10 replicas each reporting a p99, you cannot combine those into a meaningful fleet-wide p99.
When to use: Rarely. Prefer histograms in almost all cases unless you have a specific reason to compute quantiles client-side.
Quick-Reference: Which Metric Type Should I Use?
| If your value… | Use a… | Example |
|---|---|---|
| Only goes up | Counter | Total requests, total errors |
| Can go up or down | Gauge | Active connections, memory usage |
| Needs percentile analysis | Histogram | Request latency, response size |
| Needs client-side quantiles (rare) | Summary | Pre-aggregated latency quantiles |
A useful mnemonic from practitioners on DEV Community: “Counters tell you how much has happened. Gauges tell you what’s happening right now. Histograms tell you how the data is spread out.”
Monitoring Frameworks: RED, USE, and the Four Golden Signals
Knowing your metric types is necessary but not sufficient. You also need a framework for deciding what to measure. Three frameworks dominate the conversation, and each one solves a different problem.
RED Method (Rate, Errors, Duration)
RED measures how a microservice behaves from the caller’s perspective. Tom Wilkie introduced the method in 2015 while at Weaveworks, and it’s become the default starting point for service-level instrumentation.
- Rate: Requests per second
- Errors: Failed requests per second
- Duration: Distribution of response times (use histograms, not averages)
When to use: API gateways, microservices, any request-driven system. RED answers “is my service serving users well?”
Critical note: Duration must be tracked as percentiles. An average response time of 200ms can hide the fact that 5% of your users are waiting 3 seconds. Practitioners on Reddit consistently flag this as one of the most common instrumentation mistakes.
USE Method (Utilization, Saturation, Errors)
The USE Method is resource-centric and designed for infrastructure monitoring: hosts, containers, load balancers, CPU, memory, disk.
- Utilization: How busy is the resource (percentage of capacity)
- Saturation: How much queued or backlogged work exists
- Errors: Resource-level error counts
When to use: Infrastructure health checks. “Is my system capable of handling the load?” rather than “is it handling the load well?”
Four Golden Signals
Google’s SRE practices codified four foundational metrics: Latency, Traffic, Errors, and Saturation. These overlap heavily with RED and USE but combine both perspectives.
Framework Comparison
| Framework | Focus | Best For | Key Question |
|---|---|---|---|
| RED | Request performance | Services, APIs, microservices | “Are users being served well?” |
| USE | Resource performance | Infrastructure, hosts, containers | “Is my infrastructure healthy?” |
| Golden Signals | Both | SRE teams, SLO-driven orgs | “What matters most right now?” |
The USE method asks about internal resource health. RED asks about external user experience. Together they give you a comprehensive view. For a deeper breakdown of metrics types and KPIs and how to map them to developer workflows, that guide fills in the gaps.
Naming and Labeling Conventions
Inconsistent naming is one of the quietest, most expensive mistakes in instrumentation. A search for “latency” misses metrics named response_time, duration, and elapsed_ms. If one service uses requests_total and another uses request_count, you cannot sum them across the fleet.
Rules That Work
Use hierarchical naming. Start at the highest level, then add specificity. For example: http.server.request.duration rather than server_latency_http.
Keep units out of names. If your telemetry backend captures units in metadata (OpenTelemetry does), don’t repeat them. http.server.request.duration is better than http_server_request_duration_ms.
Separate dimensions from metric names. If you have 100 server instances, differentiate them with a label/dimension, not by baking the server name into the metric name. http.requests{server="web-04"} beats http_requests_web04.
Nest related metrics together. OpenTelemetry’s semantic conventions recommend grouping associated metrics in a hierarchy based on their usage. This makes discovery and correlation dramatically easier.
For more granular guidance on label naming and cardinality, that reference covers the specific strategies that prevent naming chaos at scale.
OpenTelemetry Semantic Conventions
OpenTelemetry Semantic Conventions (currently at v1.43.0) define a common set of attributes that provide meaning to data when collecting, producing, and consuming it. They specify metric instrument names, units, and attribute names along with their types and valid values.
The practical benefit: following a shared naming scheme that can be standardized across a codebase, libraries, and platforms. This allows easier correlation across services and teams.
Cardinality: The Silent Budget Killer
Cardinality refers to the number of unique time series your metrics generate. Every unique combination of a metric name and its label values creates a new time series in your backend.
This matters because cardinality explosion happens fast and costs real money. According to Chronosphere, moving from a legacy environment to a cloud-native one can push cardinality from 150,000 unique time series to 150 million. That’s a 1,000x increase.
What Triggers Explosion
The most common anti-patterns from practitioners: stuffing high-cardinality data into metric labels. Request IDs, timestamps, user IDs, full URLs with query parameters, user-provided values. Each unique value spawns a new time series.
How to Prevent It
- Ban unbounded label values. Request IDs and user IDs belong in traces, not metric labels.
- Normalize paths.
/users/123and/users/456should both map to/users/{id}. - Set cardinality caps during SDK initialization. Pre-aggregation (covered below) can enforce limits early.
- Audit regularly. New deploys can introduce high-cardinality labels without anyone noticing until the bill arrives.
Pre-aggregation and Buffering
Pre-aggregation
Pre-aggregation reduces the volume of data transmitted by combining measurements before export. This improves efficiency and makes metrics collection behavior more predictable.
In practice, pre-aggregation lets you apply cardinality limits during SDK initialization. Combined with memory preallocation, this ensures that a runaway label doesn’t blow up your telemetry pipeline at 3 AM.
Buffering and Flush Semantics
Buffering collects metrics in memory and sends them in batches rather than one at a time. This reduces network overhead and is straightforward in long-running processes.
In serverless environments, buffering gets complicated. Functions may exist for only a single request, running for milliseconds before the platform freezes them. If your buffer hasn’t flushed by then, the data is gone.
This is where flush semantics become critical. Cloudflare Workers provide ctx.waitUntil() to keep the runtime alive while asynchronous work (like flushing metrics) completes. Vercel provides after() for the same purpose. Without these patterns, you’ll lose data on virtually every invocation. For the specifics, the guides on Cloudflare Workers waitUntil() semantics and Vercel after() for metrics flush walk through the implementation details.
Distlang Metrics is built around these exact constraints: a lightweight JavaScript client with built-in flush and buffering designed for short-lived handlers, so you don’t have to wire up flush semantics from scratch. You can go from code to dashboard quickly, explore the metrics quickstart to see how.
Instrumentation Anti-Patterns
Knowing what not to do is at least as valuable as knowing what to do. These anti-patterns appear across OpenTelemetry community discussions, practitioner forums, and incident postmortems.
Over-instrumenting
Don’t create a metric for every possible measurement. More metrics means more cost, more noise, and harder debugging. If nobody will ever alert on a metric or use it in a dashboard, it shouldn’t exist.
Wrong metric types
Using a counter for a value that can decrease. Using a gauge when you need distribution analysis. Using a summary when you need to aggregate across instances. Each of these leads to queries that return misleading results.
Missing context
A metric without labels is hard to filter, group, or correlate. Always include enough labels to make the metric useful, things like service name, environment, HTTP method, status code class. But don’t go overboard (see cardinality above).
Averaging duration
This one is pervasive. An average latency of 100ms tells you almost nothing about user experience if 1% of requests take 5 seconds. Always use histograms for duration and look at p95 or p99.
Unbounded label values
User-submitted form values, full stack traces, raw SQL queries. These belong in logs or traces, never in metric labels. One bad deploy that stuffs a request ID into a label can generate millions of time series overnight.
Choosing What to Instrument
The temptation is to instrument everything and filter later. At scale, this approach breaks both your budget and your ability to find signals in noise. Enterprise observability spends routinely reach millions annually, and much of that cost is wasted on metrics nobody looks at.
Start with user impact
Work backward from critical user journeys to the components that support them. Instrument only the metrics that explain or predict changes in user-facing performance. This is where RED shines: rate, errors, and duration for the endpoints your users actually hit.
Align to SLOs
If you have service level objectives, your metrics should directly support them. An SLO of “99.9% of requests complete in under 300ms” tells you exactly which histogram you need and which percentile to track.
Expand iteratively
Start minimal. Run with it for a sprint or two. Add metrics when you find yourself unable to answer a question during an incident. This iterative approach treats telemetry like a budget, and in 2026, the smartest teams do exactly that. Standardize instrumentation, budget the cost, and reject “collect everything forever” as the default.
Serverless Instrumentation Considerations
Traditional observability assumes long-running processes with agents, sidecars, or daemons that continuously scrape and export data. Serverless environments break every one of those assumptions.
What’s different
- Ephemeral execution. Functions spin up, handle a request, and freeze (or terminate). There’s no persistent process to run an agent on.
- No sidecar support. Edge runtimes like Cloudflare Workers don’t allow you to attach a Prometheus exporter or an OpenTelemetry Collector sidecar.
- Cold starts. SDK initialization adds latency. Initialize outside the handler to minimize impact.
- Flush timing. Many telemetry libraries default to batching on intervals (every 30 seconds, say), which is longer than most serverless function lifetimes.
Practical patterns
Use SimpleSpanProcessor (or its metrics equivalent) in serverless environments to ensure data is exported before function freeze. On Cloudflare Workers, wrap your flush call in ctx.waitUntil(). On Vercel, use after(). Both keep the runtime alive long enough for the export to complete.
For a complete walkthrough of serverless instrumentation across platforms, the application instrumentation guide for Cloudflare and Vercel covers the end-to-end setup.
Why this matters for metrics instrumentation best practices
Most published best practices implicitly assume you have a long-running process, a scraping interval, and an agent. If you’re building on serverless or edge runtimes, you need to actively adapt every standard recommendation to account for short lifetimes and forced flushes. Ignoring this leads to the most frustrating class of metrics bug: instrumentation that works perfectly in local dev and loses data silently in production.
Quick Reference: All Key Terms
| Term | One-Line Definition |
|---|---|
| Instrumentation | Code that takes measurements and associates them with metric names |
| Collection | Configuring named metrics to be transmitted for external storage and analysis |
| Counter | Monotonically increasing metric; only goes up or resets to zero |
| Gauge | Point-in-time value that can go up or down |
| Histogram | Distribution of values across buckets; enables percentile calculations |
| Summary | Client-side quantile computation; hard to aggregate across instances |
| RED Method | Rate, Errors, Duration: request-focused monitoring framework |
| USE Method | Utilization, Saturation, Errors: resource-focused monitoring framework |
| Four Golden Signals | Latency, Traffic, Errors, Saturation from Google SRE |
| Cardinality | Number of unique time series generated by metric + label combinations |
| Semantic Conventions | OpenTelemetry’s standardized naming and attribute definitions |
| Pre-aggregation | Combining measurements before export to reduce volume and enforce limits |
| Flush Semantics | Patterns for ensuring data export before serverless function freeze |
| SLO | Service Level Objective: target threshold for a service metric |
Frequently Asked Questions
What is the difference between metrics instrumentation and metrics collection?
Instrumentation is the act of writing code that records measurements (incrementing a counter, observing a histogram value). Collection is the pipeline that exports those recorded measurements to an external system for storage and querying. You can instrument without collecting (metrics go nowhere), and you can have collection infrastructure with no instrumentation (nothing to collect).
How do I choose between a counter and a gauge?
If the value can only increase (total requests, total errors, bytes sent), use a counter. If it can go up and down (active connections, queue depth, memory usage), use a gauge. Using the wrong type means your queries will produce incorrect results.
What causes cardinality explosion and how do I prevent it?
Cardinality explosion happens when metric labels contain unbounded values like user IDs, request IDs, or raw URLs. Each unique label combination creates a new time series. Prevention comes down to label discipline: ban high-cardinality labels, normalize URL paths, and set cardinality caps at the SDK level.
Should I use RED or USE for monitoring?
Use both, but for different things. RED (Rate, Errors, Duration) monitors request-driven services from the user’s perspective. USE (Utilization, Saturation, Errors) monitors infrastructure resources. Most teams need both: RED for their APIs and USE for the systems underneath.
Why do averages hide latency problems?
An average combines all values into a single number, which obscures outliers. If 95% of requests take 50ms and 5% take 3 seconds, the average is about 197ms, a number that describes nobody’s actual experience. Use histograms and look at p95 or p99 instead.
How do serverless environments change metrics instrumentation best practices?
Serverless functions are ephemeral, sometimes running for only milliseconds. There’s no persistent process to run an agent, and default batch export intervals are longer than function lifetimes. You need explicit flush patterns (like waitUntil() on Cloudflare Workers or after() on Vercel) and should initialize SDKs outside the handler to reduce cold start impact.
What are OpenTelemetry semantic conventions and why should I follow them?
Semantic conventions are standardized names, types, and attribute definitions published by the OpenTelemetry project. Following them means your metrics are immediately compatible with any OTel-compatible backend and correlatable with metrics from third-party libraries that follow the same conventions. The alternative is naming chaos across your services.
How do I decide which metrics are worth instrumenting?
Start from user-facing journeys and work backward. Instrument the metrics that explain or predict changes in user experience. Align metrics to your SLOs. If you can’t articulate how a metric would be used during an incident or in a dashboard, it probably doesn’t need to exist. Expand iteratively rather than instrumenting everything upfront.
Ready to put these practices into action without setting up Prometheus, Grafana, or any collection infrastructure? Distlang Metrics gives you an API-first serverless metrics service with built-in flush semantics for Cloudflare Workers and Vercel, so you can go from instrumentation code to a live dashboard in minutes.