Prometheus & Grafana — Vector Matching

Vector Matching

When you perform binary operations between two vectors, Prometheus needs to figure out which time series on the left match which time series on the right. This is called vector matching.

One-to-One Matching (Default)

By default, Prometheus matches series that have exactly the same label set.

# Both sides must have identical labels to match
rate(http_request_duration_seconds_sum[5m])
/
rate(http_request_duration_seconds_count[5m])

If left side has {method="GET", handler="/api", instance="web-1:8080"}, it matches the right side with the exact same labels.

The Problem

# This FAILS if the label sets don't match exactly
metric_a{foo="bar", extra="label"}
/
metric_b{foo="bar"}
# No match! metric_a has "extra" label that metric_b doesn't

on() — Match on Specific Labels

Use on() to specify which labels to match on:

# Match only on "method" and "handler"
rate(http_request_duration_seconds_sum[5m])
/
on(method, handler)
rate(http_request_duration_seconds_count[5m])

This ignores all other labels and matches series that share the same method and handler values.

ignoring() — Ignore Specific Labels

The opposite of on() — match on all labels except the listed ones:

# Ignore the "status" label when matching
rate(http_requests_total{status=~"5.."}[5m])
/
ignoring(status)
rate(http_requests_total[5m])

Many-to-One Matching

What happens when one side has multiple series matching a single series on the other side?

Left side:                          Right side:
  {method="GET", status="200"}  →   {method="GET"}
  {method="GET", status="404"}  →   {method="GET"}  (same!)
  {method="GET", status="500"}  →   {method="GET"}  (same!)

Prometheus won’t allow this by default — it’s ambiguous. You need group_left or group_right.

group_left — Many-to-One (Left Has More)

# Error rate per status, divided by total per method
rate(http_requests_total{status=~"5.."}[5m])
/
on(method) group_left
sum by (method) (rate(http_requests_total[5m]))

Matching:
  Left (many):                              Right (one):
  {method="GET", status="500"}   ──────►   {method="GET"}
  {method="GET", status="502"}   ──────►   {method="GET"}
  {method="POST", status="500"}  ──────►   {method="POST"}

group_left means: “The left side has the higher cardinality (more series per match group). Keep all left-side labels in the result.”

group_right — One-to-Many (Right Has More)

# Less common — right side has more series
sum by (method) (rate(http_requests_total[5m]))
/
on(method) group_right
rate(http_requests_total[5m])

Adding Labels with group_left

You can pull labels from the “one” side to the “many” side:

# Add "version" label from build_info to CPU metrics
rate(node_cpu_seconds_total{mode="idle"}[5m])
* on(instance) group_left(version)
build_info

This is the info metric join pattern — extremely useful for adding metadata labels.

Visual Guide

One-to-One (default):
  Left:  {a="1", b="x"} ──── {a="1", b="x"} :Right
  Left:  {a="2", b="y"} ──── {a="2", b="y"} :Right

on(a) — match only on label "a":
  Left:  {a="1", b="x"} ──── {a="1", b="z"} :Right
  Left:  {a="2", b="y"} ──── {a="2", b="w"} :Right

on(a) group_left — many-to-one:
  Left:  {a="1", b="x"} ──┐
  Left:  {a="1", b="y"} ──┤── {a="1"} :Right
  Left:  {a="1", b="z"} ──┘

Common Patterns

Error Rate with Status Breakdown

rate(http_requests_total[5m])
/
ignoring(status) group_left
sum without (status) (rate(http_requests_total[5m]))

Join Version Info to Metrics

up * on(instance) group_left(version, commit)
build_info

Ratio of One Metric to Another

# Process memory as percentage of total memory
process_resident_memory_bytes
/
on(instance) group_left
node_memory_MemTotal_bytes
* 100

Decision Guide

Scenario	Use
Same labels on both sides	Default matching (no keyword needed)
Match on subset of labels	on(label1, label2)
Ignore some labels when matching	ignoring(label1)
Left side has more series	group_left
Right side has more series	group_right
Add metadata labels	group_left(extra_label)

Quick Reference

# One-to-one (exact label match)
metric_a / metric_b

# Match on specific labels
metric_a / on(label) metric_b

# Ignore labels when matching
metric_a / ignoring(label) metric_b

# Many-to-one (left has more series)
metric_a / on(label) group_left metric_b

# Many-to-one with extra label
metric_a * on(instance) group_left(version) info_metric

What’s Next?

Pre-compute expensive queries with recording rules — Recording Rules.