Metrics and Proof: Understand Contexity Value Estimates

Contexity can display heuristic value metrics after relevant work completes. These estimates help you understand where context retrieval is saving time and tokens — but they come with clearly stated limits, and they are not a substitute for rigorous benchmarking when public claims are involved.

Example Metrics Output

After a task closes, you may see a summary like this:

---
Contexity metrics, heuristic:
- Tokens saved: ~1,400 (93.4% of estimated 1,499-token total)
- Time saved: ~7m 12s (100% of estimated 7m 13s total)
- Contexity runtime: ~35ms observed
- Proof level: heuristic estimate; use paired A/B for measured proof.

What the Metrics Mean

Metric	Meaning
Tokens saved	Estimated context or graph-discovery tokens avoided by using Contexity
Saved percent	Saved amount compared with the estimated total token cost
Time saved	Heuristic time avoided through context reuse or graph lookup
Runtime	Measured Contexity tool runtime for the completed task
Proof level	Whether the number is a heuristic estimate or backed by a paired benchmark

Heuristic vs. Benchmark Proof

Heuristic metrics serve a specific and useful purpose. Use them for:

Product feedback and iteration
Internal diagnosis of retrieval quality
Explaining why Contexity helped during a particular run
Spotting patterns where retrieval consistently saves work

They are not sufficient for public marketing claims. If you need to make a verifiable, public statement about Contexity’s impact, you need paired A/B benchmarks structured as follows:

Same task, identical problem statement
Same repo state at the start of both runs
One Contexity-assisted run and one baseline run without Contexity
Multiple valid trials to account for variance
Clear exclusion rules for incomplete or invalid runs
Confidence intervals where statistical rigor is required

Heuristic output is a useful internal signal. Benchmark output is evidence you can stand behind publicly.

Controlling Metrics Display

You can disable visible metric summaries for a project while keeping structured metrics available for logging and tooling:

contexity metrics disable --project /path/to/project

To re-enable visible metrics:

contexity metrics enable --project /path/to/project

For full details on configuring and managing metrics output, see the manage metrics guide.

Get Started

Core Concepts

Guides

Help

Metrics and Proof: Understand Contexity Value Estimates

Example Metrics Output

What the Metrics Mean

Heuristic vs. Benchmark Proof

Controlling Metrics Display

​Example Metrics Output

​What the Metrics Mean

​Heuristic vs. Benchmark Proof

​Controlling Metrics Display

Example Metrics Output

What the Metrics Mean

Heuristic vs. Benchmark Proof

Controlling Metrics Display