Coverage Data Quality¶

Back to main README

This document describes how cov-loupe ensures the accuracy and reliability of coverage data through staleness detection.

Coverage Staleness Detection¶

Status¶

Accepted

Context¶

Coverage data can become outdated when source files are modified after tests run. This creates misleading results:

Coverage percentages appear lower/higher than reality
Line numbers in coverage reports don't match the current source
AI agents and users may make decisions based on stale data

We needed a staleness detection system that could:

Detect when source files have been modified since coverage was collected
Detect when source files have different line counts than coverage data
Handle edge cases (deleted files)
Support both file-level and project-level checks
Allow users to control whether staleness is reported or causes errors

Alternative Approaches Considered¶

No staleness checking: Simple, but leads to confusing/incorrect reports
Single timestamp check: Fast, but misses line count mismatches (files edited and reverted)
Content hashing: Accurate, but expensive for large projects
Multi-type detection with modes: More complex, but provides accurate detection with user control

Decision¶

We implemented a staleness detection system with configurable error modes that can identify four distinct staleness conditions.

Four Staleness Types¶

The StalenessChecker class (defined in lib/cov_loupe/staleness/staleness_checker.rb) detects four distinct types of staleness:

Type "error" (Error): The staleness check itself failed
Returned by CoverageModel#staleness_for when an exception is raised during staleness checking
Example: File permission errors, resolver failures, or other unexpected issues
The error is logged but execution continues with an "error" status instead of crashing
Type "missing" (Missing): The source file exists in coverage but is now deleted/missing
Returned by file_staleness_status when File.file?(file_abs) returns false
Example: File was deleted after tests ran
Type "newer" (Timestamp): The source file's mtime is newer than coverage timestamp
Detected by comparing File.mtime(file_abs) with coverage timestamp
Example: File was edited after tests ran
Type "length_mismatch" (Length): The source file line count doesn't match the coverage lines array length
Detected by comparing File.foreach(path).count with coverage_lines.length
Example: Lines were added/removed without changing mtime (rare but possible with version control)
Type "ok" (Not stale): The file is not stale
Returned when none of the above staleness conditions apply
Indicates the coverage data is current and accurate

Implementation Details¶

The core algorithm lives in CovLoupe::StalenessChecker#compute_file_staleness_details:

def compute_file_staleness_details(file_abs, coverage_lines)
  coverage_ts = coverage_timestamp
  exists = File.file?(file_abs)
  file_mtime = exists ? File.mtime(file_abs) : nil

  cov_len = coverage_lines.respond_to?(:length) ? coverage_lines.length : 0
  src_len = exists ? safe_count_lines(file_abs) : 0

  # If coverage timestamp is 0 (missing/invalid), we cannot determine if file is newer
  newer = if coverage_ts.to_i > 0
            !!(file_mtime && file_mtime.to_i > coverage_ts.to_i)
          else
            false
          end

  len_mismatch = (cov_len.positive? && src_len != cov_len)
  newer &&= !len_mismatch  # Prioritize length mismatch over timestamp

  {
    exists: exists,
    file_mtime: file_mtime,
    coverage_timestamp: coverage_ts,
    cov_len: cov_len,
    src_len: src_len,
    newer: newer,
    len_mismatch: len_mismatch
  }
end

Staleness Modes¶

The checker supports two modes, configured when instantiating StalenessChecker:

:off (default): Staleness is detected but only reported in responses, never raises errors
"error": Staleness raises CoverageDataStaleError or CoverageDataProjectStaleError

This allows: - Interactive tools to show warnings without crashing - CI systems to fail builds on stale coverage - AI agents to decide how to handle staleness based on their goals

File-Level vs Project-Level Checks¶

File-level (check_file! and file_staleness_status): - Checks a single file's staleness - Returns one of the staleness status strings ("ok", "missing", "newer", "length_mismatch", "error") - Used by single-file tools (summary, detailed, uncovered)

Project-level (check_project!): - Checks all covered files plus optionally tracked files - Detects: - Files newer than coverage timestamp - Files deleted since coverage was collected - Tracked files missing from coverage (newly added files) - Raises CoverageDataProjectStaleError with lists of problematic files - Used by list_tool and coverage_table_tool

Totals behavior: - project_totals excludes any stale files ("missing", "newer", "length_mismatch", "error") from aggregate counts. - Totals include explicit with_coverage/without_coverage breakdowns so callers can reconcile what was omitted. - The without_coverage payload includes counts for three categories: - missing_from_coverage: Tracked files that have no coverage data in the resultset - unreadable: Files that exist but could not be read (e.g., due to permission errors, I/O issues, or staleness check failures) - skipped: Files that were skipped during list processing due to coverage data errors (e.g., malformed entries) - The unreadable count is populated from list_result['unreadable_files'], which is collected during staleness checking when files exist but cannot be accessed or validated.

Tracked Globs Feature¶

The project-level check supports tracked_globs parameter to detect newly added files:

# Detects if lib/**/*.rb files exist that have no coverage data
checker.check_project!(coverage_map)  # with tracked_globs: ['lib/**/*.rb']

This helps teams ensure new files are included in test runs.

Resultset Path Consistency (SimpleCov)¶

SimpleCov can emit mixed path forms for the same file when resultsets are merged across suites or environments (for example, absolute vs relative paths, or different roots). This is a SimpleCov data consistency risk, not a cov-loupe behavior. Downstream tools that normalize paths may treat one entry as overriding another when multiple keys map to the same absolute path.

Guidance: Keep SimpleCov.root consistent across all suites and avoid manual path rewriting before merging resultsets.

Consequences¶

Positive¶

Accurate detection: Three types catch different staleness scenarios comprehensively
User control: Modes allow errors or warnings based on use case
Detailed information: Staleness errors include specific file lists and timestamps
Project awareness: Can detect newly added files that lack coverage
Conservative totals: Aggregate totals only include fresh coverage data

Negative¶

Complexity: Three staleness types are harder to understand than a single timestamp check
Performance: Line counting and mtime checks for every file add overhead
Ambiguity: When multiple staleness types apply, prioritization logic (length > timestamp) may surprise users

Trade-offs¶

Versus timestamp-only: More accurate but slower and more complex
Versus content hashing: Fast enough for most projects, but can't detect "edit then revert" scenarios
Versus no checking: Essential for reliable coverage reporting, worth the complexity

Edge Cases Handled¶

Deleted files: Appear as "missing" type staleness
Empty files: cov_len.positive? guard prevents false positives
No coverage timestamp: Defaults to 0, effectively disabling timestamp checks

References¶

Implementation: lib/cov_loupe/staleness_checker.rb (StalenessChecker class)
File-level checking: StalenessChecker#check_file! and #file_staleness_status
Project-level checking: StalenessChecker#check_project!
Staleness detail computation: StalenessChecker#compute_file_staleness_details
Error types: lib/cov_loupe/errors.rb (CoverageDataStaleError, CoverageDataProjectStaleError)
Usage in tools: lib/cov_loupe/tools/list_tool.rb, lib/cov_loupe/model.rb