Skip to content

Policy Validation

Back to main README

This document describes how cov-loupe allows users to define and enforce custom coverage policies through validation predicates.

Ruby instance_eval for Validation Predicates

Status

Accepted

Context

cov-loupe needed a mechanism for users to define custom coverage policies beyond simple percentage thresholds. Different projects have different requirements:

  • Some want all files above 80%, others allow a few files below threshold
  • Some need different thresholds for different directories (e.g., 90% for API code, 60% for legacy)
  • Some want total project coverage minimums
  • CI/CD pipelines need exit codes based on policy compliance

We considered several approaches:

  1. Built-in policy DSL: Define a limited language for expressing policies (e.g., YAML/JSON config)
  2. Plugin architecture: Define a protocol/interface, require users to create Ruby classes implementing it
  3. Ruby file evaluation: Load and execute arbitrary Ruby code that returns a callable predicate
  4. Sandboxed DSL: Use a restricted Ruby environment (e.g., $SAFE levels, isolated VMs)

Key Requirements

  • Flexibility: Support arbitrarily complex coverage policies
  • Simplicity: Easy for users to write and understand
  • Debuggability: Users can use standard Ruby debugging tools
  • CI/CD integration: Clear exit codes (0 = pass, 1 = fail, 2 = error)
  • Access to coverage data: Predicates need access to the full CoverageModel API

Why Not a Custom DSL?

A custom DSL would be: - Limited in expressiveness (hard to predict all future use cases) - Harder to debug (users can't use standard Ruby tools) - More maintenance burden (parsing, validation, documentation) - Still vulnerable to injection if it allowed any dynamic computation

Why Not Sandboxing?

Ruby's sandboxing options are limited: - $SAFE levels were deprecated and removed in Ruby 2.7+ - Full VM isolation (Docker, etc.) is too heavy for a CLI tool - Any Turing-complete sandbox can be escaped given enough effort - True security requires not executing untrusted code at all

Decision

We chose to evaluate Ruby files using instance_eval with prominent security warnings rather than attempting to create a false sense of security through incomplete sandboxing.

Implementation

Predicate evaluation is centralized in lib/cov_loupe/config/predicate_evaluator.rb. The CLI validate subcommand and the MCP project_validate tool both use this evaluator.

def self.evaluate_file(path, model)
  unless File.exist?(path)
    raise "Predicate file not found: #{path}"
  end

  content = File.read(path)

  # WARNING: The predicate code executes with full Ruby privileges.
  # It has unrestricted access to the file system, network, and system commands.
  # Only use predicate files from trusted sources.
  #
  # We evaluate in a fresh Object context to prevent accidental access to
  evaluation_context = Object.new
  predicate = evaluation_context.instance_eval(content, path, 1)

  validate_callable(predicate)
  predicate.call(model)
rescue SyntaxError => e
  raise "Syntax error in predicate file: #{e.message}"
end

The CLI validate command exits according to the predicate result:

result = if inline_mode
  PredicateEvaluator.evaluate_code(code, model)
else
  PredicateEvaluator.evaluate_file(code, model)
end

exit(result ? 0 : 1)

Predicate errors are converted to exit code 2:

private def handle_predicate_error(error)
  warn "Predicate error: #{error.message}"
  exit 2
end

Security Model: Treat as Executable Code

Rather than pretending to sandbox untrusted code, we treat validation predicates exactly like any other Ruby code in the project:

  1. Prominent warnings in documentation (highlighted near the top of examples/success_predicates/README.md): 

    ⚠️ SECURITY WARNING

Validation predicates execute as arbitrary Ruby code with full system privileges.
Only use predicate files from trusted sources.
- Never use predicates from untrusted or unknown sources
- Review predicates before use, especially in CI/CD environments
- Store predicates in version control with code review

  2. Code review workflow: Predicates live in version control alongside tests

  3. CI/CD best practices: Same permissions model as running tests themselves
  4. Example predicates: Well-documented examples showing safe patterns

Predicate API

Validation predicates must be callable (lambda, proc, or object with #call method):

Lambda example: 

->(model) do
  model.list["files"].all? { |f| f['percentage'] >= 80 }
end

Class example:

class CoveragePolicy
  def call(model)
    api_files = model.list['files'].select { |f| f['file'].start_with?('lib/api/') }
    api_files.all? { |f| f['percentage'] >= 90 }
  end
end

CoveragePolicy.new

The predicate receives a full CoverageModel instance with access to: - list(tracked_globs:, sort_order:) - All file coverage data - summary_for(path) - Coverage summary for a specific file - uncovered_for(path) - Uncovered lines for a specific file - detailed_for(path) - Per-line coverage data

Consequences

Positive

  1. Maximum flexibility: Users can express arbitrarily complex coverage policies using full Ruby
  2. Familiar tooling: Users can debug predicates with standard Ruby tools (pry, byebug, etc.)
  3. Simplicity: No custom DSL to learn, document, or maintain
  4. Honesty: Security model is clear and doesn't provide false confidence
  5. Composability: Users can require other libraries, define helper methods, etc.
  6. Excellent examples: We provide 5+ well-documented example predicates

Negative

  1. Security responsibility: Users must understand the security implications
  2. Potential misuse: Users might mistakenly trust untrusted predicate files
  3. No isolation: Buggy predicates can access/modify anything in the system
  4. Documentation burden: Must clearly communicate security model

Trade-offs

  • Versus custom DSL: More powerful and debuggable, but requires user awareness of security
  • Versus plugin architecture: Simpler (no gem dependencies, no protocol to learn), but same security profile
  • Versus incomplete sandboxing: Honest about capabilities rather than security theater

Threat Model

This approach is appropriate when: - Predicate files are stored in version control with code review - Users treat predicates like any other code in their project (tests, Rakefile, etc.) - CI/CD environments already execute arbitrary code (tests, build scripts)

This approach is inappropriate when: - Processing untrusted predicate files from unknown sources - Allowing users to upload predicates via web interface - Running in a multi-tenant environment without isolation

Future Considerations

If demand arises for truly untrusted predicate execution, alternatives include:

  1. JSON-based policy format: Limited expressiveness but safe
  2. WebAssembly sandbox: Execute policies in an isolated WASM runtime
  3. External process: Run predicates in separate process with restricted permissions

However, for the primary use case (CI/CD policy enforcement), the current approach is simpler and more flexible than these alternatives.

References

  • Implementation: lib/cov_loupe/config/predicate_evaluator.rb, lib/cov_loupe/commands/validate_command.rb, lib/cov_loupe/tools/project_validate_tool.rb
  • Security warnings: examples/success_predicates/README.md
  • Example predicates: examples/success_predicates/*.rb
  • CoverageModel API: lib/cov_loupe/model/model.rb
  • CLI config: lib/cov_loupe/config/app_config.rb
  • Option parsing: lib/cov_loupe/config/option_parser_builder.rb and lib/cov_loupe/commands/validate_command.rb