Skip to content

Testing

This section describes the testing strategy of Quorum (quorum-sec-scan, v0.8.3 — revision 2026-07-04): what already exists in the code today (as-is), how to run it, and what is proposed as an evolution. Quorum is a CLI/Docker consensus security scanning tool written in Go 1.26; therefore, classic parts of a corporate test plan that assume a web frontend, relational database, or REST API are declared N/A with a technical justification. The product principle "false split > false merge" and the operational axiom "0 findings is not proof of safety" also guide the testing choices: we prefer to break early (contract tests, -race, consensus gates in E2E) over passing silently.

Related documents: Architecture · CI/CD · Supply chain · DESIGN.md (§5 contract tests, §6 correlation matrix, §14 scanner status).

1. Testing pyramid overview

Quorum concentrates weight at the base (unit + deterministic contract tests over fixtures) and keeps a lean but real top (E2E with actual scanners). There is no UI testing layer because there is no UI. The base grew with the product: the 12 scanners (trivy, grype, checkov, kics, dockle, kubescape, polaris, kube-score, terrascan, tfsec, regula, conftest) each have their own fixture and contract test, and consensus is now proven over real data across SCA, IaC, and K8s (AWS/Azure/GCP/k8s crosswalk).

flowchart TB
    E2E["Real E2E (e2e.yml)<br/>real scanners → cross-engine consensus<br/>alpine:3.10 + examples/terraform"]
    INT["Integration / pipeline<br/>real parse → correlate → crosswalk → consensus<br/>(realdata_test.go, pipeline_test.go, crosswalk_test.go)"]
    UNIT["Unit + Contract tests (12 adapters)<br/>adapters, orchestrator, alias, cache, consensus,<br/>crosswalk, filter, severity, purl, model, report, metrics, cmd"]

    UNIT --> INT --> E2E

    style UNIT fill:#dff0d8,stroke:#3c763d
    style INT fill:#fcf8e3,stroke:#8a6d3b
    style E2E fill:#d9edf7,stroke:#31708f
Layer Where it lives Deterministic? Runs in PR gate
Unit internal/**/<pkg>_test.go, cmd/quorum/scan_test.go Yes go test -race -covermode=atomic ./... (ci.yml) Yes
Contract (parsers, 12 adapters) internal/adapter/adapter_test.go + realdata_test.go vs internal/adapter/testdata/*.json Yes (versioned fixtures) go test ./... Yes
Integration (pipeline + crosswalk) internal/adapter/realdata_test.go, internal/correlate/pipeline_test.go, internal/crosswalk/crosswalk_test.go Yes (fixtures + real repo crosswalk) go test ./... Yes
Advisory-layer evals internal/evals/ (golden cases → remediation coverage, OWASP reference relevance, verify-the-fix rate) Yes (no model — fakes + real pack/corpus) go test ./... Yes
E2E (real consensus) .github/workflows/e2e.yml No (external network/DBs) GitHub Actions Yes (push/PR/manual)

2. Running locally

Relevant Makefile targets:

make test      # go test ./...  → unit + contract + integration
make vet       # go vet ./...
make all       # vet + test + build
make build     # binary in ./dist/quorum (CGO_ENABLED=0, -trimpath)
make run ARGS="scan <target> --type image --scanners trivy,grype"

Useful variations (not wrapped in a target, but supported):

go test -race ./...                         # race detector (same as CI)
go test ./internal/adapter/...              # the 12 adapter contract tests
go test -run TestMVPConsensus ./internal/correlate
go test -run TestShippedCrosswalkResolves ./internal/crosswalk   # locks the repo crosswalk
go test -cover ./...                        # per-package coverage (summary)
go test -race -covermode=atomic -coverprofile=coverage.out ./... && go tool cover -func=coverage.out

CI uses exactly go test -race -covermode=atomic -coverprofile=coverage.out ./... (see CI/CD), so running with -race locally reproduces the gate. make test uses go test ./... (without -race/coverage); both are valid, but CI is the reference.

3. Unit tests (current)

They cover pure logic and business rules, with no external dependencies (no network, no scanner binaries — except the contract tests, which read fixtures from disk).

Package File What it verifies
internal/severity severity_test.go FromCVSS (ranges → severity), FromDockle (FATAL/WARN/INFO), Max, AtLeast, Parse.
internal/model model_test.go Severity rank ordering (TestSeverityRankOrdering).
internal/purl purl_test.go Build, NameVersion, Name — normalization of Package URLs used in the SCA correlation key.
internal/filter filter_test.go --min-severity (Apply), .quorumignore baseline (match by fingerprint and by correlationKey), missing baseline.
internal/cache store_test.go Put/Get/on-disk persistence (creates parent directory, perm 0600), reopening, safety on nil *Store and in-memory mode (empty path).
internal/alias alias_test.go OSV client via httptest (aliases, 500 error, retry on 503, no retry on 404, malformed id rejection), preference for local aliases (zero network), cache after 1st resolution, graceful degradation on network failure.
internal/orchestrator orchestrator_test.go Cross-scanner fan-out and merge, ran/unavailable/error status, unknown scanner discarded with a warning, probe-timeout (ProbeTime) marking unavailable, ScannerRun serializing duration in ms.
internal/consensus consensus_test.go Merge groups by key and counts distinct scanners (TestMergeGroupsAndCountsDistinctScanners); engine diversity raises confidence (TestMergeDiversityRaisesConfidence).
internal/report report_test.go SARIF (schema 2.1.0, partialFingerprints, level, detectionCount), JSON (summary.totalFindings/multiDetected), XML (quorumReport), ParseFormat (sarif/json/xml case-insensitive, rejects pdf).
internal/report metrics_test.go WriteMetrics (flag --metrics, Prometheus textfile): quorum_scan_duration_seconds, quorum_scanner_up{scanner,status}, quorum_scanner_findings, quorum_findings_after_consensus, quorum_findings_total{severity}, quorum_multi_detected, and # TYPE headers.
internal/adapter adapter_test.go Common adapter infra: splitArgs (parsing quoted args), capWriter/maxOutputBytes (DoS cap QUORUM_MAX_OUTPUT_BYTES), redactSecretText (redacting AKIA…-type secrets), extraArgs (QUORUM_<SCANNER>_ARGS passthrough, e.g. QUORUM_CHECKOV_ARGS).
cmd/quorum scan_test.go validateTargetRef (refuses targets starting with -, argument injection), checkTargetSize (cap QUORUM_MAX_TARGET_BYTES, images skipped), emit (output with filepath.Clean), resolveCrosswalkDir (flag/default/fallback /opt/quorum/crosswalk), isDir.

Points of attention (good practices already adopted in the code):

  • Table-driven cases (TestFromCVSS, TestRealParse_Counts, TestSplitArgs).
  • t.TempDir() for FS isolation (cache, baseline, crosswalk, output).
  • t.Setenv to exercise environment knobs (QUORUM_MAX_OUTPUT_BYTES, QUORUM_MAX_TARGET_BYTES, QUORUM_*_ARGS) without leaking state across tests.
  • httptest.NewServer to isolate the OSV.dev network dependency — no unit test talks to the real internet.
  • t.Run subtests with t.Skipf/t.Skip when the environment does not satisfy the precondition (e.g. bundledCrosswalkDir only exists in the Docker image).

4. Adapter contract tests (current)

They are the format guard: they parse real, versioned outputs of each scanner and break before production when a tool changes its JSON (DESIGN §5). They live in internal/adapter/ and read fixtures from internal/adapter/testdata/. There is one contract test per scanner — the 12 adapters (trivy, grype, checkov, kics, dockle, kubescape, polaris, kube-score, terrascan, tfsec, regula, conftest).

Fixtures present (internal/adapter/testdata/):

├── trivy_image.json          grype_image.json
├── sca_trivy_alpine.json     sca_grype_alpine.json
├── iac_trivy.json            iac_trivy_v071.json
├── iac_checkov.json          iac_kics.json
├── iac_tfsec.json            iac_terrascan.json
├── iac_regula.json           policy_conftest.json
├── k8s_polaris.json          k8s_kubescore.json
└── img_dockle_alpine.json

Guarantees locked by the contract tests:

flowchart LR
    F["JSON fixture<br/>(real scanner output)"] --> P["adapter.parse()"]
    P --> C["canonical model.Finding"]
    C --> A{"assertions"}
    A --> A1["finding count + FindingType"]
    A --> A2["VulnID / PURL / Severity / CVSS"]
    A --> A3["Confirmed (authoritative)"]
    A --> A4["CanonicalControl (AVD / CIS / C-####)"]
    A --> A5["Aliases (CVE/GHSA)"]

TestRealParse_Counts locks the exact counts and the FindingType per fixture (invariants: non-empty Scanner/Title):

Scanner Fixture Findings Type
trivy (IaC) iac_trivy.json 12 MISCONFIG
checkov (IaC) iac_checkov.json 17 MISCONFIG
kics (IaC) iac_kics.json 9 MISCONFIG
trivy (SCA) sca_trivy_alpine.json 1 VULN
grype (SCA) sca_grype_alpine.json 1 VULN
polaris (K8s) k8s_polaris.json 3 K8S_POSTURE
kube-score (K8s) k8s_kubescore.json 2 K8S_POSTURE
tfsec (IaC) iac_tfsec.json 2 MISCONFIG
terrascan (IaC) iac_terrascan.json 2 MISCONFIG
regula (IaC) iac_regula.json 2 MISCONFIG
conftest (policy) policy_conftest.json 2 MISCONFIG

Notable cases (and why they exist):

  • TestTrivyParse / TestGrypeParse: mapping of canonical fields (PURL, severity, CVSS, Confirmed), Trivy passing AVD through (AVD-AWS-0086), and Grype version fallback to the descriptor.
  • TestTrivyV071AVDPrefix: locks the drift fix — Trivy ≥ ~0.60 dropped the AVDID field and emits ids "AWS-0086" without a prefix; the adapter restores AVD-AWS-0086 (fixture iac_trivy_v071.json, 9 findings, all AVD-AWS-0…) to match the crosswalk.
  • TestTfsecDerivesAVD: tfsec findings that carry an AVD link get CanonicalControl = that id (AVD-AWS-0089), enabling direct correlation with Trivy (which speaks native AVD); findings without an AVD link keep an empty CanonicalControl.
  • TestDockleParse: PASS/SKIP/IGNORE lines are discarded; FATAL/WARN/INFO become ImgHardening findings; CIS-DI-* codes already canonical; WARN→MEDIUM, INFO→LOW mapping (fixture alpine:3.10, 3 findings).

Maintenance: when bumping a scanner's supported version, capture a new fixture (running the scanner via the official image/binary against examples/terraform, examples/k8s, or alpine:3.10), add it to testdata/, and lock the expected count in TestRealParse_Counts — exactly the iac_trivy_v071.json pattern.

5. Integration / pipeline and consensus tests (current)

They differ from unit tests by exercising the real end-to-end data path (no network): parse → correlate.Enrich → crosswalk → consensus.Merge, using the real repository crosswalk (../../crosswalk: aws.yaml, azure.yaml, gcp.yaml, k8s.yaml). They are the deterministic proof that consensus happens over real data.

5.1 SCA pipeline and merge rules (correlate/pipeline_test.go)

Test Product invariant
TestMVPConsensus Trivy(CVE) + Grype(GHSA) → after alias resolution, 1 finding, detectionCount=2, confidence∈(0,1], CRITICAL severity. MVP "done" criterion.
TestUnmappedNeverMerges Misconfigs without a resolved canonical control are marked Unmapped and do not merge (false split > false merge).
TestCrosswalkMerges With a crosswalk rule mapping Checkov + Trivy to the same AVD-AWS-0091, they merge (detectionCount=2). Exercises the real YAML loader via crosswalk.Load.

5.2 Consensus over real data (adapter/realdata_test.go)

They parse real outputs and drive them through the pipeline + crosswalk + consensus. They cover SCA, IaC, and the three new correlation families (multi-engine IaC, K8s, terrascan↔trivy).

Test Product invariant
TestRealSCAConsensus Trivy+Grype agree on CVE-2021-36159 (apk-tools) → 1 finding, detectionCount=2, CRITICAL severity.
TestRealIaCConsensus Trivy+Checkov+KICS over the same Terraform: 3-way consensus on AVD-AWS-0090/0089/0092/0057; 2-way on AVD-AWS-0132 (only Trivy+Checkov cover it).
TestK8sCrosswalkConsensus Kubescape (C-0016, workload-level) + Polaris (privilegeEscalationAllowed, container-level) for the same concept → merge via k8s crosswalk + object-level key, detectionCount=2.
TestK8sThreeWayConsensus Kubescape (C-0057) + Polaris (runAsPrivileged) + kube-score (container-security-context-privileged) — "privileged container" → detectionCount=3.
TestTerrascanCrosswalkConsensus terrascan AC_AWS_0210 (public ACL) crosswalks to AVD-AWS-0092 and merges with Trivy's native AVD on the same S3 bucket → detectionCount=2. Locks the terrascan→AVD mapping.

Why the hub is AVD (cloud) and C-#### (k8s): the crosswalk is DERIVED from real output and picks the hub that minimizes false merge. tfsec and trivy already emit AVD and auto-correlate; kubescape anchors the k8s rules in C-####, and polaris/kube-score converge to it. RBAC stays single-engine (kubescape RBAC requires cluster context) — documented, not tested in consensus.

5.3 Crosswalk load and resolution (crosswalk/crosswalk_test.go)

The internal/crosswalk package now has dedicated tests that exercise the loader and lock the repository data:

Test What it guarantees
TestLoadAndResolve List-document loader; non-.yaml files ignored; case-insensitive scanner, verbatim rule id; unknown rule does not resolve (never "guesses").
TestLoadVersionedDocument Loader for the versioned document (schemaVersion: 1 + controls:).
TestShippedCrosswalkResolves Loads the real crosswalk (../../crosswalk) and checks 16 representative multi-cloud mappings — AWS (RDS/KMS/CloudTrail/VPC flow logs), Azure (secure transfer, min TLS, Key Vault purge protection), GCP (uniform access, RDP, Cloud SQL SSL), and k8s (privileged, host network, host PID/IPC, secrets). A bad edit to the crosswalk breaks the test.
TestLoadMissingDirIsEmptyNotError Missing directory → empty crosswalk, no error (graceful degradation).

These tests, together with E2E (§6), close the loop: consensus proven deterministically (fixtures + repo crosswalk) and non-deterministically with live binaries.

6. E2E tests (current)

.github/workflows/e2e.yml — "End-to-end proof that consensus actually happens with REAL scanners (not fixtures)". Triggers on push to main, on pull_request, and on workflow_dispatch.

Flow:

flowchart TB
    B["Build quorum (go build)"] --> I["Install real scanners<br/>trivy 0.71.2, grype 0.114.0, checkov (pipx)"]
    I --> V["Verify versions + grype db update + docker pull alpine:3.10 + quorum list-scanners"]
    V --> IAC["IaC: quorum scan examples/terraform<br/>--scanners trivy,checkov --offline"]
    V --> SCA["SCA: quorum scan alpine:3.10<br/>--scanners trivy,grype --offline"]
    IAC --> GI{"summary.multiDetected >= 1 ?"}
    SCA --> GS{"summary.multiDetected >= 1 ?"}
    GI -- no --> FAIL["::error:: no IaC consensus → fail"]
    GS -- no --> FAIL2["::error:: no SCA consensus → fail"]
    GI -- yes --> OK
    GS -- yes --> OK["upload iac.json + sca.json"]

Key characteristics:

  • Explicit consensus gate: the job fails if jq '.summary.multiDetected' is < 1 in any scenario — i.e. it fails if no finding is corroborated by two engines. Protects against silent regressions in correlation.
  • Same local image for both scanners (docker pull alpine:3.10) to ensure Trivy and Grype resolve the same artifact.
  • grype db update runs early to fail if the vulnerability DB cannot be fetched.
  • --offline turns off OSV.dev in the scenarios (resolution relies on local aliases), reducing network flakiness.
  • iac.json/sca.json reports are published as artifacts (if: always()).

E2E scope vs. fixtures: live E2E exercises 3 engines (trivy+checkov on IaC, trivy+grype on SCA) — the other 9 scanners and the K8s/multi-cloud scenarios are covered deterministically by the contract/consensus tests (§4–§5). This keeps E2E fast and stable while the fixtures ensure breadth of coverage.

N/A — application DAST: there is no server/endpoint to attack; the "E2E" here is a scanning pipeline, not a web application.

7. Load, stress, and chaos tests (proposed)

They do not exist today. Being a batch CLI (no long-running server), the relevant metrics are scan throughput and degradation under scarce resources, not RPS.

7.1 Load (proposed)

  • Goal: measure total time and memory usage when scanning large targets (monorepos, images with many packages) with the full pool of 12 scanners in fan-out.
  • How: a matrix of growing synthetic targets + measurement of --timeout, RSS peaks, and wall time; assert regression against a baseline. There are already DoS guardrails in the code (QUORUM_MAX_OUTPUT_BYTES=512MiB, QUORUM_MAX_TARGET_BYTES=20GiB) to exercise at the extremes.
  • Suggested tooling: hyperfine for wall-clock, /usr/bin/time -v for RSS, plus an optional Go harness (go test -bench) for the correlate/consensus pipeline.

7.2 Stress (proposed)

  • Goal: behavior when a scanner is killed by OOM or exceeds --timeout/ProbeTime. The orchestrator already distinguishes timeout/killed(OOM)/unavailable; a stress test would confirm this live.
  • How: run in a container with low --memory and heavy targets; assert that the per-scanner status is reported correctly (and exported via --metrics, gauge quorum_scanner_up{status}) and that the report never claims "0 findings = safe".

7.3 Chaos (proposed)

  • Goal: robustness to unstable dependencies: intermittent/slow OSV.dev, unreachable registry, missing Grype DB, corrupted fixture/crosswalk.
  • How: fault injection (proxy returning 5xx/latency for OSV, network blocking, invalid crosswalk) and verification of graceful degradation (already covered at the unit level in alias_test.go and crosswalk_test.go, missing the E2E equivalent).
  • Suggested tooling: toxiproxy for the network; dedicated "chaos" jobs in Actions.

8. Security tests (SAST/DAST/IAST/SCA/Container/IaC)

Quorum does dogfooding: it is a security tool, so it uses itself (and its own OSS scanners) to cover parts of this matrix. The examples/terraform, examples/k8s, and examples/ci examples serve both as fixtures and as dogfooding targets.

Discipline Status How today / proposal Tooling
SCA (dependencies) Partial via dogfooding E2E scans alpine:3.10 with trivy+grype; propose quorum scan . over the Go repo itself + govulncheck in CI Trivy, Grype, (propose) govulncheck
IaC scan Current via dogfooding E2E scans examples/terraform with trivy+checkov (consensus); tfsec/terrascan/regula/kics covered in the contract/consensus tests Trivy, Checkov, KICS, tfsec, terrascan, regula
K8s posture Current (fixtures) kubescape×polaris×kube-score consensus via the k8s crosswalk (realdata_test.go); propose dogfooding over examples/k8s in E2E kubescape, polaris, kube-score
Policy-as-code Current (fixture) conftest runs the user's Rego (./policy); contract test on policy_conftest.json conftest (OPA/Rego)
Container scan Current via dogfooding E2E over alpine:3.10; propose scanning the project's own :full/:slim images in release.yml Trivy, Grype, Dockle
SAST Proposed go vet already runs in CI; propose golangci-lint + gosec + CodeQL (Go) golangci-lint, gosec, CodeQL
DAST N/A There is no runtime surface (no API/server/web) to attack dynamically —
IAST N/A IAST requires instrumenting a running app serving requests; the batch CLI model does not apply —
Supply chain / provenance Current (hardened) Keyless-signed images and binaries (cosign OIDC, with retry) + SLSA build-provenance attestation and attested SPDX SBOM (actions/attest-sbom) per image and per binary (GoReleaser/syft), sha256-pinned bases, bundled scanners verified by checksum. The knowledge pack + crosswalk also get a SLSA build-provenance attestation each release (release.yml knowledge job; verify with gh attestation verify knowledge/owasp/corpus.yaml). The GitHub Action composite cosign-verifies :full and auto-mounts /var/run/docker.sock; moving v0 tag advanced by tag-major.yml on every semver release cosign, actions/attest-build-provenance, actions/attest-sbom, GoReleaser/syft, gh attestation
Secret scanning Partial The code already redacts secrets in findings (redactSecretText, tested); there is no dedicated secret scan in the repo — propose gitleaks in CI (internal) + (propose) gitleaks

Dogfooding as a security test: running quorum scan against the project's own artifacts (repo, example Terraform/K8s, published images) is simultaneously a functional test and Quorum's own IaC/K8s/Container/SCA security scan.

Security CI hardening checklist (proposal):

  • [ ] Add golangci-lint + gosec to ci.yml.
  • [ ] Enable CodeQL (Go) in a dedicated workflow.
  • [ ] Job running quorum scan . (SCA/IaC dogfooding) with --fail-on high.
  • [ ] Scan the freshly built :full/:slim images in release.yml before pushing.
  • [ ] govulncheck ./... as a gate.
  • [ ] gitleaks for secrets in the repo history.

9. API / CLI tests

N/A — REST/HTTP API: Quorum exposes no HTTP API, so there are no API contract tests, OpenAPI, or endpoint fuzzing.

CLI (the product's real "API"): partially covered, with clear gaps.

CLI aspect Status Where
Target validation (refuses -…, argument injection) Current cmd/quorum/scan_test.go (validateTargetRef)
Target size cap (QUORUM_MAX_TARGET_BYTES) Current cmd/quorum/scan_test.go (checkTargetSize)
--output with filepath.Clean (path traversal) Current cmd/quorum/scan_test.go (emit)
--crosswalk resolution (flag/default/fallback) Current cmd/quorum/scan_test.go (resolveCrosswalkDir)
Per-scanner passthrough (QUORUM_<SCANNER>_ARGS) Current internal/adapter/adapter_test.go (extraArgs, splitArgs)
--metrics (Prometheus textfile) Current internal/report/metrics_test.go (WriteMetrics)
list-scanners (smoke) Current ci.yml (Smoke) and e2e.yml (Verify)
Real end-to-end scan (--scanners/--format/--output/--offline) Current (E2E) e2e.yml
--log-format text\|json, --quiet, --baseline, --cache Gap propose dedicated coverage
Flag parsing/validation (--type, --format, --fail-on, --min-severity, --timeout) Gap propose table tests in cmd/quorum
Exit codes (0 ok / 1 --fail-on gate / 2 usage/runtime error) Gap propose E2E tests asserting $? per scenario
Advisory flags (--advice, --advice-provider, --fix) and advise-index Gap propose table tests for flag parsing/gating (--offline blocks remote, remote refuses --fix)

Exit-code test proposal (actionable in E2E):

# 0 = no finding reached --fail-on
quorum scan examples/terraform --type repo --min-severity critical --fail-on critical; test $? -eq 0
# 1 = gate fired
quorum scan alpine:3.10 --type image --fail-on low; test $? -eq 1
# 2 = invalid usage
quorum scan; test $? -eq 2

10. Performance

There are no benchmarks in the code today. Hot-spot candidates for testing.B:

  • consensus.Merge and correlate.Enrich (grouping by correlationKey, fingerprint hashing) over large sets of findings coming from up to 12 scanners.
  • crosswalk.Resolve over the combined AWS/Azure/GCP/k8s table.
  • SARIF/JSON/XML report generation and metrics writing at high volumes.
  • Orchestrator fan-out (goroutine/timeout overhead) with the full pool.

Proposal:

go test -bench=. -benchmem ./internal/consensus/... ./internal/correlate/... ./internal/crosswalk/... ./internal/report/...

Target metrics to establish as a baseline: merge-pipeline time for N findings (e.g. 1k, 10k), allocations/op, and wall-clock of a full :full scan (12 scanners) against a reference target.

11. Coverage

11.1 How it is measured (current)

Coverage is now collected in CI (ci.yml): the test step runs go test -race -covermode=atomic -coverprofile=coverage.out ./..., a "Coverage summary" step prints the total (go tool cover -func → last line) to $GITHUB_STEP_SUMMARY, and coverage.out is published as an artifact (actions/upload-artifact). Locally:

go test -covermode=atomic -coverprofile=coverage.out ./...
go tool cover -func=coverage.out          # per-function summary + total
go tool cover -html=coverage.out -o cover.html
go test -covermode=atomic -coverpkg=./... ./...   # cross-package coverage

11.2 Actual per-package state

Every product package now has a dedicated _test.go — the previous direct-coverage holes (consensus, crosswalk, purl, model) have been closed.

Package Has _test.go?
internal/adapter Yes (12 contract + realdata + infra)
internal/orchestrator Yes
internal/correlate (pipeline) Yes
internal/consensus Yes (consensus_test.go)
internal/crosswalk Yes (crosswalk_test.go)
internal/alias Yes
internal/cache Yes
internal/filter Yes
internal/severity Yes
internal/purl Yes (purl_test.go)
internal/model Yes (model_test.go)
internal/report Yes (report_test.go + metrics_test.go)
internal/evals Yes (evals_test.go — advisory-layer golden cases)
cmd/quorum Yes (target/size/output/crosswalk)

11.3 Goals (proposed)

  • Global goal: ≥ 80% line coverage (-coverpkg=./...).
  • Critical components (correlate, consensus, crosswalk, severity, report): ≥ 90% — they are the product core.
  • Non-regression gate: collection already exists; what is missing is adding a threshold (fail CI if total coverage drops below the baseline).

Coverage checklist:

  • [x] Collect -coverprofile in ci.yml and publish the artifact (coverage.out).
  • [x] Print the total in the job summary.
  • [ ] Define a minimum threshold globally and per critical package (non-regression gate).
  • [ ] Publish a historical coverage trend (badge/external service).

12. Master matrix: type → status → tooling

Test type Status Where / how Tooling
Unit Current internal/**/*_test.go, cmd/quorum/scan_test.go go test -race -covermode=atomic
Contract (scanner format, 12 adapters) Current internal/adapter/*_test.go + testdata/*.json go test, versioned fixtures
Integration (pipeline) Current realdata_test.go, pipeline_test.go go test
Consensus / crosswalk (AWS/Azure/GCP/k8s) Current realdata_test.go, crosswalk_test.go (real crosswalk) go test
Advisory-layer evals Current internal/evals/evals_test.go (remediation coverage, OWASP ref relevance, verify-the-fix rate) go test, real pack/corpus
E2E (real consensus) Current .github/workflows/e2e.yml trivy, grype, checkov, jq, Actions
Smoke (CLI) Current ci.yml / e2e.yml (list-scanners) compiled binary
Race detection Current ci.yml (go test -race) Go race detector
Static (vet) Current ci.yml (go vet) go vet
Coverage Current collected in ci.yml (summary + artifact); threshold §11.3 proposed go tool cover
Metrics (Prometheus) Current internal/report/metrics_test.go (--metrics) go test
Performance / Benchmark Proposed consensus/correlate/crosswalk/report go test -bench, hyperfine
Load Proposed large targets, 12-scanner fan-out hyperfine, time -v
Stress (OOM/timeout) Proposed container with low --memory Docker limits
Chaos (unstable deps) Proposed OSV 5xx/latency, registry off toxiproxy
SAST Proposed static lint/sec of the Go code golangci-lint, gosec, CodeQL
SCA Partial / dogfooding E2E alpine:3.10; propose self-scan + govulncheck Trivy, Grype, govulncheck
IaC scan Current / dogfooding E2E examples/terraform; fixtures for the 6 IaC engines Trivy, Checkov, KICS, tfsec, terrascan, regula
K8s posture Current / fixtures 2/3-way consensus realdata_test.go kubescape, polaris, kube-score
Policy-as-code Current / fixture contract test policy_conftest.json conftest (Rego)
Container scan Current / dogfooding E2E alpine:3.10; propose self-scan of the images Trivy, Grype, Dockle
DAST N/A no runtime surface —
IAST N/A no instrumentable running app —
API REST testing N/A no HTTP API —
Supply chain / provenance Current SLSA + SPDX SBOM attestation re-verified; knowledge pack/crosswalk attested; Action cosign-verifies :full; scanner checksums cosign, SLSA/SBOM attest, gh attestation
Secret scanning Partial internal redaction tested; repo scan proposed (internal), (propose) gitleaks
Exit codes (CLI) Proposed assert $? per scenario in E2E shell + Actions

13. Prioritized test backlog (actionable)

  • [ ] P0 — Define a coverage threshold in ci.yml (non-regression gate over the already-collected coverage.out, §11).
  • [ ] P0 — Exit-code tests (0/1/2) in E2E, covering --fail-on (§9).
  • [ ] P1 — SAST in CI: golangci-lint + gosec + CodeQL (§8).
  • [ ] P1 — Dogfooding in CI: quorum scan . and self-scan of the published images; include examples/k8s (§8).
  • [ ] P1 — govulncheck ./... as a gate (§8).
  • [ ] P2 — Cover flag parsing (--type/--format/--fail-on/--min-severity/--timeout) and --log-format in cmd/quorum (§9).
  • [ ] P2 — Benchmarks of consensus/correlate/crosswalk/report (§10).
  • [ ] P3 — Network chaos for OSV.dev via toxiproxy (§7.3).
  • [ ] P3 — OOM/timeout stress in a memory-limited container (§7.2).

Assumptions

  • As-is verified in code: the "Current" claims were checked by reading Makefile, .github/workflows/ci.yml, .github/workflows/e2e.yml, and the *_test.go files of internal/adapter (adapter_test.go, realdata_test.go), internal/correlate/pipeline_test.go, internal/crosswalk/crosswalk_test.go, internal/evals/evals_test.go, internal/report/metrics_test.go, and cmd/quorum/scan_test.go. "Proposed" explicitly indicates what does not exist today.
  • Product version: v0.8.3 (revision 2026-07-04); Go 1.26 (per go.mod and the workflows that pin go-version: "1.26").
  • 12 scanners each with a contract test and fixture: trivy, grype, checkov, kics, dockle, kubescape, polaris, kube-score, terrascan, tfsec, regula, conftest.
  • -race + coverage in CI: ci.yml uses go test -race -covermode=atomic -coverprofile=coverage.out ./...; Makefile test uses go test ./... (without -race/coverage). Both are valid; CI is the reference.
  • Coverage is collected today (summary + coverage.out artifact); what does not exist yet is a threshold/non-regression gate and the numeric goals (80%/90%), which remain proposals.
  • The advisory layer is opt-in and presentation-only: the internal/evals harness scores the deterministic advisory paths (Phase 0 remediation coverage, Phase 2 OWASP reference relevance) plus the verify-the-fix rate, using fakes + the real pack/corpus — no heavy model in CI. Without --advice, output is byte-identical, so the tests above stay unaffected.
  • E2E is not deterministic: it depends on scanner releases (trivy 0.71.2, grype 0.114.0, checkov via pipx), on the Grype DB, and on the alpine:3.10 image; upstream changes can affect counts. That is why the E2E gates check only multiDetected >= 1, not exact counts (those stay in the deterministic contract tests). Live E2E covers 3 engines; the rest and the K8s/multi-cloud scenarios are covered by fixtures.
  • DAST/IAST/API REST/web/relational DB = N/A because Quorum is a batch CLI/Docker tool, with no server, HTTP API, frontend, relational database, or authentication. The AI/LLM advisory layer is opt-in and off by default; when disabled the product remains AI-free, so the deterministic tests carry no LLM dependency.
  • Dogfooding assumes examples/terraform, examples/k8s, and examples/ci remain valid scan targets in the repository; conftest assumes a Rego bundle in ./policy provided by the user.

Known gaps

  • The content of each individual JSON fixture was not inspected beyond what the tests assert; the cited counts come from the assertions (TestRealParse_Counts and the consensus tests).
  • Actual coverage numbers (percentages) are not cited in this document — the total value appears in the CI job summary on each run, but there is no fixed versioned baseline.
  • The K8s consensus scenarios (TestK8sCrosswalkConsensus, TestK8sThreeWayConsensus, TestTerrascanCrosswalkConsensus) use findings constructed in code (not file fixtures) for the engines not parsed in that case; the real per-scanner K8s capture lives in the k8s_polaris.json/k8s_kubescore.json fixtures of TestRealParse_Counts.
  • RBAC (kubescape) stays single-engine because it requires cluster context; there is no consensus test for that family — it is a documented architectural decision, not a test gap.