Tests for reporting baselined diagnostics

Baselined findings can be emitted at a configured severity

$ mkdir -p $TMPDIR/baseline_levels && \
> printf 'x: str = 1\n' > $TMPDIR/baseline_levels/matched.py && \
> printf 'def f() -> str:\n    return 1\n' > $TMPDIR/baseline_levels/new.py && \
> echo '{"errors":[{"column":10,"path":"matched.py","name":"bad-assignment","concise_description":"test","severity":"error"}]}' > $TMPDIR/baseline_levels/baseline.json && \
> printf 'baseline = "baseline.json"\nbaseline-error-level = "warn"\n' > $TMPDIR/baseline_levels/pyrefly.toml
[0]

The configured warning is hidden by the default error threshold.

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check matched.py --summary=none --output-format=min-text
[0]

At a matching threshold it is reported, marked as baselined, and affects the exit status normally.

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check matched.py --min-severity=warn --summary=none --output-format=min-text
 WARN matched.py:1:10-11: * [bad-assignment] [baselined] (glob)
[1]

The CLI level overrides configuration. info is likewise hidden until the minimum severity includes it.

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check matched.py --baseline-error-level=info --summary=none --output-format=min-text
[0]

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check matched.py --baseline-error-level=info --min-severity=info --summary=none --output-format=min-text
 INFO matched.py:1:10-11: * [bad-assignment] [baselined] (glob)
[1]

Explicit ignore hides matching findings while new errors are reported.

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check --baseline-error-level=ignore --summary=none --output-format=min-text
ERROR new.py:2:12-13: * [bad-return] (glob)
[1]

Baseline error levels do not increase a finding's severity

$ mkdir -p $TMPDIR/baseline_warning && \
> printf 'x: str = 1\n' > $TMPDIR/baseline_warning/warning.py && \
> printf 'baseline = "baseline.json"\n[errors]\nbad-assignment = "warn"\n' > $TMPDIR/baseline_warning/pyrefly.toml && \
> cd $TMPDIR/baseline_warning && \
> $PYREFLY check --update-baseline --min-severity=warn --summary=none --output-format=omit-errors >/dev/null 2>/dev/null; \
> $JQ -r '.errors[0].severity' baseline.json
warn
[0]

$ cd $TMPDIR/baseline_warning && \
> $PYREFLY check warning.py --baseline-error-level=error --min-severity=warn --summary=none --output-format=min-text
 WARN warning.py:1:10-11: * [bad-assignment] [baselined] (glob)
[1]

--only filters baselined and new findings

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check --baseline-error-level=error --only=bad-assignment --summary=none --output-format=min-text
ERROR matched.py:1:10-11: * [bad-assignment] [baselined] (glob)
[1]

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check --baseline-error-level=error --only=bad-return --summary=none --output-format=min-text
ERROR new.py:2:12-13: * [bad-return] (glob)
[1]

JSON records baseline provenance only when a baseline is configured

$ mkdir -p $TMPDIR/no_baseline && \
> echo "z: str = 1" > $TMPDIR/no_baseline/bad.py && \
> touch $TMPDIR/no_baseline/pyrefly.toml && \
> cd $TMPDIR/no_baseline && \
> $PYREFLY check --summary=none --output=json:diagnostics.json --output=sarif:diagnostics.sarif >/dev/null 2>/dev/null; \
> $JQ -c '[.errors[] | has("baselined")]' diagnostics.json && \
> $JQ -c '[.runs[0].results[] | has("baselineState")]' diagnostics.sarif
[false]
[false]
[0]

With a loaded baseline, matched and new findings are both identified. Using the error level reports both at full severity.

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check --baseline-error-level=error --summary=none --output=json:diagnostics.json >/dev/null 2>/dev/null; \
> $JQ -c '[.errors[] | {path, severity, baselined}]' diagnostics.json
[{"path":"matched.py","severity":"error","baselined":true},{"path":"new.py","severity":"error","baselined":false}]
[0]

SARIF records matching findings as unchanged and other findings as new

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check --baseline-error-level=error --summary=none --output=sarif:diagnostics.sarif >/dev/null 2>/dev/null; \
> $JQ -c '[.runs[0].results[] | {path: .locations[0].physicalLocation.artifactLocation.uri, baselineState}]' diagnostics.sarif
[{"path":"matched.py","baselineState":"unchanged"},{"path":"new.py","baselineState":"new"}]
[0]

Baseline regeneration reports configured but uncompared provenance

$ mkdir -p $TMPDIR/baseline_regeneration && \
> printf 'x: str = 1\n' > $TMPDIR/baseline_regeneration/bad.py && \
> touch $TMPDIR/baseline_regeneration/pyrefly.toml && \
> cd $TMPDIR/baseline_regeneration && \
> $PYREFLY check --baseline=baseline.json --update-baseline --summary=none --output=json:diagnostics.json --output=sarif:diagnostics.sarif >/dev/null 2>/dev/null; \
> $JQ -c '[.errors[] | .baselined]' diagnostics.json && \
> $JQ -c '[.runs[0].results[] | has("baselineState")]' diagnostics.sarif
[false]
[false]
[0]

GitHub Actions annotations mark baselined findings in their title

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check --baseline-error-level=warn --min-severity=warn --summary=none --output-format=github
::warning file=*/matched.py,line=1,col=10,endLine=1,endColumn=11,title=Pyrefly bad-assignment [baselined]::* (glob)
::error file=*/new.py,line=2,col=12,endLine=2,endColumn=13,title=Pyrefly bad-return::* (glob)
[1]

Omitted-error output reports the number of baselined diagnostics

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check matched.py --baseline-error-level=warn --min-severity=warn --output-format=omit-errors --progress-bar=no
 INFO 1 diagnostic (1 baselined)
[1]

Display metadata is not written into an updated baseline

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check --update-baseline --min-severity=warn --summary=none --output-format=omit-errors >/dev/null 2>/dev/null; \
> $JQ -c '[.errors[] | {severity, baselined: has("baselined")}]' baseline.json
[{"severity":"error","baselined":false},{"severity":"error","baselined":false}]
[0]

Pruning writes stored baseline severity and omits provenance

$ mkdir -p $TMPDIR/baseline_prune_provenance && \
> printf 'x: str = 1\n' > $TMPDIR/baseline_prune_provenance/matched.py && \
> echo '{"errors":[{"column":10,"path":"matched.py","name":"bad-assignment","concise_description":"test","severity":"info"},{"column":1,"path":"gone.py","name":"bad-return","concise_description":"stale","severity":"error"}]}' > $TMPDIR/baseline_prune_provenance/baseline.json && \
> touch $TMPDIR/baseline_prune_provenance/pyrefly.toml && \
> cd $TMPDIR/baseline_prune_provenance && \
> $PYREFLY check matched.py --baseline=baseline.json --baseline-error-level=warn --prune-baseline --summary=none --output-format=omit-errors >/dev/null 2>/dev/null; \
> $JQ -c '[.errors[] | {severity, baselined: has("baselined")}]' baseline.json
[{"severity":"info","baselined":false}]
[0]

Baselined display findings are not converted into inline suppressions

$ cd $TMPDIR/baseline_levels && \
> $PYREFLY check matched.py --baseline-error-level=error --only=bad-assignment --suppress-errors --summary=none >/dev/null 2>/dev/null; \
> ! grep -q pyrefly matched.py
[0]