Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[red-knot] add ecosystem check #15697

Open
carljm opened this issue Jan 23, 2025 · 0 comments
Open

[red-knot] add ecosystem check #15697

carljm opened this issue Jan 23, 2025 · 0 comments
Labels
red-knot Multi-file analysis & type inference

Comments

@carljm
Copy link
Contributor

carljm commented Jan 23, 2025

Description

We regularly observe the effects of changes to red-knot in the diagnostics emitted when running our (performance) benchmark against tomllib, and sometimes catch issues by seeing new false positives. But a) tomllib is one small codebase, not necessarily representative, and b) the ergonomics of catching these issues in the performance benchmark is poor.

We should select some larger real-world codebases and implement tooling to run red-knot on them and snapshot the diagnostics emitted, such that we can see (and easily update) this snapshot when we make changes to red-knot.

We should run this in CI to ensure our snapshot stays up to date.

(This is very similar to the ruff ecosystem check, and we can likely reuse some of that infrastructure?)

Bonus: if we implement #15696, we can also track changes to the prevalence of Todo types over time.

@carljm carljm added the red-knot Multi-file analysis & type inference label Jan 23, 2025
@carljm carljm added this to the Red Knot Q1 2025 milestone Jan 23, 2025
@carljm carljm changed the title [red-knot] correctness "benchmark" against real codebases [red-knot] add ecosystem check Jan 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
red-knot Multi-file analysis & type inference
Projects
None yet
Development

No branches or pull requests

1 participant