[pyupgrade] Restore the keep-runtime-typing setting (#5470)

## Summary

This PR reverts #4427. See the included documentation for a detailed
explanation.

Closes #5434.
This commit is contained in:
Charlie Marsh 2023-07-02 22:11:31 -04:00 committed by GitHub
parent 6cc04d64e4
commit c8b9a46e2b
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
13 changed files with 373 additions and 13 deletions

View file

@ -1,5 +1,31 @@
# Breaking Changes
## 0.0.276
### The `keep-runtime-typing` setting has been reinstated ([#5470](https://github.com/astral-sh/ruff/pull/5470))
The `keep-runtime-typing` setting has been reinstated with revised semantics. This setting was
removed in [#4427](https://github.com/astral-sh/ruff/pull/4427), as it was equivalent to ignoring
the `UP006` and `UP007` rules via Ruff's standard `ignore` mechanism.
Taking `UP006` (rewrite `List[int]` to `list[int]`) as an example, the setting now behaves as
follows:
- On Python 3.7 and Python 3.8, setting `keep-runtime-typing = true` will cause Ruff to ignore
`UP006` violations, even if `from __future__ import annotations` is present in the file.
While such annotations are valid in Python 3.7 and Python 3.8 when combined with
`from __future__ import annotations`, they aren't supported by libraries like Pydantic and
FastAPI, which rely on runtime type checking.
- On Python 3.9 and above, the setting has no effect, as `list[int]` is a valid type annotation,
and libraries like Pydantic and FastAPI support it without issue.
In short: `keep-runtime-typing` can be used to ensure that Ruff doesn't introduce type annotations
that are not supported at runtime by the current Python version, which are unsupported by libraries
like Pydantic and FastAPI.
Note that this is not a breaking change, but is included here to complement the previous removal
of `keep-runtime-typing`.
## 0.0.268
### The `keep-runtime-typing` setting has been removed ([#4427](https://github.com/astral-sh/ruff/pull/4427))

View file

@ -2108,6 +2108,7 @@ where
&& self.settings.target_version >= PythonVersion::Py37
&& !self.semantic.future_annotations()
&& self.semantic.in_annotation()
&& !self.settings.pyupgrade.keep_runtime_typing
{
flake8_future_annotations::rules::future_rewritable_type_annotation(
self, value,
@ -2118,7 +2119,8 @@ where
if self.settings.target_version >= PythonVersion::Py310
|| (self.settings.target_version >= PythonVersion::Py37
&& self.semantic.future_annotations()
&& self.semantic.in_annotation())
&& self.semantic.in_annotation()
&& !self.settings.pyupgrade.keep_runtime_typing)
{
pyupgrade::rules::use_pep604_annotation(
self, expr, slice, operator,
@ -2216,6 +2218,7 @@ where
&& self.settings.target_version >= PythonVersion::Py37
&& !self.semantic.future_annotations()
&& self.semantic.in_annotation()
&& !self.settings.pyupgrade.keep_runtime_typing
{
flake8_future_annotations::rules::future_rewritable_type_annotation(
self, expr,
@ -2226,7 +2229,8 @@ where
if self.settings.target_version >= PythonVersion::Py39
|| (self.settings.target_version >= PythonVersion::Py37
&& self.semantic.future_annotations()
&& self.semantic.in_annotation())
&& self.semantic.in_annotation()
&& !self.settings.pyupgrade.keep_runtime_typing)
{
pyupgrade::rules::use_pep585_annotation(
self,
@ -2291,6 +2295,7 @@ where
&& self.settings.target_version >= PythonVersion::Py37
&& !self.semantic.future_annotations()
&& self.semantic.in_annotation()
&& !self.settings.pyupgrade.keep_runtime_typing
{
flake8_future_annotations::rules::future_rewritable_type_annotation(
self, expr,
@ -2301,7 +2306,8 @@ where
if self.settings.target_version >= PythonVersion::Py39
|| (self.settings.target_version >= PythonVersion::Py37
&& self.semantic.future_annotations()
&& self.semantic.in_annotation())
&& self.semantic.in_annotation()
&& !self.settings.pyupgrade.keep_runtime_typing)
{
pyupgrade::rules::use_pep585_annotation(self, expr, &replacement);
}

View file

@ -60,6 +60,7 @@ NPY003.py:5:1: NPY003 [*] `np.cumproduct` is deprecated; use `np.cumprod` instea
5 |+np.cumprod(np.random.rand(5, 5))
6 6 | np.sometrue(np.random.rand(5, 5))
7 7 | np.alltrue(np.random.rand(5, 5))
8 8 |
NPY003.py:6:1: NPY003 [*] `np.sometrue` is deprecated; use `np.any` instead
|
@ -78,6 +79,8 @@ NPY003.py:6:1: NPY003 [*] `np.sometrue` is deprecated; use `np.any` instead
6 |-np.sometrue(np.random.rand(5, 5))
6 |+np.any(np.random.rand(5, 5))
7 7 | np.alltrue(np.random.rand(5, 5))
8 8 |
9 9 | from numpy import round_, product, cumproduct, sometrue, alltrue
NPY003.py:7:1: NPY003 [*] `np.alltrue` is deprecated; use `np.all` instead
|
@ -85,6 +88,8 @@ NPY003.py:7:1: NPY003 [*] `np.alltrue` is deprecated; use `np.all` instead
6 | np.sometrue(np.random.rand(5, 5))
7 | np.alltrue(np.random.rand(5, 5))
| ^^^^^^^^^^ NPY003
8 |
9 | from numpy import round_, product, cumproduct, sometrue, alltrue
|
= help: Replace with `np.all`
@ -94,5 +99,103 @@ NPY003.py:7:1: NPY003 [*] `np.alltrue` is deprecated; use `np.all` instead
6 6 | np.sometrue(np.random.rand(5, 5))
7 |-np.alltrue(np.random.rand(5, 5))
7 |+np.all(np.random.rand(5, 5))
8 8 |
9 9 | from numpy import round_, product, cumproduct, sometrue, alltrue
10 10 |
NPY003.py:11:1: NPY003 [*] `np.round_` is deprecated; use `np.round` instead
|
9 | from numpy import round_, product, cumproduct, sometrue, alltrue
10 |
11 | round_(np.random.rand(5, 5), 2)
| ^^^^^^ NPY003
12 | product(np.random.rand(5, 5))
13 | cumproduct(np.random.rand(5, 5))
|
= help: Replace with `np.round`
Suggested fix
8 8 |
9 9 | from numpy import round_, product, cumproduct, sometrue, alltrue
10 10 |
11 |-round_(np.random.rand(5, 5), 2)
11 |+round(np.random.rand(5, 5), 2)
12 12 | product(np.random.rand(5, 5))
13 13 | cumproduct(np.random.rand(5, 5))
14 14 | sometrue(np.random.rand(5, 5))
NPY003.py:12:1: NPY003 [*] `np.product` is deprecated; use `np.prod` instead
|
11 | round_(np.random.rand(5, 5), 2)
12 | product(np.random.rand(5, 5))
| ^^^^^^^ NPY003
13 | cumproduct(np.random.rand(5, 5))
14 | sometrue(np.random.rand(5, 5))
|
= help: Replace with `np.prod`
Suggested fix
9 9 | from numpy import round_, product, cumproduct, sometrue, alltrue
10 10 |
11 11 | round_(np.random.rand(5, 5), 2)
12 |-product(np.random.rand(5, 5))
12 |+prod(np.random.rand(5, 5))
13 13 | cumproduct(np.random.rand(5, 5))
14 14 | sometrue(np.random.rand(5, 5))
15 15 | alltrue(np.random.rand(5, 5))
NPY003.py:13:1: NPY003 [*] `np.cumproduct` is deprecated; use `np.cumprod` instead
|
11 | round_(np.random.rand(5, 5), 2)
12 | product(np.random.rand(5, 5))
13 | cumproduct(np.random.rand(5, 5))
| ^^^^^^^^^^ NPY003
14 | sometrue(np.random.rand(5, 5))
15 | alltrue(np.random.rand(5, 5))
|
= help: Replace with `np.cumprod`
Suggested fix
10 10 |
11 11 | round_(np.random.rand(5, 5), 2)
12 12 | product(np.random.rand(5, 5))
13 |-cumproduct(np.random.rand(5, 5))
13 |+cumprod(np.random.rand(5, 5))
14 14 | sometrue(np.random.rand(5, 5))
15 15 | alltrue(np.random.rand(5, 5))
NPY003.py:14:1: NPY003 [*] `np.sometrue` is deprecated; use `np.any` instead
|
12 | product(np.random.rand(5, 5))
13 | cumproduct(np.random.rand(5, 5))
14 | sometrue(np.random.rand(5, 5))
| ^^^^^^^^ NPY003
15 | alltrue(np.random.rand(5, 5))
|
= help: Replace with `np.any`
Suggested fix
11 11 | round_(np.random.rand(5, 5), 2)
12 12 | product(np.random.rand(5, 5))
13 13 | cumproduct(np.random.rand(5, 5))
14 |-sometrue(np.random.rand(5, 5))
14 |+any(np.random.rand(5, 5))
15 15 | alltrue(np.random.rand(5, 5))
NPY003.py:15:1: NPY003 [*] `np.alltrue` is deprecated; use `np.all` instead
|
13 | cumproduct(np.random.rand(5, 5))
14 | sometrue(np.random.rand(5, 5))
15 | alltrue(np.random.rand(5, 5))
| ^^^^^^^ NPY003
|
= help: Replace with `np.all`
Suggested fix
12 12 | product(np.random.rand(5, 5))
13 13 | cumproduct(np.random.rand(5, 5))
14 14 | sometrue(np.random.rand(5, 5))
15 |-alltrue(np.random.rand(5, 5))
15 |+all(np.random.rand(5, 5))

View file

@ -2,6 +2,7 @@
mod fixes;
mod helpers;
pub(crate) mod rules;
pub mod settings;
pub(crate) mod types;
#[cfg(test)]
@ -12,6 +13,7 @@ mod tests {
use test_case::test_case;
use crate::registry::Rule;
use crate::rules::pyupgrade;
use crate::settings::types::PythonVersion;
use crate::test::test_path;
use crate::{assert_messages, settings};
@ -85,6 +87,38 @@ mod tests {
Ok(())
}
#[test]
fn future_annotations_keep_runtime_typing_p37() -> Result<()> {
let diagnostics = test_path(
Path::new("pyupgrade/future_annotations.py"),
&settings::Settings {
pyupgrade: pyupgrade::settings::Settings {
keep_runtime_typing: true,
},
target_version: PythonVersion::Py37,
..settings::Settings::for_rule(Rule::NonPEP585Annotation)
},
)?;
assert_messages!(diagnostics);
Ok(())
}
#[test]
fn future_annotations_keep_runtime_typing_p310() -> Result<()> {
let diagnostics = test_path(
Path::new("pyupgrade/future_annotations.py"),
&settings::Settings {
pyupgrade: pyupgrade::settings::Settings {
keep_runtime_typing: true,
},
target_version: PythonVersion::Py310,
..settings::Settings::for_rule(Rule::NonPEP585Annotation)
},
)?;
assert_messages!(diagnostics);
Ok(())
}
#[test]
fn future_annotations_pep_585_p37() -> Result<()> {
let diagnostics = test_path(

View file

@ -0,0 +1,76 @@
//! Settings for the `pyupgrade` plugin.
use ruff_macros::{CacheKey, CombineOptions, ConfigurationOptions};
use serde::{Deserialize, Serialize};
#[derive(
Debug, PartialEq, Eq, Serialize, Deserialize, Default, ConfigurationOptions, CombineOptions,
)]
#[serde(
deny_unknown_fields,
rename_all = "kebab-case",
rename = "PyUpgradeOptions"
)]
#[cfg_attr(feature = "schemars", derive(schemars::JsonSchema))]
pub struct Options {
#[option(
default = r#"false"#,
value_type = "bool",
example = r#"
# Preserve types, even if a file imports `from __future__ import annotations`.
keep-runtime-typing = true
"#
)]
/// Whether to avoid PEP 585 (`List[int]` -> `list[int]`) and PEP 604
/// (`Union[str, int]` -> `str | int`) rewrites even if a file imports
/// `from __future__ import annotations`.
///
/// This setting is only applicable when the target Python version is below
/// 3.9 and 3.10 respectively, and is most commonly used when working with
/// libraries like Pydantic and FastAPI, which rely on the ability to parse
/// type annotations at runtime. The use of `from __future__ import annotations`
/// causes Python to treat the type annotations as strings, which typically
/// allows for the use of language features that appear in later Python
/// versions but are not yet supported by the current version (e.g., `str |
/// int`). However, libraries that rely on runtime type annotations will
/// break if the annotations are incompatible with the current Python
/// version.
///
/// For example, while the following is valid Python 3.8 code due to the
/// presence of `from __future__ import annotations`, the use of `str| int`
/// prior to Python 3.10 will cause Pydantic to raise a `TypeError` at
/// runtime:
///
/// ```python
/// from __future__ import annotations
///
/// import pydantic
///
/// class Foo(pydantic.BaseModel):
/// bar: str | int
/// ```
///
///
pub keep_runtime_typing: Option<bool>,
}
#[derive(Debug, Default, CacheKey)]
pub struct Settings {
pub keep_runtime_typing: bool,
}
impl From<Options> for Settings {
fn from(options: Options) -> Self {
Self {
keep_runtime_typing: options.keep_runtime_typing.unwrap_or_default(),
}
}
}
impl From<Settings> for Options {
fn from(settings: Settings) -> Self {
Self {
keep_runtime_typing: Some(settings.keep_runtime_typing),
}
}
}

View file

@ -0,0 +1,75 @@
---
source: crates/ruff/src/rules/pyupgrade/mod.rs
---
future_annotations.py:34:18: UP006 [*] Use `list` instead of `List` for type annotation
|
34 | def f(x: int) -> List[int]:
| ^^^^ UP006
35 | y = List[int]()
36 | y.append(x)
|
= help: Replace with `list`
Fix
31 31 | return cls(x=0, y=0)
32 32 |
33 33 |
34 |-def f(x: int) -> List[int]:
34 |+def f(x: int) -> list[int]:
35 35 | y = List[int]()
36 36 | y.append(x)
37 37 | return y
future_annotations.py:35:9: UP006 [*] Use `list` instead of `List` for type annotation
|
34 | def f(x: int) -> List[int]:
35 | y = List[int]()
| ^^^^ UP006
36 | y.append(x)
37 | return y
|
= help: Replace with `list`
Fix
32 32 |
33 33 |
34 34 | def f(x: int) -> List[int]:
35 |- y = List[int]()
35 |+ y = list[int]()
36 36 | y.append(x)
37 37 | return y
38 38 |
future_annotations.py:42:27: UP006 [*] Use `list` instead of `List` for type annotation
|
40 | x: Optional[int] = None
41 |
42 | MyList: TypeAlias = Union[List[int], List[str]]
| ^^^^ UP006
|
= help: Replace with `list`
Fix
39 39 |
40 40 | x: Optional[int] = None
41 41 |
42 |-MyList: TypeAlias = Union[List[int], List[str]]
42 |+MyList: TypeAlias = Union[list[int], List[str]]
future_annotations.py:42:38: UP006 [*] Use `list` instead of `List` for type annotation
|
40 | x: Optional[int] = None
41 |
42 | MyList: TypeAlias = Union[List[int], List[str]]
| ^^^^ UP006
|
= help: Replace with `list`
Fix
39 39 |
40 40 | x: Optional[int] = None
41 41 |
42 |-MyList: TypeAlias = Union[List[int], List[str]]
42 |+MyList: TypeAlias = Union[List[int], list[str]]

View file

@ -0,0 +1,4 @@
---
source: crates/ruff/src/rules/pyupgrade/mod.rs
---

View file

@ -20,7 +20,7 @@ use crate::rules::{
flake8_copyright, flake8_errmsg, flake8_gettext, flake8_implicit_str_concat,
flake8_import_conventions, flake8_pytest_style, flake8_quotes, flake8_self,
flake8_tidy_imports, flake8_type_checking, flake8_unused_arguments, isort, mccabe, pep8_naming,
pycodestyle, pydocstyle, pyflakes, pylint,
pycodestyle, pydocstyle, pyflakes, pylint, pyupgrade,
};
use crate::settings::options::Options;
use crate::settings::types::{
@ -93,6 +93,7 @@ pub struct Configuration {
pub pydocstyle: Option<pydocstyle::settings::Options>,
pub pyflakes: Option<pyflakes::settings::Options>,
pub pylint: Option<pylint::settings::Options>,
pub pyupgrade: Option<pyupgrade::settings::Options>,
}
impl Configuration {
@ -247,6 +248,7 @@ impl Configuration {
pydocstyle: options.pydocstyle,
pyflakes: options.pyflakes,
pylint: options.pylint,
pyupgrade: options.pyupgrade,
})
}
@ -334,6 +336,7 @@ impl Configuration {
pydocstyle: self.pydocstyle.combine(config.pydocstyle),
pyflakes: self.pyflakes.combine(config.pyflakes),
pylint: self.pylint.combine(config.pylint),
pyupgrade: self.pyupgrade.combine(config.pyupgrade),
}
}
}

View file

@ -1,10 +1,11 @@
use std::collections::HashSet;
use once_cell::sync::Lazy;
use path_absolutize::path_dedot;
use regex::Regex;
use rustc_hash::FxHashSet;
use std::collections::HashSet;
use super::types::{FilePattern, PythonVersion};
use super::Settings;
use crate::codes::{self, RuleCodePrefix};
use crate::line_width::{LineLength, TabSize};
use crate::registry::Linter;
@ -14,13 +15,10 @@ use crate::rules::{
flake8_copyright, flake8_errmsg, flake8_gettext, flake8_implicit_str_concat,
flake8_import_conventions, flake8_pytest_style, flake8_quotes, flake8_self,
flake8_tidy_imports, flake8_type_checking, flake8_unused_arguments, isort, mccabe, pep8_naming,
pycodestyle, pydocstyle, pyflakes, pylint,
pycodestyle, pydocstyle, pyflakes, pylint, pyupgrade,
};
use crate::settings::types::FilePatternSet;
use super::types::{FilePattern, PythonVersion};
use super::Settings;
pub const PREFIXES: &[RuleSelector] = &[
prefix_to_selector(RuleCodePrefix::Pycodestyle(codes::Pycodestyle::E)),
RuleSelector::Linter(Linter::Pyflakes),
@ -114,6 +112,7 @@ impl Default for Settings {
pydocstyle: pydocstyle::settings::Settings::default(),
pyflakes: pyflakes::settings::Settings::default(),
pylint: pylint::settings::Settings::default(),
pyupgrade: pyupgrade::settings::Settings::default(),
}
}
}

View file

@ -20,7 +20,7 @@ use crate::rules::{
flake8_copyright, flake8_errmsg, flake8_gettext, flake8_implicit_str_concat,
flake8_import_conventions, flake8_pytest_style, flake8_quotes, flake8_self,
flake8_tidy_imports, flake8_type_checking, flake8_unused_arguments, isort, mccabe, pep8_naming,
pycodestyle, pydocstyle, pyflakes, pylint,
pycodestyle, pydocstyle, pyflakes, pylint, pyupgrade,
};
use crate::settings::configuration::Configuration;
use crate::settings::types::{FilePatternSet, PerFileIgnore, PythonVersion, SerializationFormat};
@ -130,6 +130,7 @@ pub struct Settings {
pub pydocstyle: pydocstyle::settings::Settings,
pub pyflakes: pyflakes::settings::Settings,
pub pylint: pylint::settings::Settings,
pub pyupgrade: pyupgrade::settings::Settings,
}
impl Settings {
@ -284,6 +285,10 @@ impl Settings {
.pylint
.map(pylint::settings::Settings::from)
.unwrap_or_default(),
pyupgrade: config
.pyupgrade
.map(pyupgrade::settings::Settings::from)
.unwrap_or_default(),
})
}

View file

@ -12,7 +12,7 @@ use crate::rules::{
flake8_copyright, flake8_errmsg, flake8_gettext, flake8_implicit_str_concat,
flake8_import_conventions, flake8_pytest_style, flake8_quotes, flake8_self,
flake8_tidy_imports, flake8_type_checking, flake8_unused_arguments, isort, mccabe, pep8_naming,
pycodestyle, pydocstyle, pyflakes, pylint,
pycodestyle, pydocstyle, pyflakes, pylint, pyupgrade,
};
use crate::settings::types::{PythonVersion, SerializationFormat, Version};
@ -551,6 +551,9 @@ pub struct Options {
#[option_group]
/// Options for the `pylint` plugin.
pub pylint: Option<pylint::settings::Options>,
#[option_group]
/// Options for the `pyupgrade` plugin.
pub pyupgrade: Option<pyupgrade::settings::Options>,
// Tables are required to go last.
#[option(
default = "{}",

View file

@ -13,7 +13,7 @@ use ruff::rules::{
flake8_copyright, flake8_errmsg, flake8_gettext, flake8_implicit_str_concat,
flake8_import_conventions, flake8_pytest_style, flake8_quotes, flake8_self,
flake8_tidy_imports, flake8_type_checking, flake8_unused_arguments, isort, mccabe, pep8_naming,
pycodestyle, pydocstyle, pyflakes, pylint,
pycodestyle, pydocstyle, pyflakes, pylint, pyupgrade,
};
use ruff::settings::configuration::Configuration;
use ruff::settings::options::Options;
@ -166,6 +166,7 @@ pub fn defaultSettings() -> Result<JsValue, JsValue> {
pydocstyle: Some(pydocstyle::settings::Settings::default().into()),
pyflakes: Some(pyflakes::settings::Settings::default().into()),
pylint: Some(pylint::settings::Settings::default().into()),
pyupgrade: Some(pyupgrade::settings::Settings::default().into()),
})?)
}

25
ruff.schema.json generated
View file

@ -475,6 +475,17 @@
}
]
},
"pyupgrade": {
"description": "Options for the `pyupgrade` plugin.",
"anyOf": [
{
"$ref": "#/definitions/PyUpgradeOptions"
},
{
"type": "null"
}
]
},
"required-version": {
"description": "Require a specific version of Ruff to be running (useful for unifying results across many environments, e.g., with a `pyproject.toml` file).",
"anyOf": [
@ -1419,6 +1430,19 @@
},
"additionalProperties": false
},
"PyUpgradeOptions": {
"type": "object",
"properties": {
"keep-runtime-typing": {
"description": "Whether to avoid PEP 585 (`List[int]` -> `list[int]`) and PEP 604 (`Union[str, int]` -> `str | int`) rewrites even if a file imports `from __future__ import annotations`.\n\nThis setting is only applicable when the target Python version is below 3.9 and 3.10 respectively, and is most commonly used when working with libraries like Pydantic and FastAPI, which rely on the ability to parse type annotations at runtime. The use of `from __future__ import annotations` causes Python to treat the type annotations as strings, which typically allows for the use of language features that appear in later Python versions but are not yet supported by the current version (e.g., `str | int`). However, libraries that rely on runtime type annotations will break if the annotations are incompatible with the current Python version.\n\nFor example, while the following is valid Python 3.8 code due to the presence of `from __future__ import annotations`, the use of `str| int` prior to Python 3.10 will cause Pydantic to raise a `TypeError` at runtime:\n\n```python from __future__ import annotations\n\nimport pydantic\n\nclass Foo(pydantic.BaseModel): bar: str | int ```",
"type": [
"boolean",
"null"
]
}
},
"additionalProperties": false
},
"Pycodestyle": {
"type": "object",
"properties": {
@ -2042,6 +2066,7 @@
"NPY00",
"NPY001",
"NPY002",
"NPY003",
"PD",
"PD0",
"PD00",