Use Jupyter mode while parsing Notebook files (#5552)

## Summary

Enable using the new `Mode::Jupyter` for the tokenizer/parser to parse
Jupyter line magic tokens.

The individual call to the lexer i.e., `lex_starts_at` done by various
rules should consider the context of the source code (is this content
from a Jupyter Notebook?). Thus, a new field `source_type` (of type
`PySourceType`) is added to `Checker` which is being passed around as an
argument to the relevant functions. This is then used to determine the
`Mode` for the lexer.

## Test Plan

Add new test cases to make sure that the magic statement is considered
while generating the diagnostic and autofix:
* For `I001`, if there's a magic statement in between two import blocks,
they should be sorted independently

fixes: #6090
This commit is contained in:
Dhruv Manilawala 2023-08-05 06:02:07 +05:30 committed by GitHub
parent d788957ec4
commit 32fa05765a
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
52 changed files with 652 additions and 196 deletions

View file

@ -9,6 +9,7 @@ use ruff::linter::lint_only;
use ruff::settings::{flags, Settings};
use ruff::RuleSelector;
use ruff_benchmark::{TestCase, TestCaseSpeed, TestFile, TestFileDownloadError};
use ruff_python_ast::PySourceType;
#[cfg(target_os = "windows")]
#[global_allocator]
@ -57,13 +58,15 @@ fn benchmark_linter(mut group: BenchmarkGroup<WallTime>, settings: &Settings) {
&case,
|b, case| {
b.iter(|| {
let path = case.path();
let result = lint_only(
case.code(),
&case.path(),
&path,
None,
settings,
flags::Noqa::Enabled,
None,
PySourceType::from(path.as_path()),
);
// Assert that file contains no parse errors