Commit graph

12 commits

Author SHA1 Message Date
Wei Lee
2288cc7478
[airflow]: extend names moved from core to provider (AIR303) (#15159) 2024-12-27 17:04:02 +00:00
Wei Lee
5d6aae839e
[airflow]: extend moved names (AIR303) (#15145) 2024-12-26 10:55:07 +01:00
Wei Lee
14a5a2629e
[airflow]: extend removed method calls (AIR302) (#15054)
## Summary


Airflow 3.0 removes various deprecated functions, members, modules, and
other values. They have been deprecated in 2.x, but the removal causes
incompatibilities that we want to detect. This PR deprecates the
following names and add a function for removed methods

* `airflow.datasets.manager.DatasetManager.register_dataset_change` →
`airflow.assets.manager.AssetManager.register_asset_change`
* `airflow.datasets.manager.DatasetManager.create_datasets` →
`airflow.assets.manager.AssetManager.create_assets`
* `airflow.datasets.manager.DatasetManager.notify_dataset_created` →
`airflow.assets.manager.AssetManager.notify_asset_created`
* `airflow.datasets.manager.DatasetManager.notify_dataset_changed` →
`airflow.assets.manager.AssetManager.notify_asset_changed`
* `airflow.datasets.manager.DatasetManager.notify_dataset_alias_created`
→ `airflow.assets.manager.AssetManager.notify_asset_alias_created`
*
`airflow.providers.amazon.auth_manager.aws_auth_manager.AwsAuthManager.is_authorized_dataset`
→
`airflow.providers.amazon.auth_manager.aws_auth_manager.AwsAuthManager.is_authorized_asset`
* `airflow.lineage.hook.HookLineageCollector.create_dataset` →
`airflow.lineage.hook.HookLineageCollector.create_asset`
* `airflow.lineage.hook.HookLineageCollector.add_input_dataset` →
`airflow.lineage.hook.HookLineageCollector.add_input_asset`
* `airflow.lineage.hook.HookLineageCollector.add_output_dataset` →
`airflow.lineage.hook.HookLineageCollector.dd_output_asset`
* `airflow.lineage.hook.HookLineageCollector.collected_datasets` →
`airflow.lineage.hook.HookLineageCollector.collected_assets`
*
`airflow.providers_manager.ProvidersManager.initialize_providers_dataset_uri_resources`
→
`airflow.providers_manager.ProvidersManager.initialize_providers_asset_uri_resources`

## Test Plan

A test fixture is included in the PR.
2024-12-20 08:30:30 +01:00
Wei Lee
867a8f9497
feat(AIR302): extend the following rules (#15015)
## Summary


Airflow 3.0 removes various deprecated functions, members, modules, and
other values. They have been deprecated in 2.x, but the removal causes
incompatibilities that we want to detect. This PR deprecates the
following names.

* `airflow.api_connexion.security.requires_access_dataset` →
`airflow.api_connexion.security.requires_access_asset`
* `airflow.auth.managers.base_auth_manager.is_authorized_dataset` →
`airflow.auth.managers.base_auth_manager.is_authorized_asset`
* `airflow.auth.managers.models.resource_details.DatasetDetails` →
`airflow.auth.managers.models.resource_details.AssetDetails`
* `airflow.lineage.hook.DatasetLineageInfo` →
`airflow.lineage.hook.AssetLineageInfo`
* `airflow.security.permissions.RESOURCE_DATASET` →
`airflow.security.permissions.RESOURCE_ASSET`
* `airflow.www.auth.has_access_dataset` →
`airflow.www.auth.has_access_dataset.has_access_asset`
* remove `airflow.datasets.DatasetAliasEvent`
* `airflow.datasets.Dataset` → `airflow.sdk.definitions.asset.Asset`
* `airflow.Dataset` → `airflow.sdk.definitions.asset.Asset`
* `airflow.datasets.DatasetAlias` →
`airflow.sdk.definitions.asset.AssetAlias`
* `airflow.datasets.DatasetAll` →
`airflow.sdk.definitions.asset.AssetAll`
* `airflow.datasets.DatasetAny` →
`airflow.sdk.definitions.asset.AssetAny`
* `airflow.datasets.metadata` → `airflow.sdk.definitions.asset.metadata`
* `airflow.datasets.expand_alias_to_datasets` →
`airflow.sdk.definitions.asset.expand_alias_to_assets`
* `airflow.datasets.manager.dataset_manager` → `airflow.assets.manager`
* `airflow.datasets.manager.resolve_dataset_manager` →
`airflow.assets.resolve_asset_manager`
* `airflow.datasets.manager.DatasetManager` →
`airflow.assets.AssetManager`
* `airflow.listeners.spec.dataset.on_dataset_created` →
`airflow.listeners.spec.asset.on_asset_created`
* `airflow.listeners.spec.dataset.on_dataset_changed` →
`airflow.listeners.spec.asset.on_asset_changed`
* `airflow.timetables.simple.DatasetTriggeredTimetable` →
`airflow.timetables.simple.AssetTriggeredTimetable`
* `airflow.timetables.datasets.DatasetOrTimeSchedule` →
`airflow.timetables.assets.AssetOrTimeSchedule`
*
`airflow.providers.amazon.auth_manager.avp.entities.AvpEntities.DATASET`
→ `airflow.providers.amazon.auth_manager.avp.entities.AvpEntities.ASSET`
* `airflow.providers.amazon.aws.datasets.s3.create_dataset` →
`airflow.providers.amazon.aws.assets.s3.create_asset`
*
`airflow.providers.amazon.aws.datasets.s3.convert_dataset_to_openlineage`
→
`airflow.providers.amazon.aws.datasets.s3.convert_dataset_to_openlineage`
* `airflow.providers.amazon.aws.datasets.s3.sanitize_uri` →
`airflow.providers.amazon.aws.assets.s3.sanitize_uri`
*
`airflow.providers.common.io.datasets.file.convert_dataset_to_openlineage`
→ `airflow.providers.common.io.assets.file.convert_asset_to_openlineage`
* `airflow.providers.common.io.datasets.file.sanitize_uri` →
`airflow.providers.common.io.assets.file.sanitize_uri`
* `airflow.providers.common.io.datasets.file.create_dataset` →
`airflow.providers.common.io.assets.file.create_asset`
* `airflow.providers.google.datasets.bigquery.sanitize_uri` →
`airflow.providers.google.assets.bigquery.sanitize_uri`
* `airflow.providers.google.datasets.gcs.create_dataset` →
`airflow.providers.google.assets.gcs.create_asset`
* `airflow.providers.google.datasets.gcs.sanitize_uri` →
`airflow.providers.google.assets.gcs.sanitize_uri`
* `airflow.providers.google.datasets.gcs.convert_dataset_to_openlineage`
→ `airflow.providers.google.assets.gcs.convert_asset_to_openlineage`
*
`airflow.providers.fab.auth_manager.fab_auth_manager.is_authorized_dataset`
→
`airflow.providers.fab.auth_manager.fab_auth_manager.is_authorized_asset`
* `airflow.providers.openlineage.utils.utils.DatasetInfo` →
`airflow.providers.openlineage.utils.utils.AssetInfo`
* `airflow.providers.openlineage.utils.utils.translate_airflow_dataset`
→ `airflow.providers.openlineage.utils.utils.translate_airflow_asset`
* `airflow.providers.postgres.datasets.postgres.sanitize_uri` →
`airflow.providers.postgres.assets.postgres.sanitize_uri`
* `airflow.providers.mysql.datasets.mysql.sanitize_uri` →
`airflow.providers.mysql.assets.mysql.sanitize_uri`
* `airflow.providers.trino.datasets.trino.sanitize_uri` →
`airflow.providers.trino.assets.trino.sanitize_uri`

In additional to the newly added rules above, the message for
`airflow.contrib.*` and `airflow.subdag.*` has been extended,
`airflow.sensors.external_task.ExternalTaskSensorLink` error has been
fixed and the test fixture has been reorganized

## Test Plan

A test fixture is included in the PR.
2024-12-17 08:32:48 +01:00
Wei Lee
dfd7f38009
[airflow]: Import modules that has been moved to airflow providers (AIR303) (#14764)
## Summary

Many core Airflow features have been deprecated and moved to Airflow
Providers since users might need to install an additional package (e.g.,
`apache-airflow-provider-fab==1.0.0`); a separate rule (AIR303) is
created for this.

As some of the changes only relate to the module/package moved, instead
of listing out all the functions, variables, and classes in a module or
a package, it warns the user to import from the new path instead of the
specific name.

The following is the ones that has been moved to
`apache-airflow-provider-fab==1.0.0`

* module moved
* `airflow.api.auth.backend.basic_auth` →
`airflow.providers.fab.auth_manager.api.auth.backend.basic_auth`
* `airflow.api.auth.backend.kerberos_auth` →
`airflow.providers.fab.auth_manager.api.auth.backend.kerberos_auth`
* `airflow.auth.managers.fab.api.auth.backend.kerberos_auth` →
`airflow.providers.fab.auth_manager.api.auth.backend.kerberos_auth`
* `airflow.auth.managers.fab.security_manager.override` →
`airflow.providers.fab.auth_manager.security_manager.override`
* classes (e.g., functions, classes) moved
* `airflow.www.security.FabAirflowSecurityManagerOverride` →
`airflow.providers.fab.auth_manager.security_manager.override.FabAirflowSecurityManagerOverride`
* `airflow.auth.managers.fab.fab_auth_manager.FabAuthManager` →
`airflow.providers.fab.auth_manager.security_manager.FabAuthManager`

## Test Plan


A test fixture has been included for the rule.
2024-12-13 10:38:07 +01:00
Wei Lee
3ea14d7a74
[airflow]: extend removed args (AIR302) (#14765)
## Summary

Airflow 3.0 removes various deprecated functions, members, modules, and
other values. They have been deprecated in 2.x, but the removal causes
incompatibilities that we want to detect. This PR deprecates the
following names.

* in `DAG`
    * `sla_miss_callback` was removed
* in `airflow.operators.trigger_dagrun.TriggerDagRunOperator`
    * `execution_date` was removed
* in `airflow.operators.weekday.DayOfWeekSensor`,
`airflow.operators.datetime.BranchDateTimeOperator` and
`airflow.operators.weekday.BranchDayOfWeekOperator`
* `use_task_execution_day` was removed in favor of
`use_task_logical_date`

The full list of rules we will extend
https://github.com/apache/airflow/issues/44556

## Test Plan

<!-- How was it tested? -->
A test fixture is included in the PR.
2024-12-06 17:00:23 +01:00
Wei Lee
39623f8d40
[airflow]: extend removed names (AIR302) (#14804)
## Summary

Airflow 3.0 removes various deprecated functions, members, modules, and
other values. They have been deprecated in 2.x, but the removal causes
incompatibilities that we want to detect. This PR deprecates the
following names.
The full list of rules we will extend
https://github.com/apache/airflow/issues/44556


#### package
* `airflow.contrib.*` 

#### module
* `airflow.operators.subdag.*` 

#### class
* `airflow.sensors.external_task.ExternalTaskSensorLink` →
`airflow.sensors.external_task.ExternalDagLin`
* `airflow.operators.bash_operator.BashOperator` →
`airflow.operators.bash.BashOperator`
* `airflow.operators.branch_operator.BaseBranchOperator` →
`airflow.operators.branch.BaseBranchOperator`
* `airflow.operators.dummy.EmptyOperator` →
`airflow.operators.empty.EmptyOperator`
* `airflow.operators.dummy.DummyOperator` →
`airflow.operators.empty.EmptyOperator`
* `airflow.operators.dummy_operator.EmptyOperator` →
`airflow.operators.empty.EmptyOperator`
* `airflow.operators.dummy_operator.DummyOperator` →
`airflow.operators.empty.EmptyOperator`
* `airflow.operators.email_operator.EmailOperator` →
`airflow.operators.email.EmailOperator`
* `airflow.sensors.base_sensor_operator.BaseSensorOperator` →
`airflow.sensors.base.BaseSensorOperator`
* `airflow.sensors.date_time_sensor.DateTimeSensor` →
`airflow.sensors.date_time.DateTimeSensor`
* `airflow.sensors.external_task_sensor.ExternalTaskMarker` →
`airflow.sensors.external_task.ExternalTaskMarker`
* `airflow.sensors.external_task_sensor.ExternalTaskSensor` →
`airflow.sensors.external_task.ExternalTaskSensor`
* `airflow.sensors.external_task_sensor.ExternalTaskSensorLink` →
`airflow.sensors.external_task.ExternalTaskSensorLink`
* `airflow.sensors.time_delta_sensor.TimeDeltaSensor` →
`airflow.sensors.time_delta.TimeDeltaSensor`

#### function
* `airflow.utils.decorators.apply_defaults`
* `airflow.www.utils.get_sensitive_variables_fields` →
`airflow.utils.log.secrets_masker.get_sensitive_variables_fields`
* `airflow.www.utils.should_hide_value_for_key` →
`airflow.utils.log.secrets_masker.should_hide_value_for_key`
* `airflow.configuration.get` → `airflow.configuration.conf.get` 
* `airflow.configuration.getboolean` →
`airflow.configuration.conf.getboolean`
* `airflow.configuration.getfloat` →
`airflow.configuration.conf.getfloat`
* `airflow.configuration.getint` → `airflow.configuration.conf.getint` 
* `airflow.configuration.has_option` →
`airflow.configuration.conf.has_option`
* `airflow.configuration.remove_option` →
`airflow.configuration.conf.remove_option`
* `airflow.configuration.as_dict` → `airflow.configuration.conf.as_dict`
* `airflow.configuration.set` → `airflow.configuration.conf.set` 
* `airflow.secrets.local_filesystem.load_connections` →
`airflow.secrets.local_filesystem.load_connections_dict`
* `airflow.secrets.local_filesystem.get_connection` →
`airflow.secrets.local_filesystem.load_connections_dict`
* `airflow.utils.helpers.chain` → `airflow.models.baseoperator.chain` 
* `airflow.utils.helpers.cross_downstream` →
`airflow.models.baseoperator.cross_downstream`

#### attribute
* in `airflow.utils.trigger_rule.TriggerRule`
    * `DUMMY` 
    * `NONE_FAILED_OR_SKIPPED` 

#### constant / variable
* `airflow.PY\d\d`
2024-12-06 11:34:48 +01:00
Dhruv Manilawala
575deb5d4d
Check AIR001 from builtin or providers operators module (#14631)
## Summary

This PR makes changes to the `AIR001` rule as per
https://github.com/astral-sh/ruff/pull/14627#discussion_r1860212307.

Additionally,
* Avoid returning the `Diagnostic` and update the checker in the rule
logic for consistency
* Remove test case for different keyword position (I don't think it's
required here)

## Test Plan

Add test cases for multiple operators from various modules.
2024-12-04 13:30:47 +05:30
Wei Lee
edce559431
[airflow]: extend removed names (AIR302) (#14734) 2024-12-03 21:39:43 +00:00
Tzu-ping Chung
76d2e56501
[airflow] Avoid deprecated values (AIR302) (#14582) 2024-12-02 07:39:26 +00:00
Tzu-ping Chung
fbff4dec3a
[airflow] Avoid implicit DAG schedule (AIR301) (#14581) 2024-11-26 13:38:18 +01:00
Charlie Marsh
5849a75223
Rename ruff crate to ruff_linter (#7529) 2023-09-20 08:38:27 +02:00