• Home
• airflow.providers.amazon
• airflow.providers.amazon.aws
• airflow.providers.amazon.aws.sensors
• airflow.providers.amazon.aws.sensors.comprehend

airflow.providers.amazon.aws.sensors.comprehend

Module Contents

Classes

ComprehendBaseSensor	General sensor behavior for Amazon Comprehend.
ComprehendStartPiiEntitiesDetectionJobCompletedSensor	Poll the state of the pii entities detection job until it reaches a completed state; fails if the job fails.
ComprehendCreateDocumentClassifierCompletedSensor	Poll the state of the document classifier until it reaches a completed state; fails if the job fails.

class airflow.providers.amazon.aws.sensors.comprehend.ComprehendBaseSensor(deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)

Bases: airflow.providers.amazon.aws.sensors.base_aws.AwsBaseSensor[airflow.providers.amazon.aws.hooks.comprehend.ComprehendHook]

General sensor behavior for Amazon Comprehend.

Subclasses must implement following methods:

• get_state()

Subclasses must set the following fields:

• INTERMEDIATE_STATES

• FAILURE_STATES

• SUCCESS_STATES

• FAILURE_MESSAGE

Parameters

deferrable (bool) – If True, the sensor will operate in deferrable mode. This mode requires aiobotocore module to be installed. (default: False, but can be overridden in config file by setting default_deferrable to True)

aws_hook_class

INTERMEDIATE_STATES: tuple[str, Ellipsis] = ()

FAILURE_STATES: tuple[str, Ellipsis] = ()

SUCCESS_STATES: tuple[str, Ellipsis] = ()

FAILURE_MESSAGE = ''

ui_color = '#66c3ff'

poke(context, **kwargs)

Override when deriving this class.

abstract get_state()

Implement in subclasses.

class airflow.providers.amazon.aws.sensors.comprehend.ComprehendStartPiiEntitiesDetectionJobCompletedSensor(*, job_id, max_retries=75, poke_interval=120, **kwargs)

Bases: ComprehendBaseSensor

Poll the state of the pii entities detection job until it reaches a completed state; fails if the job fails.

See also

For more information on how to use this sensor, take a look at the guide: Wait for an Amazon Comprehend Start PII Entities Detection Job

Parameters

• job_id (str) – The id of the Comprehend pii entities detection job.

• deferrable – If True, the sensor will operate in deferrable mode. This mode requires aiobotocore module to be installed. (default: False, but can be overridden in config file by setting default_deferrable to True)

• poke_interval (int) – Polling period in seconds to check for the status of the job. (default: 120)

• max_retries (int) – Number of times before returning the current state. (default: 75)

• aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).

• region_name – AWS region_name. If not specified then the default boto3 behaviour is used.

• verify – Whether to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html

• botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

INTERMEDIATE_STATES: tuple[str, Ellipsis] = ('IN_PROGRESS',)

FAILURE_STATES: tuple[str, Ellipsis] = ('FAILED', 'STOP_REQUESTED', 'STOPPED')

SUCCESS_STATES: tuple[str, Ellipsis] = ('COMPLETED',)

FAILURE_MESSAGE = 'Comprehend start pii entities detection job sensor failed.'

template_fields: Sequence[str]

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

get_state()

Implement in subclasses.

class airflow.providers.amazon.aws.sensors.comprehend.ComprehendCreateDocumentClassifierCompletedSensor(*, document_classifier_arn, fail_on_warnings=False, max_retries=75, poke_interval=120, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), aws_conn_id='aws_default', **kwargs)

Bases: airflow.providers.amazon.aws.sensors.base_aws.AwsBaseSensor[airflow.providers.amazon.aws.hooks.comprehend.ComprehendHook]

Poll the state of the document classifier until it reaches a completed state; fails if the job fails.

See also

For more information on how to use this sensor, take a look at the guide: Wait for an Amazon Comprehend Document Classifier

Parameters

• document_classifier_arn (str) – The arn of the Comprehend document classifier.

• fail_on_warnings (bool) – If set to True, the document classifier training job will throw an error when the status is TRAINED_WITH_WARNING. (default False)

• deferrable (bool) – If True, the sensor will operate in deferrable mode. This mode requires aiobotocore module to be installed. (default: False, but can be overridden in config file by setting default_deferrable to True)

• poke_interval (int) – Polling period in seconds to check for the status of the job. (default: 120)

• max_retries (int) – Number of times before returning the current state. (default: 75)

• aws_conn_id (str | None) – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).

• region_name – AWS region_name. If not specified then the default boto3 behaviour is used.

• verify – Whether to verify SSL certificates. See: https://boto3.amazonaws.com/v1/documentation/api/latest/reference/core/session.html

• botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

aws_hook_class

INTERMEDIATE_STATES: tuple[str, Ellipsis] = ('SUBMITTED', 'TRAINING')

FAILURE_STATES: tuple[str, Ellipsis] = ('DELETING', 'STOP_REQUESTED', 'STOPPED', 'IN_ERROR')

SUCCESS_STATES: tuple[str, Ellipsis] = ('TRAINED', 'TRAINED_WITH_WARNING')

FAILURE_MESSAGE = 'Comprehend document classifier failed.'

template_fields: Sequence[str]

execute(context)

Derive when creating an operator.

Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

poke(context, **kwargs)

Override when deriving this class.

Previous Next

Was this entry helpful?