airflow.providers.google.cloud.hooks.cloud_composer

Module Contents

Classes

CloudComposerHook

Hook for Google Cloud Composer APIs.

CloudComposerAsyncHook

Hook for Google Cloud Composer async APIs.
class airflow.providers.google.cloud.hooks.cloud_composer.CloudComposerHook(gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Google Cloud Composer APIs.

client_options[source]
get_environment_client()[source]

Retrieve client library object that allow access Environments service.

get_image_versions_client()[source]

Retrieve client library object that allow access Image Versions service.

wait_for_operation(operation, timeout=None)[source]

Wait for long-lasting operation to complete.

get_operation(operation_name)[source]
get_environment_name(project_id, region, environment_id)[source]
get_parent(project_id, region)[source]
create_environment(project_id, region, environment, retry=DEFAULT, timeout=None, metadata=())[source]

Create a new environment.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the service belongs to.

  • environment (google.cloud.orchestration.airflow.service_v1.types.Environment | dict) – The environment to create. This corresponds to the environment field on the request instance; if request is provided, this should not be set.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

delete_environment(project_id, region, environment_id, retry=DEFAULT, timeout=None, metadata=())[source]

Delete an environment.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the service belongs to.

  • environment_id (str) – Required. The ID of the Google Cloud environment that the service belongs to.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

get_environment(project_id, region, environment_id, retry=DEFAULT, timeout=None, metadata=())[source]

Get an existing environment.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the service belongs to.

  • environment_id (str) – Required. The ID of the Google Cloud environment that the service belongs to.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

list_environments(project_id, region, page_size=None, page_token=None, retry=DEFAULT, timeout=None, metadata=())[source]

List environments.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the service belongs to.

  • page_size (int | None) – The maximum number of environments to return.

  • page_token (str | None) – The next_page_token value returned from a previous List request, if any.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

update_environment(project_id, region, environment_id, environment, update_mask, retry=DEFAULT, timeout=None, metadata=())[source]

Update an environment.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the service belongs to.

  • environment_id (str) – Required. The ID of the Google Cloud environment that the service belongs to.

  • environment (google.cloud.orchestration.airflow.service_v1.types.Environment | dict) –

    A patch environment. Fields specified by the updateMask will be copied from the patch environment into the environment under update.

    This corresponds to the environment field on the request instance; if request is provided, this should not be set.

  • update_mask (dict | google.protobuf.field_mask_pb2.FieldMask) – Required. A comma-separated list of paths, relative to Environment, of fields to update. If a dict is provided, it must be of the same form as the protobuf message FieldMask

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

list_image_versions(project_id, region, page_size=None, page_token=None, include_past_releases=False, retry=DEFAULT, timeout=None, metadata=())[source]

List ImageVersions for provided location.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the service belongs to.

  • page_size (int | None) – The maximum number of environments to return.

  • page_token (str | None) – The next_page_token value returned from a previous List request, if any.

  • include_past_releases (bool) – Flag to include past releases

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

execute_airflow_command(project_id, region, environment_id, command, subcommand, parameters, retry=DEFAULT, timeout=None, metadata=())[source]

Execute Airflow command for provided Composer environment.

Parameters

  • project_id (str) – The ID of the Google Cloud project that the service belongs to.

  • region (str) – The ID of the Google Cloud region that the service belongs to.

  • environment_id (str) – The ID of the Google Cloud environment that the service belongs to.

  • command (str) – Airflow command.

  • subcommand (str) – Airflow subcommand.

  • parameters (collections.abc.MutableSequence[str]) – Parameters for the Airflow command/subcommand as an array of arguments. It may contain positional arguments like ["my-dag-id"], key-value parameters like ["--foo=bar"] or ["--foo","bar"], or other flags like ["-f"].

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

poll_airflow_command(project_id, region, environment_id, execution_id, pod, pod_namespace, next_line_number, retry=DEFAULT, timeout=None, metadata=())[source]

Poll Airflow command execution result for provided Composer environment.

Parameters

  • project_id (str) – The ID of the Google Cloud project that the service belongs to.

  • region (str) – The ID of the Google Cloud region that the service belongs to.

  • environment_id (str) – The ID of the Google Cloud environment that the service belongs to.

  • execution_id (str) – The unique ID of the command execution.

  • pod (str) – The name of the pod where the command is executed.

  • pod_namespace (str) – The namespace of the pod where the command is executed.

  • next_line_number (int) – Line number from which new logs should be fetched.

  • retry (google.api_core.retry.Retry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

wait_command_execution_result(project_id, region, environment_id, execution_cmd_info, retry=DEFAULT, timeout=None, metadata=(), poll_interval=10)[source]
class airflow.providers.google.cloud.hooks.cloud_composer.CloudComposerAsyncHook(gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Google Cloud Composer async APIs.

client_options[source]
get_environment_client()[source]

Retrieve client library object that allow access Environments service.

get_environment_name(project_id, region, environment_id)[source]
get_parent(project_id, region)[source]
async get_operation(operation_name)[source]
async create_environment(project_id, region, environment, retry=DEFAULT, timeout=None, metadata=())[source]

Create a new environment.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the service belongs to.

  • environment (google.cloud.orchestration.airflow.service_v1.types.Environment | dict) – The environment to create. This corresponds to the environment field on the request instance; if request is provided, this should not be set.

  • retry (google.api_core.retry_async.AsyncRetry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

async delete_environment(project_id, region, environment_id, retry=DEFAULT, timeout=None, metadata=())[source]

Delete an environment.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the service belongs to.

  • environment_id (str) – Required. The ID of the Google Cloud environment that the service belongs to.

  • retry (google.api_core.retry_async.AsyncRetry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

async update_environment(project_id, region, environment_id, environment, update_mask, retry=DEFAULT, timeout=None, metadata=())[source]

Update an environment.

Parameters

  • project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.

  • region (str) – Required. The ID of the Google Cloud region that the service belongs to.

  • environment_id (str) – Required. The ID of the Google Cloud environment that the service belongs to.

  • environment (google.cloud.orchestration.airflow.service_v1.types.Environment | dict) –

    A patch environment. Fields specified by the updateMask will be copied from the patch environment into the environment under update.

    This corresponds to the environment field on the request instance; if request is provided, this should not be set.

  • update_mask (dict | google.protobuf.field_mask_pb2.FieldMask) – Required. A comma-separated list of paths, relative to Environment, of fields to update. If a dict is provided, it must be of the same form as the protobuf message FieldMask

  • retry (google.api_core.retry_async.AsyncRetry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

async execute_airflow_command(project_id, region, environment_id, command, subcommand, parameters, retry=DEFAULT, timeout=None, metadata=())[source]

Execute Airflow command for provided Composer environment.

Parameters

  • project_id (str) – The ID of the Google Cloud project that the service belongs to.

  • region (str) – The ID of the Google Cloud region that the service belongs to.

  • environment_id (str) – The ID of the Google Cloud environment that the service belongs to.

  • command (str) – Airflow command.

  • subcommand (str) – Airflow subcommand.

  • parameters (collections.abc.MutableSequence[str]) – Parameters for the Airflow command/subcommand as an array of arguments. It may contain positional arguments like ["my-dag-id"], key-value parameters like ["--foo=bar"] or ["--foo","bar"], or other flags like ["-f"].

  • retry (google.api_core.retry_async.AsyncRetry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

async poll_airflow_command(project_id, region, environment_id, execution_id, pod, pod_namespace, next_line_number, retry=DEFAULT, timeout=None, metadata=())[source]

Poll Airflow command execution result for provided Composer environment.

Parameters

  • project_id (str) – The ID of the Google Cloud project that the service belongs to.

  • region (str) – The ID of the Google Cloud region that the service belongs to.

  • environment_id (str) – The ID of the Google Cloud environment that the service belongs to.

  • execution_id (str) – The unique ID of the command execution.

  • pod (str) – The name of the pod where the command is executed.

  • pod_namespace (str) – The namespace of the pod where the command is executed.

  • next_line_number (int) – Line number from which new logs should be fetched.

  • retry (google.api_core.retry_async.AsyncRetry | google.api_core.gapic_v1.method._MethodDefault) – Designation of what errors, if any, should be retried.

  • timeout (float | None) – The timeout for this request.

  • metadata (collections.abc.Sequence[tuple[str, str]]) – Strings which should be sent along with the request as metadata.

async wait_command_execution_result(project_id, region, environment_id, execution_cmd_info, retry=DEFAULT, timeout=None, metadata=(), poll_interval=10)[source]

Was this entry helpful?