airflow.providers.google.cloud.transfers.http_to_gcs¶
This module contains operator to move data from HTTP endpoint to GCS.
Classes¶
Calls an endpoint on an HTTP system to execute an action and store the result in GCS. |
Module Contents¶
- class airflow.providers.google.cloud.transfers.http_to_gcs.HttpToGCSOperator(*, endpoint=None, method='GET', data=None, headers=None, extra_options=None, http_conn_id='http_default', log_response=False, auth_type=None, tcp_keep_alive=True, tcp_keep_alive_idle=120, tcp_keep_alive_count=20, tcp_keep_alive_interval=30, gcp_conn_id='google_cloud_default', impersonation_chain=None, bucket_name, object_name, mime_type=None, gzip=False, encoding=None, chunk_size=None, timeout=None, num_max_attempts=3, metadata=None, cache_control=None, user_project=None, **kwargs)[source]¶
Bases:
airflow.models.BaseOperatorCalls an endpoint on an HTTP system to execute an action and store the result in GCS.
- Parameters:
http_conn_id (str) – The http connection to run the operator against
endpoint (str | None) – The relative part of the full url. (templated)
method (str) – The HTTP method to use, default = “POST”
data (Any) – The data to pass. POST-data in POST/PUT and params in the URL for a GET request. (templated)
headers (dict[str, str] | None) – The HTTP headers to be added to the GET request
response_check – A check against the ‘requests’ response object. The callable takes the response object as the first positional argument and optionally any number of keyword arguments available in the context dictionary. It should return True for ‘pass’ and False otherwise.
response_filter – A function allowing you to manipulate the response text. e.g response_filter=lambda response: json.loads(response.text). The callable takes the response object as the first positional argument and optionally any number of keyword arguments available in the context dictionary.
extra_options (dict[str, Any] | None) – Extra options for the ‘requests’ library, see the ‘requests’ documentation (options to modify timeout, ssl, etc.)
log_response (bool) – Log the response (default: False)
auth_type (type[requests.auth.AuthBase] | None) – The auth type for the service
tcp_keep_alive (bool) – Enable TCP Keep Alive for the connection.
tcp_keep_alive_idle (int) – The TCP Keep Alive Idle parameter (corresponds to
socket.TCP_KEEPIDLE).tcp_keep_alive_count (int) – The TCP Keep Alive count parameter (corresponds to
socket.TCP_KEEPCNT)tcp_keep_alive_interval (int) – The TCP Keep Alive interval parameter (corresponds to
socket.TCP_KEEPINTVL)gcp_conn_id (str) – The connection ID to use when fetching connection info.
impersonation_chain (str | collections.abc.Sequence[str] | None) – Optional service account to impersonate using short-term credentials, or chained list of accounts required to get the access_token of the last account in the list, which will be impersonated in the request. If set as a string, the account must grant the originating account the Service Account Token Creator IAM role. If set as a sequence, the identities from the list must grant Service Account Token Creator IAM role to the directly preceding identity, with first account from the list granting this role to the originating account.
bucket_name (str) – The bucket to upload to.
object_name (str) – The object name to set when uploading the file.
mime_type (str | None) – The file mime type set when uploading the file.
gzip (bool) – Option to compress local file or file data for upload
encoding (str | None) – bytes encoding for file data if provided as string
chunk_size (int | None) – Blob chunk size.
timeout (int | None) – Request timeout in seconds.
num_max_attempts (int) – Number of attempts to try to upload the file.
metadata (dict | None) – The metadata to be uploaded with the file.
cache_contro – Cache-Control metadata field.
user_project (str | None) – The identifier of the Google Cloud project to bill for the request. Required for Requester Pays buckets.
- template_fields: collections.abc.Sequence[str] = ('http_conn_id', 'endpoint', 'data', 'headers', 'gcp_conn_id', 'bucket_name', 'object_name')[source]¶
- template_ext: collections.abc.Sequence[str] = ()[source]¶
- property http_hook: airflow.providers.http.hooks.http.HttpHook[source]¶
Create and return an HttpHook.
- property gcs_hook: airflow.providers.google.cloud.hooks.gcs.GCSHook[source]¶
Create and return an GCSHook.