airflow.providers.amazon.aws.operators.neptune_analytics¶

Classes¶

`NeptuneCreateGraphOperator`	Creates an empty Amazon Neptune Graph database.
`NeptuneCreatePrivateGraphEndpointOperator`	Creates a Neptune Graph private endpoint.
`NeptuneDeletePrivateGraphEndpointOperator`	Deletes a Neptune Graph private endpoint.
`NeptuneDeleteGraphOperator`	Deletes an Amazon Neptune Graph database.
`NeptuneCreateGraphWithImportOperator`	Creates a Neptune Graph and imports data into it.
`NeptuneStartImportTaskOperator`	Starts a bulk data import task to load data into an empty Neptune graph.
`NeptuneCancelImportTaskOperator`	Cancels an active Neptune Graph import task.

Module Contents¶

class airflow.providers.amazon.aws.operators.neptune_analytics.NeptuneCreateGraphOperator(graph_name, vector_search_config, provisioned_memory, public_connectivity=None, replica_count=None, deletion_protection=False, kms_key_id=None, tags=None, wait_for_completion=True, waiter_delay=30, waiter_max_attempts=60, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.neptune_analytics.NeptuneAnalyticsHook]

Creates an empty Amazon Neptune Graph database.

Neptune Analytics is a memory-optimized graph database engine for analytics. With Neptune Analytics, you can get insights and find trends by processing large amounts of graph data in seconds.

See also

For more information on how to use this operator, take a look at the guide: Create a new Neptune Graph

Parameters:

graph_name (str) – Name of Neptune graph to create
vector_search_config (dict) – Specifies the number of dimensions for vector embeddings that will be loaded into the graph.
provisioned_memory (int) – The provisioned memory-optimized Neptune Capacity Units (m-NCUs) to use for the graph.
public_connectivity (bool | None) – Specifies whether or not the graph can be reachable over the internet.
replica_count (int | None) – The number of replicas in other AZs.
deletion_protection (bool) – Indicates whether or not to enable deletion protection on the graph. The graph can’t be deleted when deletion protection is enabled.
kms_key_id (str | None) – Specifies a KMS key to use to encrypt data in the new graph.
tags (dict | None) – Specifies metadata tags to add to the graph.
wait_for_completion (bool) – Whether to wait for the graph to start. (default: True)
deferrable (bool) – If True, the operator will wait asynchronously for the graph to start. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
waiter_delay (int) – Time in seconds to wait between status checks.
waiter_max_attempts (int) – Maximum number of attempts to check for job completion.
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

Returns:

dictionary with Neptune graph id

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

template_fields_renderers[source]¶

operator_extra_links[source]¶

graph_name[source]¶

vector_search_config[source]¶

replica_count = None[source]¶

provisioned_memory[source]¶

public_connectivity = None[source]¶

deletion_protect = False[source]¶

kms_key_id = None[source]¶

tags = None[source]¶

wait_for_completion = True[source]¶

deferrable[source]¶

waiter_delay = 30[source]¶

waiter_max_attempts = 60[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]¶

class airflow.providers.amazon.aws.operators.neptune_analytics.NeptuneCreatePrivateGraphEndpointOperator(graph_identifier, vpc_id=None, subnet_ids=None, vpc_security_group_ids=None, wait_for_completion=True, waiter_delay=30, waiter_max_attempts=60, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.neptune_analytics.NeptuneAnalyticsHook]

Creates a Neptune Graph private endpoint.

See also

For more information on how to use this operator, take a look at the guide: Create a Neptune Graph private endpoint

Parameters:

graph_identifier (str) – Neptune Graph id
vpc_id (str | None) – VPC to create endpoint in
subnet_ids (list[str] | None) – Subnets in which private graph endpoint ENIs are created
vpc_security_group_ids (list[str] | None) – Security groups to be attached to the private graph endpoint
wait_for_completion (bool) – Whether to wait for the endpoint to be available. (default: True)
deferrable (bool) – If True, the operator will wait asynchronously for the endpoint to become available. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
waiter_delay (int) – Time in seconds to wait between status checks.
waiter_max_attempts (int) – Maximum number of attempts to check for job completion.
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

Returns:

dictionary with Neptune graph id

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

operator_extra_links[source]¶

graph_identifier[source]¶

vpc_id = None[source]¶

subnet_ids = None[source]¶

vpc_security_group_ids = None[source]¶

wait_for_completion = True[source]¶

deferrable[source]¶

waiter_delay = 30[source]¶

waiter_max_attempts = 60[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None, vpc_id='')[source]¶

class airflow.providers.amazon.aws.operators.neptune_analytics.NeptuneDeletePrivateGraphEndpointOperator(graph_identifier, vpc_id, wait_for_completion=True, waiter_delay=30, waiter_max_attempts=60, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.neptune_analytics.NeptuneAnalyticsHook]

Deletes a Neptune Graph private endpoint.

See also

For more information on how to use this operator, take a look at the guide: Delete a Neptune Graph private endpoint

Parameters:

graph_identifier (str) – Neptune Graph id
vpc_id (str) – VPC where endpoint resides
wait_for_completion (bool) – Whether to wait for the endpoint to be deleted. (default: True)
deferrable (bool) – If True, the operator will wait asynchronously for the endpoint to be deleted. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
waiter_delay (int) – Time in seconds to wait between status checks.
waiter_max_attempts (int) – Maximum number of attempts to check for job completion.
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

Returns:

dictionary with Neptune graph id

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

graph_identifier[source]¶

vpc_id[source]¶

wait_for_completion = True[source]¶

deferrable[source]¶

waiter_delay = 30[source]¶

waiter_max_attempts = 60[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]¶

class airflow.providers.amazon.aws.operators.neptune_analytics.NeptuneDeleteGraphOperator(graph_id, skip_snapshot, wait_for_completion=True, waiter_delay=30, waiter_max_attempts=60, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.neptune_analytics.NeptuneAnalyticsHook]

Deletes an Amazon Neptune Graph database.

Neptune Analytics is a memory-optimized graph database engine for analytics. With Neptune Analytics, you can get insights and find trends by processing large amounts of graph data in seconds.

See also

For more information on how to use this operator, take a look at the guide: Delete a Neptune Graph

Parameters:

graph_id (str) – Name of Neptune graph to delete
skip_snapshot (bool) – Determines whether a final graph snapshot is created before the graph is deleted. If true is specified, no graph snapshot is created. If false is specified, a graph snapshot is created before the graph is deleted.
wait_for_completion (bool) – Whether to wait for the graph to delete. (default: True)
deferrable (bool) – If True, the operator will wait asynchronously for the graph to be deleted. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
waiter_delay (int) – Time in seconds to wait between status checks.
waiter_max_attempts (int) – Maximum number of attempts to check for job completion.
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

Returns:

dictionary with Neptune graph id

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

graph_id[source]¶

skip_snapshot[source]¶

wait_for_completion = True[source]¶

deferrable[source]¶

waiter_delay = 30[source]¶

waiter_max_attempts = 60[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]¶

class airflow.providers.amazon.aws.operators.neptune_analytics.NeptuneCreateGraphWithImportOperator(graph_name, vector_search_config, source, role_arn, blank_node_handling=None, parquet_type=None, format=None, min_provisioned_memory=None, max_provisioned_memory=None, fail_on_error=None, public_connectivity=None, replica_count=None, deletion_protection=None, kms_key_id=None, tags=None, import_options=None, wait_for_completion=True, waiter_delay=30, waiter_max_attempts=60, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.neptune_analytics.NeptuneAnalyticsHook]

Creates a Neptune Graph and imports data into it.

Neptune Analytics is a memory-optimized graph database engine for analytics. With Neptune Analytics, you can get insights and find trends by processing large amounts of graph data in seconds.

See also

For more information on how to use this operator, take a look at the guide: Create a Neptune Graph with a data import task

Parameters:

graph_name (str) – Name of Neptune graph to create
vector_search_config (dict) – Specifies the number of dimensions for vector embeddings that will be loaded into the graph.
source (str) – The source from which to import data. Can be an S3 URI or Neptune database snapshot.
role_arn (str) – The ARN of the IAM role that Neptune Analytics can assume to access the data source.
blank_node_handling (str | None) – The method to handle blank nodes in the dataset. Options include ‘convertToIri’ or other handling strategies.
parquet_type (str | None) – The type of Parquet files in the data source (if applicable).
format (str | None) – The format of the data to be imported (e.g., ‘csv’, ‘opencypher’, ‘ntriples’, ‘nquads’, ‘rdfxml’, ‘turtle’).
min_provisioned_memory (int | None) – The minimum provisioned memory for the graph in GBs.
max_provisioned_memory (int | None) – The maximum provisioned memory for the graph in GBs.
fail_on_error (bool | None) – If True, the import will fail if any errors are encountered. If False, the import will continue despite errors.
public_connectivity (bool | None) – Specifies whether or not the graph can be reachable over the internet.
replica_count (int | None) – The number of replicas in other AZs.
deletion_protection (bool | None) – Indicates whether or not to enable deletion protection on the graph. The graph can’t be deleted when deletion protection is enabled. (default: False)
kms_key_id (str | None) – Specifies a KMS key to use to encrypt data in the new graph.
tags (dict | None) – Specifies metadata tags to add to the graph.
import_options (dict | None) – Contains options for controlling the import process.
wait_for_completion (bool) – Whether to wait for the graph to be created and data imported. (default: True)
waiter_delay (int) – Time in seconds to wait between status checks. (default: 30)
waiter_max_attempts (int) – Maximum number of attempts to check for job completion. (default: 60)
deferrable (bool) – If True, the operator will wait asynchronously for the graph to be created and data imported. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

Returns:

dictionary with Neptune graph id

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

template_fields_renderers[source]¶

operator_extra_links[source]¶

graph_name[source]¶

vector_search_config[source]¶

source[source]¶

role_arn[source]¶

blank_node_handling = None[source]¶

parquet_type = None[source]¶

format = None[source]¶

min_provisioned_memory = None[source]¶

max_provisioned_memory = None[source]¶

fail_on_error = None[source]¶

public_connectivity = None[source]¶

replica_count = None[source]¶

deletion_protect = None[source]¶

kms_key_id = None[source]¶

tags = None[source]¶

import_options = None[source]¶

wait_for_completion = True[source]¶

deferrable[source]¶

waiter_delay = 30[source]¶

waiter_max_attempts = 60[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

defer_wait_for_task(context, event=None, import_task_id=None)[source]¶

Defers for import task completion.

execute_complete(context, event=None, graph_id=None)[source]¶

class airflow.providers.amazon.aws.operators.neptune_analytics.NeptuneStartImportTaskOperator(graph_identifier, role_arn, source, blank_node_handling=None, fail_on_error=True, format=None, import_options=None, parquet_type='COLUMNAR', wait_for_completion=True, waiter_delay=30, waiter_max_attempts=60, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.neptune_analytics.NeptuneAnalyticsHook]

Starts a bulk data import task to load data into an empty Neptune graph.

See also

For more information on how to use this operator, take a look at the guide: Import data into an existing Neptune Graph

Parameters:

graph_identifier (str) – Graph Id of target Neptune Graph
role_arn (str) – IAM role ARN granting access to source data
source (str) – URL identifying the source data location.
blank_node_handling (str | None) – Method to handle blank nodes in dataset.
fail_on_error (bool) – If set to true, the task halts when an import error is encountered. If set to false, the task skips the data that caused the error and continues if possible.
format (str | None) – Specifies the format of the Amazon S3 data to be imported.
import_options (dict | None) – Options on how to perform an import
parquet_type (str | None) – Parquet type of import task
wait_for_completion (bool) – Whether to wait for the endpoint to be available. (default: True)
deferrable (bool) – If True, the operator will wait asynchronously for the endpoint to become available. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
waiter_delay (int) – Time in seconds to wait between status checks.
waiter_max_attempts (int) – Maximum number of attempts to check for job completion.
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

Returns:

dictionary with Neptune graph id

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

template_fields_renderers[source]¶

operator_extra_links[source]¶

graph_identifier[source]¶

role_arn[source]¶

source[source]¶

blank_node_handling = None[source]¶

fail_on_error = True[source]¶

format = None[source]¶

import_options = None[source]¶

parquet_type = 'COLUMNAR'[source]¶

wait_for_completion = True[source]¶

waiter_delay = 30[source]¶

waiter_max_attempts = 60[source]¶

deferrable[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]¶

class airflow.providers.amazon.aws.operators.neptune_analytics.NeptuneCancelImportTaskOperator(import_task_id, wait_for_completion=True, waiter_delay=30, waiter_max_attempts=60, deferrable=conf.getboolean('operators', 'default_deferrable', fallback=False), **kwargs)[source]¶

Bases: airflow.providers.amazon.aws.operators.base_aws.AwsBaseOperator[airflow.providers.amazon.aws.hooks.neptune_analytics.NeptuneAnalyticsHook]

Cancels an active Neptune Graph import task.

See also

For more information on how to use this operator, take a look at the guide: Cancel a running import task

Parameters:

import_task_id (str) – Neptune Graph import task id to cancel.
wait_for_completion (bool) – Whether to wait for the task to be cancelled. If the task is already in a completed state, the operator will end successfully. (default: True)
deferrable (bool) – If True, the operator will wait asynchronously for the task to be cancelled. This implies waiting for completion. This mode requires aiobotocore module to be installed. (default: False)
waiter_delay (int) – Time in seconds to wait between status checks.
waiter_max_attempts (int) – Maximum number of attempts to check for job completion.
aws_conn_id – The Airflow connection used for AWS credentials. If this is None or empty then the default boto3 behaviour is used. If running Airflow in a distributed manner and aws_conn_id is None or empty, then default boto3 configuration would be used (and must be maintained on each worker node).
region_name – AWS region_name. If not specified then the default boto3 behaviour is used.
botocore_config – Configuration dictionary (key-values) for botocore client. See: https://botocore.amazonaws.com/v1/documentation/api/latest/reference/config.html

Returns:

dictionary with Neptune graph id

aws_hook_class[source]¶

template_fields: collections.abc.Sequence[str][source]¶

import_task_id[source]¶

wait_for_completion = True[source]¶

waiter_delay = 30[source]¶

waiter_max_attempts = 60[source]¶

deferrable[source]¶

execute(context)[source]¶

Derive when creating an operator.

The main method to execute the task. Context is the same dictionary used as when rendering jinja templates.

Refer to get_template_context for more context.

execute_complete(context, event=None)[source]¶