airflow.providers.google.cloud.hooks.vertex_ai.ray¶

This module contains a Google Cloud Vertex AI hook.

Classes¶

RayHook

Hook for Google Cloud Vertex AI Ray APIs.

Module Contents¶

class airflow.providers.google.cloud.hooks.vertex_ai.ray.RayHook(gcp_conn_id='google_cloud_default', impersonation_chain=None, **kwargs)[source]¶

Bases: airflow.providers.google.common.hooks.base_google.GoogleBaseHook

Hook for Google Cloud Vertex AI Ray APIs.

extract_cluster_id(cluster_path)[source]¶

Extract cluster_id from cluster_path.

serialize_cluster_obj(cluster_obj)[source]¶

Serialize Cluster dataclass to dict.

create_ray_cluster(project_id, location, head_node_type=resources.Resources(), python_version='3.10', ray_version='2.33', network=None, service_account=None, cluster_name=None, worker_node_types=None, custom_images=None, enable_metrics_collection=True, enable_logging=True, psc_interface_config=None, reserved_ip_ranges=None, labels=None)[source]¶

Create a Ray cluster on the Vertex AI.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud location that the service belongs to.
head_node_type (google.cloud.aiplatform.vertex_ray.util.resources.Resources) – The head node resource. Resources.node_count must be 1. If not set, default value of Resources() class will be used.
python_version (str) – Python version for the ray cluster.
ray_version (str) – Ray version for the ray cluster. Default is 2.33.0.
network (str | None) – Virtual private cloud (VPC) network. For Ray Client, VPC peering is required to connect to the Ray Cluster managed in the Vertex API service. For Ray Job API, VPC network is not required because Ray Cluster connection can be accessed through dashboard address.
service_account (str | None) – Service account to be used for running Ray programs on the cluster.
cluster_name (str | None) – This value may be up to 63 characters, and valid characters are [a-z0-9_-]. The first character cannot be a number or hyphen.
worker_node_types (list[google.cloud.aiplatform.vertex_ray.util.resources.Resources] | None) – The list of Resources of the worker nodes. The same Resources object should not appear multiple times in the list.
custom_images (google.cloud.aiplatform.vertex_ray.util.resources.NodeImages | None) – The NodeImages which specifies head node and worker nodes images. All the workers will share the same image. If each Resource has a specific custom image, use Resources.custom_image for head/worker_node_type(s). Note that configuring Resources.custom_image will override custom_images here. Allowlist only.
enable_metrics_collection (bool) – Enable Ray metrics collection for visualization.
enable_logging (bool) – Enable exporting Ray logs to Cloud Logging.
psc_interface_config (google.cloud.aiplatform.vertex_ray.util.resources.PscIConfig | None) – PSC-I config.
reserved_ip_ranges (list[str] | None) – A list of names for the reserved IP ranges under the VPC network that can be used for this cluster. If set, we will deploy the cluster within the provided IP ranges. Otherwise, the cluster is deployed to any IP ranges under the provided VPC network. Example: [“vertex-ai-ip-range”].
labels (dict[str, str] | None) – The labels with user-defined metadata to organize Ray cluster. Label keys and values can be no longer than 64 characters (Unicode codepoints), can only contain lowercase letters, numeric characters, underscores and dashes. International characters are allowed. See https://goo.gl/xmQnxf for more information and examples of labels.

list_ray_clusters(project_id, location)[source]¶

List Ray clusters under the currently authenticated project.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud location that the service belongs to.

get_ray_cluster(project_id, location, cluster_id)[source]¶

Get Ray cluster.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud location that the service belongs to.
cluster_id (str) – Cluster resource ID.

update_ray_cluster(project_id, location, cluster_id, worker_node_types)[source]¶

Update Ray cluster (currently support resizing node counts for worker nodes).

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud location that the service belongs to.
cluster_id (str) – Cluster resource ID.
worker_node_types (list[google.cloud.aiplatform.vertex_ray.util.resources.Resources]) – The list of Resources of the resized worker nodes. The same Resources object should not appear multiple times in the list.

delete_ray_cluster(project_id, location, cluster_id)[source]¶

Delete Ray cluster.

Parameters:

project_id (str) – Required. The ID of the Google Cloud project that the service belongs to.
location (str) – Required. The ID of the Google Cloud location that the service belongs to.
cluster_id (str) – Cluster resource ID.