ASF: Apache Software Foundation¶

Airflow supports various software created by Apache Software Foundation.

Software¶

These integrations allow you to perform various operations within software developed by Apache Software Foundation.

Apache Beam¶

Operators:: airflow.providers.apache.beam.operators.beam.
Hooks:: airflow.providers.apache.beam.hooks.beam.
Guides:: Apache Beam Operators.
Provider:: apache-airflow-providers-apache-beam
Product documentation:: Apache Beam

Apache Cassandra¶

Hooks:: airflow.providers.apache.cassandra.hooks.cassandra.
Sensors:: airflow.providers.apache.cassandra.sensors.record, airflow.providers.apache.cassandra.sensors.table.
Guides:: Apache Cassandra Operators.
Provider:: apache-airflow-providers-apache-cassandra
Product documentation:: Apache Cassandra

Apache Drill¶

Hooks:: airflow.providers.apache.drill.hooks.drill.
Guides:: Connect to Apache Drill via SQLExecuteQueryOperator.
Provider:: apache-airflow-providers-apache-drill
Product documentation:: Apache Drill

Apache Druid¶

Operators:: airflow.providers.apache.druid.operators.druid.
Hooks:: airflow.providers.apache.druid.hooks.druid.
Guides:: Apache Druid Operators.
Provider:: apache-airflow-providers-apache-druid
Product documentation:: Apache Druid

Apache Flink¶

Operators:: airflow.providers.apache.flink.operators.flink_kubernetes.
Sensors:: airflow.providers.apache.flink.sensors.flink_kubernetes.
Guides:: Apache Flink Operators.
Provider:: apache-airflow-providers-apache-flink
Product documentation:: Apache Flink

Apache Hive¶

Operators:: airflow.providers.apache.hive.operators.hive, airflow.providers.apache.hive.operators.hive_stats.
Hooks:: airflow.providers.apache.hive.hooks.hive.
Sensors:: airflow.providers.apache.hive.sensors.hive_partition, airflow.providers.apache.hive.sensors.metastore_partition, airflow.providers.apache.hive.sensors.named_hive_partition.
Guides:: Apache Hive Operators.
Provider:: apache-airflow-providers-apache-hive
Product documentation:: Apache Hive

Apache Impala¶

Hooks:: airflow.providers.apache.impala.hooks.impala.
Provider:: apache-airflow-providers-apache-impala
Product documentation:: Apache Impala

Apache Kafka¶

Operators:: airflow.providers.apache.kafka.operators.consume, airflow.providers.apache.kafka.operators.produce.
Hooks:: airflow.providers.apache.kafka.hooks.base, airflow.providers.apache.kafka.hooks.client, airflow.providers.apache.kafka.hooks.consume, airflow.providers.apache.kafka.hooks.produce.
Sensors:: airflow.providers.apache.kafka.sensors.kafka.
Provider:: apache-airflow-providers-apache-kafka
Product documentation:: Apache Kafka

Apache Kylin¶

Operators:: airflow.providers.apache.kylin.operators.kylin_cube.
Hooks:: airflow.providers.apache.kylin.hooks.kylin.
Provider:: apache-airflow-providers-apache-kylin
Product documentation:: Apache Kylin

Apache Livy¶

Operators:: airflow.providers.apache.livy.operators.livy.
Hooks:: airflow.providers.apache.livy.hooks.livy.
Sensors:: airflow.providers.apache.livy.sensors.livy.
Guides:: Apache Livy Operators.
Provider:: apache-airflow-providers-apache-livy
Product documentation:: Apache Livy

Apache Pig¶

Operators:: airflow.providers.apache.pig.operators.pig.
Hooks:: airflow.providers.apache.pig.hooks.pig.
Guides:: Apache Pig Operators.
Provider:: apache-airflow-providers-apache-pig
Product documentation:: Apache Pig

Apache Pinot¶

Hooks:: airflow.providers.apache.pinot.hooks.pinot.
Guides:: Apache Pinot Hooks.
Provider:: apache-airflow-providers-apache-pinot
Product documentation:: Apache Pinot

Apache Spark¶

Operators:: airflow.providers.apache.spark.operators.spark_jdbc, airflow.providers.apache.spark.operators.spark_sql, airflow.providers.apache.spark.operators.spark_submit.
Hooks:: airflow.providers.apache.spark.hooks.spark_connect, airflow.providers.apache.spark.hooks.spark_jdbc, airflow.providers.apache.spark.hooks.spark_jdbc_script, airflow.providers.apache.spark.hooks.spark_sql, airflow.providers.apache.spark.hooks.spark_submit.
Guides:: Apache Spark Operators.
Provider:: apache-airflow-providers-apache-spark
Product documentation:: Apache Spark

Standard¶

Operators:: airflow.providers.standard.operators.datetime, airflow.providers.standard.operators.weekday, airflow.providers.standard.operators.bash, airflow.providers.standard.operators.python, airflow.providers.standard.operators.empty, airflow.providers.standard.operators.generic_transfer, airflow.providers.standard.operators.trigger_dagrun, airflow.providers.standard.operators.latest_only.
Hooks:: airflow.providers.standard.hooks.filesystem, airflow.providers.standard.hooks.package_index, airflow.providers.standard.hooks.subprocess.
Sensors:: airflow.providers.standard.sensors.date_time, airflow.providers.standard.sensors.time_delta, airflow.providers.standard.sensors.time, airflow.providers.standard.sensors.weekday, airflow.providers.standard.sensors.bash, airflow.providers.standard.sensors.python, airflow.providers.standard.sensors.filesystem, airflow.providers.standard.sensors.external_task.
Guides:: BashOperator, PythonOperator, BranchDateTimeOperator.
Provider:: apache-airflow-providers-standard
Product documentation:: Standard

WebHDFS¶

Hooks:: airflow.providers.apache.hdfs.hooks.webhdfs.
Sensors:: airflow.providers.apache.hdfs.sensors.web_hdfs.
Guides:: WebHDFS Operators.
Provider:: apache-airflow-providers-apache-hdfs
Product documentation:: WebHDFS

Transfers¶

These integrations allow you to copy data from/to software developed by Apache Software Foundation.

Apache Hive to Apache Druid¶

Source product documentation:: Apache Hive
Target product documentation:: Apache Druid
Python API:: airflow.providers.apache.druid.transfers.hive_to_druid
Provider:: apache-airflow-providers-apache-druid

Vertica to Apache Hive¶

Source product documentation:: Vertica
Target product documentation:: Apache Hive
Python API:: airflow.providers.apache.hive.transfers.vertica_to_hive
Provider:: apache-airflow-providers-apache-hive

Apache Hive to MySQL¶

Source product documentation:: Apache Hive
Target product documentation:: MySQL
Python API:: airflow.providers.apache.hive.transfers.hive_to_mysql
Provider:: apache-airflow-providers-apache-hive

Apache Hive to Samba¶

Source product documentation:: Apache Hive
Target product documentation:: Samba
Python API:: airflow.providers.apache.hive.transfers.hive_to_samba
Provider:: apache-airflow-providers-apache-hive

Amazon Simple Storage Service (S3) to Apache Hive¶

Source product documentation:: Amazon Simple Storage Service (S3)
Target product documentation:: Apache Hive
Python API:: airflow.providers.apache.hive.transfers.s3_to_hive
Provider:: apache-airflow-providers-apache-hive

MySQL to Apache Hive¶

Source product documentation:: MySQL
Target product documentation:: Apache Hive
Python API:: airflow.providers.apache.hive.transfers.mysql_to_hive
Provider:: apache-airflow-providers-apache-hive

Microsoft SQL Server (MSSQL) to Apache Hive¶

Source product documentation:: Microsoft SQL Server (MSSQL)
Target product documentation:: Apache Hive
Python API:: airflow.providers.apache.hive.transfers.mssql_to_hive
Provider:: apache-airflow-providers-apache-hive

Apache Cassandra to Google Cloud Storage (GCS)¶

Source product documentation:: Apache Cassandra
Target product documentation:: Google Cloud Storage (GCS)
Python API:: airflow.providers.google.cloud.transfers.cassandra_to_gcs
Provider:: apache-airflow-providers-google

Apache Hive to Amazon DynamoDB¶

Source product documentation:: Apache Hive
Target product documentation:: Amazon DynamoDB
Operator guide:: Apache Hive to Amazon DynamoDB
Python API:: airflow.providers.amazon.aws.transfers.hive_to_dynamodb
Provider:: apache-airflow-providers-amazon

Was this entry helpful?