Complete the airflow survey & get a free airflow 3 certification!

ASF: Apache Software Foundation¶

Airflow supports various software created by Apache Software Foundation.

Software¶

These integrations allow you to perform various operations within software developed by Apache Software Foundation.

Apache Beam¶

Operators:: airflow.providers.apache.beam.operators.beam.
Hooks:: airflow.providers.apache.beam.hooks.beam.
Guides:: Apache Beam Operators.
Provider:: apache-airflow-providers-apache-beam
Product documentation:: Apache Beam

Apache Cassandra¶

Hooks:: airflow.providers.apache.cassandra.hooks.cassandra.
Sensors:: airflow.providers.apache.cassandra.sensors.record, airflow.providers.apache.cassandra.sensors.table.
Guides:: Apache Cassandra Operators.
Provider:: apache-airflow-providers-apache-cassandra
Product documentation:: Apache Cassandra

Apache Drill¶

Hooks:: airflow.providers.apache.drill.hooks.drill.
Guides:: Connect to Apache Drill via SQLExecuteQueryOperator.
Provider:: apache-airflow-providers-apache-drill
Product documentation:: Apache Drill

Apache Druid¶

Operators:: airflow.providers.apache.druid.operators.druid.
Hooks:: airflow.providers.apache.druid.hooks.druid.
Guides:: SQLExecuteQueryOperator to connect to Apache Druid.
Provider:: apache-airflow-providers-apache-druid
Product documentation:: Apache Druid

Apache Flink¶

Operators:: airflow.providers.apache.flink.operators.flink_kubernetes.
Sensors:: airflow.providers.apache.flink.sensors.flink_kubernetes.
Guides:: Apache Flink Operators.
Provider:: apache-airflow-providers-apache-flink
Product documentation:: Apache Flink

Apache Hive¶

Operators:: airflow.providers.apache.hive.operators.hive, airflow.providers.apache.hive.operators.hive_stats.
Hooks:: airflow.providers.apache.hive.hooks.hive.
Sensors:: airflow.providers.apache.hive.sensors.hive_partition, airflow.providers.apache.hive.sensors.metastore_partition, airflow.providers.apache.hive.sensors.named_hive_partition.
Guides:: SQLExecuteQueryOperator to connect to Apache Hive.
Provider:: apache-airflow-providers-apache-hive
Product documentation:: Apache Hive

Apache Impala¶

Hooks:: airflow.providers.apache.impala.hooks.impala.
Guides:: SQLExecuteQueryOperator to connect to Apache Impala.
Provider:: apache-airflow-providers-apache-impala
Product documentation:: Apache Impala

Apache Kafka¶

Operators:: airflow.providers.apache.kafka.operators.consume, airflow.providers.apache.kafka.operators.produce.
Hooks:: airflow.providers.apache.kafka.hooks.base, airflow.providers.apache.kafka.hooks.client, airflow.providers.apache.kafka.hooks.consume, airflow.providers.apache.kafka.hooks.produce.
Sensors:: airflow.providers.apache.kafka.sensors.kafka.
Guides:: Apache Kafka Sensors.
Provider:: apache-airflow-providers-apache-kafka
Product documentation:: Apache Kafka

Apache Kylin¶

Operators:: airflow.providers.apache.kylin.operators.kylin_cube.
Hooks:: airflow.providers.apache.kylin.hooks.kylin.
Guides:: SQLExecuteQueryOperator to connect to Apache Kylin.
Provider:: apache-airflow-providers-apache-kylin
Product documentation:: Apache Kylin

Apache Livy¶

Operators:: airflow.providers.apache.livy.operators.livy.
Hooks:: airflow.providers.apache.livy.hooks.livy.
Sensors:: airflow.providers.apache.livy.sensors.livy.
Guides:: Apache Livy Operators.
Provider:: apache-airflow-providers-apache-livy
Product documentation:: Apache Livy

Apache Pig¶

Operators:: airflow.providers.apache.pig.operators.pig.
Hooks:: airflow.providers.apache.pig.hooks.pig.
Guides:: Apache Pig Operators.
Provider:: apache-airflow-providers-apache-pig
Product documentation:: Apache Pig

Apache Pinot¶

Hooks:: airflow.providers.apache.pinot.hooks.pinot.
Guides:: SQLExecuteQueryOperator to connect to Apache Pinot.
Provider:: apache-airflow-providers-apache-pinot
Product documentation:: Apache Pinot

Apache Spark¶

Operators:: airflow.providers.apache.spark.operators.spark_jdbc, airflow.providers.apache.spark.operators.spark_sql, airflow.providers.apache.spark.operators.spark_submit.
Hooks:: airflow.providers.apache.spark.hooks.spark_connect, airflow.providers.apache.spark.hooks.spark_jdbc, airflow.providers.apache.spark.hooks.spark_jdbc_script, airflow.providers.apache.spark.hooks.spark_sql, airflow.providers.apache.spark.hooks.spark_submit.
Guides:: Apache Spark Operators.
Provider:: apache-airflow-providers-apache-spark
Product documentation:: Apache Spark

Apache TinkerPop¶

Operators:: airflow.providers.apache.tinkerpop.operators.gremlin.
Hooks:: airflow.providers.apache.tinkerpop.hooks.gremlin.
Guides:: Apache TinkerPop Operators.
Provider:: apache-airflow-providers-apache-tinkerpop
Product documentation:: Apache TinkerPop

Standard¶

Operators:: airflow.providers.standard.operators.datetime, airflow.providers.standard.operators.weekday, airflow.providers.standard.operators.bash, airflow.providers.standard.operators.python, airflow.providers.standard.operators.empty, airflow.providers.standard.operators.trigger_dagrun, airflow.providers.standard.operators.latest_only, airflow.providers.standard.operators.smooth, airflow.providers.standard.operators.branch, airflow.providers.standard.operators.hitl.
Hooks:: airflow.providers.standard.hooks.filesystem, airflow.providers.standard.hooks.package_index, airflow.providers.standard.hooks.subprocess.
Sensors:: airflow.providers.standard.sensors.date_time, airflow.providers.standard.sensors.time_delta, airflow.providers.standard.sensors.time, airflow.providers.standard.sensors.weekday, airflow.providers.standard.sensors.bash, airflow.providers.standard.sensors.python, airflow.providers.standard.sensors.filesystem, airflow.providers.standard.sensors.external_task.
Guides:: BashOperator, PythonOperator, BranchDateTimeOperator, TriggerDagRunOperator, LatestOnlyOperator, BashSensor, PythonSensor, TimeDeltaSensor, FileSensor, Cross-Dag Dependencies.
Provider:: apache-airflow-providers-standard
Product documentation:: Standard

WebHDFS¶

Hooks:: airflow.providers.apache.hdfs.hooks.webhdfs.
Sensors:: airflow.providers.apache.hdfs.sensors.web_hdfs.
Guides:: WebHDFS Operators.
Provider:: apache-airflow-providers-apache-hdfs
Product documentation:: WebHDFS

Transfers¶

These integrations allow you to copy data from/to software developed by Apache Software Foundation.

Apache Hive to Amazon DynamoDB¶

Source product documentation:: Apache Hive
Target product documentation:: Amazon DynamoDB
Operator guide:: Apache Hive to Amazon DynamoDB
Python API:: airflow.providers.amazon.aws.transfers.hive_to_dynamodb
Provider:: apache-airflow-providers-amazon

Apache Hive to Apache Druid¶

Source product documentation:: Apache Hive
Target product documentation:: Apache Druid
Python API:: airflow.providers.apache.druid.transfers.hive_to_druid
Provider:: apache-airflow-providers-apache-druid

Vertica to Apache Hive¶

Source product documentation:: Vertica
Target product documentation:: Apache Hive
Python API:: airflow.providers.apache.hive.transfers.vertica_to_hive
Provider:: apache-airflow-providers-apache-hive

Apache Hive to MySQL¶

Source product documentation:: Apache Hive
Target product documentation:: MySQL
Python API:: airflow.providers.apache.hive.transfers.hive_to_mysql
Provider:: apache-airflow-providers-apache-hive

Apache Hive to Samba¶

Source product documentation:: Apache Hive
Target product documentation:: Samba
Python API:: airflow.providers.apache.hive.transfers.hive_to_samba
Provider:: apache-airflow-providers-apache-hive

Amazon Simple Storage Service (S3) to Apache Hive¶

Source product documentation:: Amazon Simple Storage Service (S3)
Target product documentation:: Apache Hive
Python API:: airflow.providers.apache.hive.transfers.s3_to_hive
Provider:: apache-airflow-providers-apache-hive

MySQL to Apache Hive¶

Source product documentation:: MySQL
Target product documentation:: Apache Hive
Python API:: airflow.providers.apache.hive.transfers.mysql_to_hive
Provider:: apache-airflow-providers-apache-hive

Microsoft SQL Server (MSSQL) to Apache Hive¶

Source product documentation:: Microsoft SQL Server (MSSQL)
Target product documentation:: Apache Hive
Python API:: airflow.providers.apache.hive.transfers.mssql_to_hive
Provider:: apache-airflow-providers-apache-hive

Apache Cassandra to Google Cloud Storage (GCS)¶

Source product documentation:: Apache Cassandra
Target product documentation:: Google Cloud Storage (GCS)
Python API:: airflow.providers.google.cloud.transfers.cassandra_to_gcs
Provider:: apache-airflow-providers-google

Was this entry helpful?