Configuration Reference¶
This page contains the list of all available Airflow configurations for the
apache-airflow-providers-elasticsearch provider that can be set in the airflow.cfg file or using environment variables.
Note
The configuration embedded in provider packages started to be used as of Airflow 2.7.0. Previously the configuration was described and configured in the Airflow core package - so if you are using Airflow below 2.7.0, look at Airflow documentation for the list of available configuration options that were available in Airflow core.
Note
For more information see Setting Configuration Options.
[elasticsearch]¶
end_of_log_mark¶
Added in version 1.10.4.
Used to mark the end of a log stream for a task
- Type:
string
- Default:
end_of_log- Environment Variable:
AIRFLOW__ELASTICSEARCH__END_OF_LOG_MARK
frontend¶
Added in version 1.10.4.
Qualified URL for an elasticsearch frontend (like Kibana) with a template argument for log_id Code will construct log_id using the log_id template from the argument above. NOTE: scheme will default to https if one is not provided
- Type:
string
- Default:
''- Environment Variable:
AIRFLOW__ELASTICSEARCH__FRONTEND- Example:
http://localhost:5601/app/kibana#/discover?_a=(columns:!(message),query:(language:kuery,query:'log_id: "{log_id}"'),sort:!(log.offset,asc))
host¶
Added in version 1.10.4.
Elasticsearch host
- Type:
string
- Default:
''- Environment Variable:
AIRFLOW__ELASTICSEARCH__HOST
host_field¶
Added in version 2.1.1.
The field where host name is stored (normally either host or host.name)
- Type:
string
- Default:
host- Environment Variable:
AIRFLOW__ELASTICSEARCH__HOST_FIELD
index_patterns¶
Added in version 2.6.0.
Comma separated list of index patterns to use when searching for logs (default: _all). The index_patterns_callable takes precedence over this.
- Type:
string
- Default:
_all- Environment Variable:
AIRFLOW__ELASTICSEARCH__INDEX_PATTERNS- Example:
something-*
index_patterns_callable¶
Added in version 5.5.0.
A string representing the full path to the Python callable path which accept TI object and return comma separated list of index patterns. This will takes precedence over index_patterns.
- Type:
string
- Default:
''- Environment Variable:
AIRFLOW__ELASTICSEARCH__INDEX_PATTERNS_CALLABLE- Example:
module.callable
json_fields¶
Added in version 1.10.4.
Log fields to also attach to the json output, if enabled
- Type:
string
- Default:
asctime, filename, lineno, levelname, message- Environment Variable:
AIRFLOW__ELASTICSEARCH__JSON_FIELDS
json_format¶
Added in version 1.10.4.
Instead of the default log formatter, write the log lines as JSON
- Type:
string
- Default:
False- Environment Variable:
AIRFLOW__ELASTICSEARCH__JSON_FORMAT
log_id_template¶
Added in version 1.10.4.
Format of the log_id, which is used to query for a given tasks logs
- Type:
string
- Default:
{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}- Environment Variable:
AIRFLOW__ELASTICSEARCH__LOG_ID_TEMPLATE
offset_field¶
Added in version 2.1.1.
The field where offset is stored (normally either offset or log.offset)
- Type:
string
- Default:
offset- Environment Variable:
AIRFLOW__ELASTICSEARCH__OFFSET_FIELD
target_index¶
Added in version 5.5.4.
Name of the index to write to, when enabling writing the task logs to the ElasticSearch
- Type:
string
- Default:
airflow-logs- Environment Variable:
AIRFLOW__ELASTICSEARCH__TARGET_INDEX
write_stdout¶
Added in version 1.10.4.
Write the task logs to the stdout of the worker, rather than the default files
- Type:
string
- Default:
False- Environment Variable:
AIRFLOW__ELASTICSEARCH__WRITE_STDOUT
write_to_es¶
Added in version 5.5.4.
Write the task logs to the ElasticSearch
- Type:
string
- Default:
False- Environment Variable:
AIRFLOW__ELASTICSEARCH__WRITE_TO_ES
[elasticsearch_configs]¶
http_compress¶
Added in version 1.10.5.
- Type:
string
- Default:
False- Environment Variable:
AIRFLOW__ELASTICSEARCH_CONFIGS__HTTP_COMPRESS
verify_certs¶
Added in version 1.10.5.
- Type:
string
- Default:
True- Environment Variable:
AIRFLOW__ELASTICSEARCH_CONFIGS__VERIFY_CERTS