Configuration Reference¶
This page contains the list of all available Airflow configurations for the
apache-airflow-providers-elasticsearch provider that can be set in the airflow.cfg file or using environment variables.
Note
For more information see Setting Configuration Options.
[elasticsearch]¶
end_of_log_mark¶
Added in version 1.10.4.
Used to mark the end of a log stream for a task
- Type:
string
- Default:
end_of_log- Environment Variable:
AIRFLOW__ELASTICSEARCH__END_OF_LOG_MARK
frontend¶
Added in version 1.10.4.
Qualified URL for an elasticsearch frontend (like Kibana) with a template argument for log_id Code will construct log_id using the log_id template from the argument above. NOTE: scheme will default to https if one is not provided
- Type:
string
- Default:
''- Environment Variable:
AIRFLOW__ELASTICSEARCH__FRONTEND- Example:
http://localhost:5601/app/kibana#/discover?_a=(columns:!(message),query:(language:kuery,query:'log_id: "{log_id}"'),sort:!(log.offset,asc))
host¶
Added in version 1.10.4.
Elasticsearch host
- Type:
string
- Default:
''- Environment Variable:
AIRFLOW__ELASTICSEARCH__HOST
host_field¶
Added in version 2.1.1.
The field where host name is stored (normally either host or host.name)
- Type:
string
- Default:
host- Environment Variable:
AIRFLOW__ELASTICSEARCH__HOST_FIELD
index_patterns¶
Added in version 2.6.0.
Comma separated list of index patterns to use when searching for logs (default: _all). The index_patterns_callable takes precedence over this.
- Type:
string
- Default:
_all- Environment Variable:
AIRFLOW__ELASTICSEARCH__INDEX_PATTERNS- Example:
something-*
index_patterns_callable¶
Added in version 5.5.0.
A string representing the full path to the Python callable path which accept TI object and return comma separated list of index patterns. This will takes precedence over index_patterns.
- Type:
string
- Default:
''- Environment Variable:
AIRFLOW__ELASTICSEARCH__INDEX_PATTERNS_CALLABLE- Example:
module.callable
json_fields¶
Added in version 1.10.4.
Log fields to also attach to the json output, if enabled
- Type:
string
- Default:
asctime, filename, lineno, levelname, message- Environment Variable:
AIRFLOW__ELASTICSEARCH__JSON_FIELDS
json_format¶
Added in version 1.10.4.
Instead of the default log formatter, write the log lines as JSON
- Type:
string
- Default:
False- Environment Variable:
AIRFLOW__ELASTICSEARCH__JSON_FORMAT
log_id_template¶
Added in version 1.10.4.
Format of the log_id, which is used to query for a given tasks logs
- Type:
string
- Default:
{dag_id}-{task_id}-{run_id}-{map_index}-{try_number}- Environment Variable:
AIRFLOW__ELASTICSEARCH__LOG_ID_TEMPLATE
offset_field¶
Added in version 2.1.1.
The field where offset is stored (normally either offset or log.offset)
- Type:
string
- Default:
offset- Environment Variable:
AIRFLOW__ELASTICSEARCH__OFFSET_FIELD
target_index¶
Added in version 5.5.4.
Name of the index to write to, when enabling writing the task logs to the ElasticSearch
- Type:
string
- Default:
airflow-logs- Environment Variable:
AIRFLOW__ELASTICSEARCH__TARGET_INDEX
write_stdout¶
Added in version 1.10.4.
Write the task logs to the stdout of the worker, rather than the default files
- Type:
string
- Default:
False- Environment Variable:
AIRFLOW__ELASTICSEARCH__WRITE_STDOUT
write_to_es¶
Added in version 5.5.4.
Write the task logs to the ElasticSearch
- Type:
string
- Default:
False- Environment Variable:
AIRFLOW__ELASTICSEARCH__WRITE_TO_ES
[elasticsearch_configs]¶
http_compress¶
Added in version 1.10.5.
- Type:
string
- Default:
False- Environment Variable:
AIRFLOW__ELASTICSEARCH_CONFIGS__HTTP_COMPRESS
verify_certs¶
Added in version 1.10.5.
- Type:
string
- Default:
True- Environment Variable:
AIRFLOW__ELASTICSEARCH_CONFIGS__VERIFY_CERTS