Params

Params enable you to provide runtime configuration to tasks. You can configure default Params in your DAG code and supply additional Params, or overwrite Param values, at runtime when you trigger a DAG. Param values are validated with JSON Schema. For scheduled DAG runs, default Param values are used.

Also defined Params are used to render a nice UI when triggering manually. When you trigger a DAG manually, you can modify its Params before the dagrun starts. If the user-supplied values don’t pass validation, Airflow shows a warning instead of creating the dagrun.

DAG-level Params

To add Params to a DAG, initialize it with the params kwarg. Use a dictionary that maps Param names to either a Param or an object indicating the parameter’s default value.

 from airflow import DAG
 from airflow.decorators import task
 from airflow.models.param import Param

 with DAG(
     "the_dag",
     params={
         "x": Param(5, type="integer", minimum=3),
         "my_int_param": 6
     },
 ) as dag:

     @task.python
     def example_task(params: dict):
         # This will print the default value, 6:
         dag.log.info(dag.params['my_int_param'])

         # This will print the manually-provided value, 42:
         dag.log.info(params['my_int_param'])

         # This will print the default value, 5, since it wasn't provided manually:
         dag.log.info(params['x'])

     example_task()

 if __name__ == "__main__":
     dag.test(
         run_conf={"my_int_param": 42}
     )

Note

DAG-level parameters are the default values passed on to tasks. These should not be confused with values manually provided through the UI form or CLI, which exist solely within the context of a DagRun and a TaskInstance. This distinction is crucial for TaskFlow DAGs, which may include logic within the with DAG(...) as dag: block. In such cases, users might try to access the manually-provided parameter values using the dag object, but this will only ever contain the default values. To ensure that the manually-provided values are accessed, use a template variable such as params or ti within your task.

Task-level Params

You can also add Params to individual tasks.

def print_my_int_param(params):
  print(params.my_int_param)

PythonOperator(
    task_id="print_my_int_param",
    params={"my_int_param": 10},
    python_callable=print_my_int_param,
)

Task-level params take precedence over DAG-level params, and user-supplied params (when triggering the DAG) take precedence over task-level params.

Referencing Params in a Task

Params can be referenced in templated strings under params. For example:

 PythonOperator(
     task_id="from_template",
     op_args=[
         "{{ params.my_int_param + 10 }}",
     ],
     python_callable=(
         lambda my_int_param: print(my_int_param)
     ),
 )

Even though Params can use a variety of types, the default behavior of templates is to provide your task with a string. You can change this by setting render_template_as_native_obj=True while initializing the DAG.

 with DAG(
     "the_dag",
     params={"my_int_param": Param(5, type="integer", minimum=3)},
     render_template_as_native_obj=True
 ):

This way, the Param’s type is respected when it’s provided to your task:

# prints <class 'str'> by default
# prints <class 'int'> if render_template_as_native_obj=True
PythonOperator(
    task_id="template_type",
    op_args=[
        "{{ params.my_int_param }}",
    ],
    python_callable=(
        lambda my_int_param: print(type(my_int_param))
    ),
)

Another way to access your param is via a task’s context kwarg.

 def print_my_int_param(**context):
     print(context["params"]["my_int_param"])

 PythonOperator(
     task_id="print_my_int_param",
     python_callable=print_my_int_param,
     params={"my_int_param": 12345},
 )

JSON Schema Validation

Param makes use of JSON Schema, so you can use the full JSON Schema specifications mentioned at https://json-schema.org/draft/2020-12/json-schema-validation.html to define Param objects.

with DAG(
    "my_dag",
    params={
        # an int with a default value
        "my_int_param": Param(10, type="integer", minimum=0, maximum=20),

        # a required param which can be of multiple types
        # a param must have a default value
        "multi_type_param": Param(5, type=["null", "number", "string"]),

        # an enum param, must be one of three values
        "enum_param": Param("foo", enum=["foo", "bar", 42]),

        # a param which uses json-schema formatting
        "email": Param(
            default="example@example.com",
            type="string",
            format="idn-email",
            minLength=5,
            maxLength=255,
        ),
    },
):

Note

If schedule is defined for a DAG, params with defaults must be valid. This is validated during DAG parsing. If schedule=None then params are not validated during DAG parsing but before triggering a DAG. This is useful in cases where the DAG author does not want to provide defaults but wants to force users provide valid parameters at time of trigger.

Note

As of now, for security reasons, one can not use Param objects derived out of custom classes. We are planning to have a registration system for custom Param classes, just like we’ve for Operator ExtraLinks.

Use Params to Provide a Trigger UI Form

New in version 2.6.0.

DAG level params are used to render a user friendly trigger form. This form is provided when a user clicks on the “Trigger DAG” button.

The Trigger UI Form is rendered based on the pre-defined DAG Params. If the DAG has no params defined, the trigger form is skipped. The form elements can be defined with the Param class and attributes define how a form field is displayed.

The following features are supported in the Trigger UI Form:

  • Direct scalar values (boolean, int, string, lists, dicts) from top-level DAG params are auto-boxed into Param objects. From the native Python data type the type attribute is auto detected. So these simple types render to a corresponding field type. The name of the parameter is used as label and no further validation is made, all values are treated as optional.

  • If you use the Param class as definition of the parameter value, the following attributes can be added:

    • The Param attribute title is used to render the form field label of the entry box. If no title is defined the parameter name/key is used instead.

    • The Param attribute description is rendered below an entry field as help text in gray color. If you want to provide special formatting or links you need to use the Param attribute description_md. See tutorial DAG Params UI example DAG for an example.

    • The Param attribute type influences how a field is rendered. The following types are supported:

      Param type

      Form element type

      Additional supported attributes

      Example

      string

      Generates a single-line text box or a text area to edit text.

      • minLength: Minimum text length

      • maxLength: Maximum text length

      • format="date": Generate a date-picker
        with calendar pop-up
      • format="date-time": Generate a date and
        time-picker with calendar pop-up

      • format="time": Generate a time-picker

      • format="multiline": Generate a multi-line textarea

      • enum=["a", "b", "c"]: Generates a
        drop-down select list for scalar values.
        As of JSON validation, a value must be
        selected or the field must be marked as
        optional explicit. See also details inside
        the JSON Schema Description for Enum.
      • values_display={"a": "Alpha", "b": "Beta"}:
        For select drop-downs generated via
        enum you can add the attribute
        values_display with a dict and map data
        values to display labels.
      • examples=["One", "Two", "Three"]: If you
        want to present proposals for values
        (not restricting the user to a fixed enum
        as above) you can make use of examples
        which is a list of items.
      Also see
      further JSON Schema string type validation options
      which are checked before DAG trigger in the backend.

      Param("default", type="string", maxLength=10)

      Param(f"{datetime.date.today()}", type="string", format="date")

      number or

      integer
      Generates a field which restricts adding
      numeric values only. The HTML browser
      typically also adds a spinner on the
      right side to increase or decrease the
      value. integer only permits int
      numbers, number allows also
      fractional values.

      • minimum: Minimum number value

      • maximum: Maximum number value

      Also see
      further JSON Schema numeric type validation options
      which are checked before DAG trigger in the backend.

      Param(42, type="integer", minimum=14, multipleOf=7)

      boolean
      Generates a toggle button to be used
      as True or False.

      none.

      Param(True, type="boolean")

      array
      Generates a HTML multi line text field,
      every line edited will be made into a
      string array as value.
      • If you add the attribute examples
        with a list, a multi-value select option
        will be generated instead of a free text field.
      • values_display={"a": "Alpha", "b": "Beta"}:
        For multi-value selects examples you can add
        the attribute values_display with a dict and
        map data values to display labels.
      • If you add the attribute items with a
        dictionary that contains a field type
        with a value other than “string”, a JSON entry
        field will be generated for more array types and
        additional type validation as described in
        JSON Schema Array Items.

      Param(["a", "b", "c"], type="array")

      Param(["two", "three"], type="array", examples=["one", "two", "three", "four", "five"])

      Param(["one@example.com", "two@example.com"], type="array", items={"type": "string", "format": "idn-email"})

      object
      Generates a JSON entry field with
      text validation.
      The HTML form does only validate the syntax of the
      JSON input. In order to validate the content for
      specific structures take a look to the
      JSON Schema Object details.

      Param({"key": "value"}, type=["object", "null"])

      null
      Specifies that no content is expected.
      Standalone this does not make much sense
      but it is useful for type combinations
      like type=["null", "string"] as the
      type attribute also accepts a list of
      types.
      Per default if you specify a type, a
      field will be made required with
      input - because of JSON validation.
      If you want to have a field value being
      added optional only, you must allow
      JSON schema validation allowing null
      values.

      Param(None, type=["null", "string"])

  • If a form field is left empty, it is passed as None value to the params dict.

  • Form fields are rendered in the order of definition of params in the DAG.

  • If you want to add sections to the Form, add the attribute section to each field. The text will be used as section label. Fields w/o section will be rendered in the default area. Additional sections will be collapsed per default.

  • If you want to have params not being displayed, use the const attribute. These Params will be submitted but hidden in the Form. The const value must match the default value to pass JSON Schema validation.

  • On the bottom of the form the generated JSON configuration can be expanded. If you want to change values manually, the JSON configuration can be adjusted. Changes are overridden when form fields change.

  • To pre-populate values in the form when publishing a link to the trigger form you can call the trigger URL /dags/<dag_name>/trigger and add query parameter to the URL in the form name=value, for example /dags/example_params_ui_tutorial/trigger?required_field=some%20text. To pre-define the run id of the DAG run, use the URL parameter run_id.

  • Fields can be required or optional. Typed fields are required by default to ensure they pass JSON schema validation. To make typed fields optional, you must allow the “null” type.

  • Fields without a “section” will be rendered in the default area. Additional sections will be collapsed by default.

Note

If the field is required the default value must be valid according to the schema as well. If the DAG is defined with schedule=None the parameter value validation is made at time of trigger.

For examples, please take a look at the two example DAGs provided: Params trigger example DAG and Params UI example DAG.

airflow/example_dags/example_params_trigger_ui.py[source]

with DAG(
    dag_id=Path(__file__).stem,
    dag_display_name="Params Trigger UI",
    description=__doc__.partition(".")[0],
    doc_md=__doc__,
    schedule=None,
    start_date=datetime.datetime(2022, 3, 4),
    catchup=False,
    tags=["example", "params"],
    params={
        "names": Param(
            ["Linda", "Martha", "Thomas"],
            type="array",
            description="Define the list of names for which greetings should be generated in the logs."
            " Please have one name per line.",
            title="Names to greet",
        ),
        "english": Param(True, type="boolean", title="English"),
        "german": Param(True, type="boolean", title="German (Formal)"),
        "french": Param(True, type="boolean", title="French"),
    },
) as dag:

    @task(task_id="get_names", task_display_name="Get names")
    def get_names(**kwargs) -> list[str]:
        params: ParamsDict = kwargs["params"]
        if "names" not in params:
            print("Uuups, no names given, was no UI used to trigger?")
            return []
        return params["names"]

    @task.branch(task_id="select_languages", task_display_name="Select languages")
    def select_languages(**kwargs) -> list[str]:
        params: ParamsDict = kwargs["params"]
        selected_languages = []
        for lang in ["english", "german", "french"]:
            if params[lang]:
                selected_languages.append(f"generate_{lang}_greeting")
        return selected_languages

    @task(task_id="generate_english_greeting", task_display_name="Generate English greeting")
    def generate_english_greeting(name: str) -> str:
        return f"Hello {name}!"

    @task(task_id="generate_german_greeting", task_display_name="Erzeuge Deutsche Begrüßung")
    def generate_german_greeting(name: str) -> str:
        return f"Sehr geehrter Herr/Frau {name}."

    @task(task_id="generate_french_greeting", task_display_name="Produire un message d'accueil en français")
    def generate_french_greeting(name: str) -> str:
        return f"Bonjour {name}!"

    @task(task_id="print_greetings", task_display_name="Print greetings", trigger_rule=TriggerRule.ALL_DONE)
    def print_greetings(greetings1, greetings2, greetings3) -> None:
        for g in greetings1 or []:
            print(g)
        for g in greetings2 or []:
            print(g)
        for g in greetings3 or []:
            print(g)
        if not (greetings1 or greetings2 or greetings3):
            print("sad, nobody to greet :-(")

    lang_select = select_languages()
    names = get_names()
    english_greetings = generate_english_greeting.expand(name=names)
    german_greetings = generate_german_greeting.expand(name=names)
    french_greetings = generate_french_greeting.expand(name=names)
    lang_select >> [english_greetings, german_greetings, french_greetings]
    results_print = print_greetings(english_greetings, german_greetings, french_greetings)

airflow/example_dags/example_params_ui_tutorial.py[source]

        params={
            # Let's start simple: Standard dict values are detected from type and offered as entry form fields.
            # Detected types are numbers, text, boolean, lists and dicts.
            # Note that such auto-detected parameters are treated as optional (not required to contain a value)
            "x": 3,
            "text": "Hello World!",
            "flag": False,
            "a_simple_list": ["one", "two", "three", "actually one value is made per line"],
            # But of course you might want to have it nicer! Let's add some description to parameters.
            # Note if you can add any Markdown formatting to the description, you need to use the description_md
            # attribute.
            "most_loved_number": Param(
                42,
                type="integer",
                title="Your favorite number",
                description_md="Everybody should have a **favorite** number. Not only _math teachers_. "
                "If you can not think of any at the moment please think of the 42 which is very famous because"
                "of the book [The Hitchhiker's Guide to the Galaxy]"
                "(https://en.wikipedia.org/wiki/Phrases_from_The_Hitchhiker%27s_Guide_to_the_Galaxy#"
                "The_Answer_to_the_Ultimate_Question_of_Life,_the_Universe,_and_Everything_is_42).",
            ),
            # If you want to have a selection list box then you can use the enum feature of JSON schema
            "pick_one": Param(
                "value 42",
                type="string",
                title="Select one Value",
                description="You can use JSON schema enum's to generate drop down selection boxes.",
                enum=[f"value {i}" for i in range(16, 64)],
            ),

airflow/example_dags/example_params_ui_tutorial.py[source]

            "required_field": Param(
                # In this example we have no default value
                # Form will enforce a value supplied by users to be able to trigger
                type="string",
                title="Required text field",
                description="This field is required. You can not submit without having text in here.",
            ),
            "optional_field": Param(
                "optional text, you can trigger also w/o text",
                type=["null", "string"],
                title="Optional text field",
                description_md="This field is optional. As field content is JSON schema validated you must "
                "allow the `null` type.",
            ),

airflow/example_dags/example_params_ui_tutorial.py[source]

    @task(task_display_name="Show used parameters")
    def show_params(**kwargs) -> None:
        params: ParamsDict = kwargs["params"]
        print(f"This DAG was triggered with the following parameters:\n\n{json.dumps(params, indent=4)}\n")

    show_params()
../_images/trigger-dag-tutorial-form.png

New in version 2.7.0: The trigger form can also be forced to be displayed also if no params are defined using the configuration switch webserver.show_trigger_form_if_no_params.

Changed in version 3.0.0: By default custom HTML is not allowed to prevent injection of scripts or other malicious HTML code. The previous field named description_html is now super-seeded with the attribute description_md. description_html is not supported anymore. Custom form elements using the attribute custom_html_form was deprecated in version 2.8.0 and support was removed in 3.0.0.

Disabling Runtime Param Modification

The ability to update params while triggering a DAG depends on the flag core.dag_run_conf_overrides_params. Setting this config to False will effectively turn your default params into constants.

Was this entry helpful?