Microsoft Graph API Operators

Prerequisite Tasks

To use these operators, you must do a few things:

MSGraphAsyncOperator

Use the MSGraphAsyncOperator to call Microsoft Graph API.

Below is an example of using this operator to get a Sharepoint site.

tests/system/microsoft/azure/example_msgraph.py[source]

    site_task = MSGraphAsyncOperator(
        task_id="news_site",
        conn_id="msgraph_api",
        url="sites/850v1v.sharepoint.com:/sites/news",
        result_processor=lambda context, response: response["id"].split(",")[1],  # type: ignore[typeddict-item]
    )

Below is an example of using this operator to get a Sharepoint site pages.

tests/system/microsoft/azure/example_msgraph.py[source]

    site_pages_task = MSGraphAsyncOperator(
        task_id="news_pages",
        conn_id="msgraph_api",
        api_version="beta",
        url=("sites/%s/pages" % "{{ ti.xcom_pull(task_ids='news_site') }}"),  # noqa: UP031
    )

Below is an example of using this operator to get PowerBI workspaces.

tests/system/microsoft/azure/example_powerbi.py[source]

    workspaces_task = MSGraphAsyncOperator(
        task_id="workspaces",
        conn_id="powerbi",
        url="myorg/admin/workspaces/modified",
        result_processor=lambda context, response: list(map(lambda workspace: workspace["id"], response)),  # type: ignore[index]
    )

Below is an example of using this operator to get PowerBI workspaces info.

tests/system/microsoft/azure/example_powerbi.py[source]

    workspaces_info_task = MSGraphAsyncOperator(
        task_id="get_workspace_info",
        conn_id="powerbi",
        url="myorg/admin/workspaces/getInfo",
        method="POST",
        query_parameters={
            "lineage": True,
            "datasourceDetails": True,
            "datasetSchema": True,
            "datasetExpressions": True,
            "getArtifactUsers": True,
        },
        data={"workspaces": workspaces_task.output},
        result_processor=lambda context, response: {"scanId": response["id"]},  # type: ignore[typeddict-item]
    )

Below is an example of using this operator to refresh PowerBI dataset.

tests/system/microsoft/azure/example_powerbi.py[source]

    refresh_dataset_task = MSGraphAsyncOperator(
        task_id="refresh_dataset",
        conn_id="powerbi_api",
        url="myorg/groups/{workspaceId}/datasets/{datasetId}/refreshes",
        method="POST",
        path_parameters={
            "workspaceId": "9a7e14c6-9a7d-4b4c-b0f2-799a85e60a51",
            "datasetId": "ffb6096e-d409-4826-aaeb-b5d4b165dc4d",
        },
        data={"type": "full"},  # Needed for enhanced refresh
        result_processor=lambda context, response: response["requestid"],  # type: ignore[typeddict-item]
    )

    refresh_dataset_history_task = MSGraphSensor(
        task_id="refresh_dataset_history",
        conn_id="powerbi_api",
        url="myorg/groups/{workspaceId}/datasets/{datasetId}/refreshes/{refreshId}",
        path_parameters={
            "workspaceId": "9a7e14c6-9a7d-4b4c-b0f2-799a85e60a51",
            "datasetId": "ffb6096e-d409-4826-aaeb-b5d4b165dc4d",
            "refreshId": refresh_dataset_task.output,
        },
        timeout=350.0,
        event_processor=lambda context, event: event["status"] == "Completed",  # type: ignore[typeddict-item]
    )

Below is an example of using this operator to create an item schedule in Fabric.

tests/system/microsoft/azure/example_msfabric.py[source]

    # https://learn.microsoft.com/en-us/rest/api/fabric/core/job-scheduler/create-item-schedule?tabs=HTTP
    workspaces_task = MSGraphAsyncOperator(
        task_id="schedule_datapipeline",
        conn_id="powerbi",
        method="POST",
        url="workspaces/{workspaceId}/items/{itemId}/jobs/instances",
        path_parameters={
            "workspaceId": "e90b2873-4812-4dfb-9246-593638165644",
            "itemId": "65448530-e5ec-4aeb-a97e-7cebf5d67c18",
        },
        query_parameters={"jobType": "Pipeline"},
        dag=dag,
        outlets=[
            Asset(
                "workspaces/e90b2873-4812-4dfb-9246-593638165644/items/65448530-e5ec-4aeb-a97e-7cebf5d67c18/jobs/instances?jobType=Pipeline"
            )
        ],
    )

Durable execution

Set durable=True to make the operator durable. It keeps every processed page and the link to the next one in the task state store, so a retry continues at the page it stopped at. For a request that is not a GET, it also keeps the result, so a retry returns it instead of sending the request again. The pages and the link are removed from the store once the result is returned. A page that is no longer in the store, for instance because it expired under [state_store] default_retention_days, fails the try and removes the pages and the link, so the next try starts again from the first page.

Pages are stored as they are, so pages larger than [state_store] max_value_storage_bytes need a state store backend that offloads large values. Durable execution requires Airflow 3.3 or newer.

Starting from the triggerer

The operator does all its requests in the triggerer, so the only thing it does on a worker at the start is to hand the first request over. With start_from_trigger=True the scheduler defers the task itself and that first worker run is skipped. The worker gets involved once the response is there, to process it and to follow the pagination.

site_task = MSGraphAsyncOperator(
    task_id="news_site",
    conn_id="msgraph_api",
    url="sites/{{ params.site }}",
    start_from_trigger=True,
)

This needs Airflow 3.4 or later, on older versions the task starts on a worker as it does without the argument. It also starts on a worker when:

  • an argument which is passed on to the trigger is an XComArg, a callable or a file-like object, as the triggerer cannot resolve or receive those;

  • the task is mapped with expand;

  • durable=True is set, as a retry then resumes from the task state store on a worker.

Keep the following in mind when you enable it:

  • The templated fields of the first request are rendered by the triggerer, which does not know the user_defined_macros and user_defined_filters of the Dag and always renders strings, also when render_template_as_native_obj is set.

  • The task goes straight from scheduled to deferred, so the first request does not wait for a pool slot or for a worker.

Cross-References

Reference

For further information, look at:

Was this entry helpful?