Microsoft Graph API Operators¶
Prerequisite Tasks¶
To use these operators, you must do a few things:
Create necessary resources using AZURE PORTAL or AZURE CLI.
Install API libraries via pip.
pip install 'apache-airflow[azure]'Detailed information is available Installation of Airflow®
MSGraphAsyncOperator¶
Use the
MSGraphAsyncOperator to call Microsoft Graph API.
Below is an example of using this operator to get a Sharepoint site.
site_task = MSGraphAsyncOperator(
task_id="news_site",
conn_id="msgraph_api",
url="sites/850v1v.sharepoint.com:/sites/news",
result_processor=lambda context, response: response["id"].split(",")[1], # type: ignore[typeddict-item]
)
Below is an example of using this operator to get a Sharepoint site pages.
site_pages_task = MSGraphAsyncOperator(
task_id="news_pages",
conn_id="msgraph_api",
api_version="beta",
url=("sites/%s/pages" % "{{ ti.xcom_pull(task_ids='news_site') }}"), # noqa: UP031
)
Below is an example of using this operator to get PowerBI workspaces.
workspaces_task = MSGraphAsyncOperator(
task_id="workspaces",
conn_id="powerbi",
url="myorg/admin/workspaces/modified",
result_processor=lambda context, response: list(map(lambda workspace: workspace["id"], response)), # type: ignore[index]
)
Below is an example of using this operator to get PowerBI workspaces info.
workspaces_info_task = MSGraphAsyncOperator(
task_id="get_workspace_info",
conn_id="powerbi",
url="myorg/admin/workspaces/getInfo",
method="POST",
query_parameters={
"lineage": True,
"datasourceDetails": True,
"datasetSchema": True,
"datasetExpressions": True,
"getArtifactUsers": True,
},
data={"workspaces": workspaces_task.output},
result_processor=lambda context, response: {"scanId": response["id"]}, # type: ignore[typeddict-item]
)
Below is an example of using this operator to refresh PowerBI dataset.
refresh_dataset_task = MSGraphAsyncOperator(
task_id="refresh_dataset",
conn_id="powerbi_api",
url="myorg/groups/{workspaceId}/datasets/{datasetId}/refreshes",
method="POST",
path_parameters={
"workspaceId": "9a7e14c6-9a7d-4b4c-b0f2-799a85e60a51",
"datasetId": "ffb6096e-d409-4826-aaeb-b5d4b165dc4d",
},
data={"type": "full"}, # Needed for enhanced refresh
result_processor=lambda context, response: response["requestid"], # type: ignore[typeddict-item]
)
refresh_dataset_history_task = MSGraphSensor(
task_id="refresh_dataset_history",
conn_id="powerbi_api",
url="myorg/groups/{workspaceId}/datasets/{datasetId}/refreshes/{refreshId}",
path_parameters={
"workspaceId": "9a7e14c6-9a7d-4b4c-b0f2-799a85e60a51",
"datasetId": "ffb6096e-d409-4826-aaeb-b5d4b165dc4d",
"refreshId": refresh_dataset_task.output,
},
timeout=350.0,
event_processor=lambda context, event: event["status"] == "Completed", # type: ignore[typeddict-item]
)
Below is an example of using this operator to create an item schedule in Fabric.
# https://learn.microsoft.com/en-us/rest/api/fabric/core/job-scheduler/create-item-schedule?tabs=HTTP
workspaces_task = MSGraphAsyncOperator(
task_id="schedule_datapipeline",
conn_id="powerbi",
method="POST",
url="workspaces/{workspaceId}/items/{itemId}/jobs/instances",
path_parameters={
"workspaceId": "e90b2873-4812-4dfb-9246-593638165644",
"itemId": "65448530-e5ec-4aeb-a97e-7cebf5d67c18",
},
query_parameters={"jobType": "Pipeline"},
dag=dag,
outlets=[
Asset(
"workspaces/e90b2873-4812-4dfb-9246-593638165644/items/65448530-e5ec-4aeb-a97e-7cebf5d67c18/jobs/instances?jobType=Pipeline"
)
],
)
Durable execution¶
Set durable=True to make the operator durable.
It keeps every processed page and the link to the next one in the
task state store, so a retry continues at the page
it stopped at. For a request that is not a GET, it also keeps the result, so a retry returns it instead
of sending the request again. The pages and the link are removed from the store once the result is
returned. A page that is no longer in the store, for instance because it expired under
[state_store] default_retention_days, fails the try and removes the pages and the link, so the next try starts again
from the first page.
Pages are stored as they are, so pages larger than [state_store] max_value_storage_bytes need a state
store backend that offloads large values. Durable execution requires Airflow 3.3 or newer.
Starting from the triggerer¶
The operator does all its requests in the triggerer, so the only thing it does on a worker at the start
is to hand the first request over. With start_from_trigger=True the scheduler defers the task itself
and that first worker run is skipped. The worker gets involved once the response is there, to process it
and to follow the pagination.
site_task = MSGraphAsyncOperator(
task_id="news_site",
conn_id="msgraph_api",
url="sites/{{ params.site }}",
start_from_trigger=True,
)
This needs Airflow 3.4 or later, on older versions the task starts on a worker as it does without the argument. It also starts on a worker when:
an argument which is passed on to the trigger is an
XComArg, a callable or a file-like object, as the triggerer cannot resolve or receive those;the task is mapped with
expand;durable=Trueis set, as a retry then resumes from the task state store on a worker.
Keep the following in mind when you enable it:
The templated fields of the first request are rendered by the triggerer, which does not know the
user_defined_macrosanduser_defined_filtersof the Dag and always renders strings, also whenrender_template_as_native_objis set.The task goes straight from scheduled to deferred, so the first request does not wait for a pool slot or for a worker.
Cross-References¶
Microsoft Graph Filesystem - For file operations using ObjectStoragePath
Reference¶
For further information, look at: