Azure Data Factory - add message to Blob Storage Queue

Image from https://www.pexels.com/@khizar-hayat-431742/

Recently, I have had the opportunity to work on Azure Data Factory again. This time is about adding a message to Azure Blob Storage Queue via HTTP API.

Create a Queue

I created a queue, demo. Take note of its URL.

demo queue

Grant Access

Add the Azure Data Factory's System Assigned Managed Identity to Blob Storage's Storage Queue Data Message Sender role.

A new task in Azure Data Factory

And here are its configuration.

The HTTP headers are

x-ms-version: 2020-04-08
x-ms-date: @{formatDateTime(utcNow(), 'r')}

We have the message base64 encoded because the message is to be consumed in Azure Function App's Blob Storage Queue Trigger (read this for more information).

Once this Azure Data Factory pipeline is executed, the message will be added to the queue and the function app will be executed too.

Learnings

There are a few learnings that I take away from this.

1. The message needs to be base64 encoded, otherwise, the function app will not process it and move it to a poison queue.
2. I have a hard time figuring out the HTTP header, especially the x-ms-version.
3. And I have to figure out the Resource (https://storage.azure.com/) for System Assigned Managed Identity.