Connect to Azure Data Lake Store using MuleSoft


Azure Data Lake Store is an enterprise-wide hyper-scale repository for big data analytic workloads. Azure Data Lake enables you to capture data of any size, type, and ingestion speed in one single place for operational and exploratory analytics.

This guide shows how easy it is to connect to Azure Data Lake Store using MuleSoft by using the webHDFS REST APIs that are exposed. Azure Data Lake Store is essentially a version of HDFS. While MuleSoft has a connector for HDFS, Azure requires authentication through Active Directory using OAuth. The guide below will show you how to setup an Azure Data Lake Store, setup an Azure Active Directory app, and connect to Azure using a pre-configured project in MuleSoft Anypoint Studio.


  • Microsoft Azure account
  • MuleSoft Anypoint Studio

Create an Azure Data Lake Store

If you don’t already have an Azure Data Lake Store, these are the steps to setup one yourself. It does require that you have an Azure account. You can sign up for a 30 day account or just use the Pay-As-You-Go account which is what I used.

  1. Login to the Azure portal and click on All services on the left hand navigation bar.
  2. In the search field, type in data lake and click on Data Lake Store

  3. In Data Lake Store, click on Create Data Lake Store in the center or click on Add in the top left.
  4. In the New Data Lake Store window, enter the following information:
    • Name: mulesoft
    • Subscription: This can depend on your account. If you recently signed up, you should have a 30 day subscription. I selected Pay-As-You-Go
    • Resource group:
      Either create a new resource group or use an existing one. (e.g. mulesoft)
    • Location: leave the default East US 2
    • Encryption settings: I set this to Do not enable encryption.
  5. Click on Create
  6. Once the Data Lake Store is created, click on Data explorer in the left hand navigation menu. We want to grant access to all users for this demo, so click on the Access button.
  7. In the Access screen, check the Read, Write, and Execute checkboxes under Everyone else and then click on Save

Create an Azure Active Directory “Web” Application

    1. In the left hand navigation menu, click on Azure Active Directory. If the menu item isn’t there, click on All services and search for it.
    2. In the navigation menu for Azure Active Directory, click on App registrations
    3. Click on New application registration
    4. In the Create window, enter the following data:
      • Name: mule
      • Application type: Keep the default Web app / API
      • Sign-on URL: Just enter http://localhost:8081. This can be changed later and doesn’t affect anything in this demo.
      • Click on Create
        Once the app has been created, copy down the Application ID: e.g. bdcabff5-af3c-4127-b69b-38bcf1792bfd
    5. Next, click on Settings and then click on Required permissions
    6. In the Required permissions screen, click on Add and then click on Select an API. Then select Azure Data Lake from the list of available APIs and then click on Select
    7. In the Enable Access screen, check the Have full access to the Azure Data Lake service checkbox and then click on Select
    8. Next we need to generate a key. Click on Keys on the left hand navigation bar for the app settings.
    9. In the Keys window, enter the following:
      • Description: mule-app-key
      • Expires: Set this to Never expires
      • And then click on Save. A value will appear for the key. Copy that value down. e.g. +zAbZQgXomvqsfHgCH32Yv+VCvkT3ZcxRyw5CWaw4dw=
    10. Next, let’s get your tenant ID. In Azure Active Directory click on Properties in the left hand navigation bar.
    11. Copy down the Directory ID value. That’s your tenant ID that you’ll need to make the OAuth call. e.g. 57744783-79ff-49ab-b27e-26245d4d97ef


Download and Run the Mule Project

  1. Download the project from Github
  2. Import the project into Anypoint Studio
  3. Open the and modify the following properties:
    • <name of your Data Lake Store>
    • oauth.path: <tenant ID from Step 2.11>/oauth2/token
    • adls.client_id: This is the ID from Step 2.5
    • adls.client_secret: This is the key from Step 2.9
  4. The first flow will list the files and folders from the root directory of the Data Lake Store. Once the flow receives the request it creates the parameters to be sent to the OAuth request. If the OAuth request is successful, it returns an access token that is used to make the HDFS request to list the folders.
  5. The second flow shows how to upload data to the Data Lake Store. Similar to the first flow, it makes an OAuth request and passes the access token to make the HDFS request.
  6. Run the project and open up Postman.
  7. Let’s test the first flow. Paste the following into the request URL: http://localhost:8081/liststatus and click Send. The screenshot below shows the results. If you add some folders and files, you’ll receive more data from the API call.
  8. Open another tab and paste the following URL: http://localhost:8081/create?name=list2.txt
  9. Change the method to PUT. Under the Body section, select the binary radio button and select the file name list.txt from the src/main/resources folder from the project. Click on Send.
  10. If successful, when you switch back the Data Lake Store, you should see the file at the root directory. Click to open the file.
  11. You should see the following data in the file if everything was configured successfully.

Deploy Domain Projects using the Mule Plugin for Maven

Continuous Integration (CI) is a DevOps development practice that requires developers to check code into a shared repository (e.g. Github) after which automated builds and tests are run. This process allows teams to detect problems early and locate them more easily.

The MuleSoft Anypoint Platform provides the flexibility and tools to easily build a CI process into your API Lifecycle or tie into an existing CI process using common tooling like Jenkins, Maven, TeamCity, etc…

The mule-maven-plugin is a tool that we provide that allows you to deploy Mule applications to different kinds of Mule Runtime servers: Standalone (both Community and Enterprise), Clustered, and CloudHub.

Recently I had to figure out how to deploy a Mule Domain project to a standalone Mule Runtime. I documented the process below.

My environment consisted of the following components all running locally on my Macbook.

In Studio, I created a ‘Mule Domain Project’. Be sure to check the Use Maven checkbox when you create the project. Also set the ‘Project Name’ as domain. This is important for the deployment to work for this particular exercise. I’ll go into detail why later.

In the domain, I just created an HTTP listener on port 8082 that would be shared by projects in the domain.

Next, I created a really simple ‘Hello World!’ project as part of the domain. Again, be sure the check the Use Maven checkbox when you create the project. The flow uses the shared HTTP listener resource as you can see in the code below.

For Maven to compile and deploy the project, we need to modify the pom.xml file and add the plugin.

You can use the code below. There are three configurations that are required for this to work.

  • <deploymentType> – This ensures that the project is deployed to a standalone Mule Runtime
  • <muleHome> – Setting this to the base directory of your Mule Runtime ensures that Maven doesn’t download and install a Mule standalone server.
  • <domain> – This is the location of the mule-domain-config.xml file. The plugin uses the name of the folder that the file sits in when it creates the domain on the Runtime server. This name needs to match the domain that the Mule projects are tied to. For this article, I created a domain called domain so I wouldn’t have to move the mule-domain-config.xml to a different folder that matched the Mule project.

Once everything is setup, you can execute the deployment by running the following command:

If configured successfully, Maven will create a domain and compile and deploy the Mule project to the apps folder. If the Mule Runtime is stopped, it will start it as well.

Hopefully you’re successful in setting this project up. Leave a comment if you run into any issues or have any questions. Good Luck!


Create a Facebook Messenger Bot using MuleSoft and the Anypoint Platform.

With over 900 million users a month, the Facebook Messenger platform has a pretty big audience.  The introduction of Messenger Bots earlier this year helped extend this adoption by helping drive personalized, scaled experiences with customers. Instead of building an app, just build the message thread as the place where you connect and interact with your users.

I was curious though to see how easy it was to build my own Bot using MuleSoft and the Anypoint Platform. Instead of writing custom code and running my own dedicated server, I leveraged Anypoint Studio to configure an app and CloudHub for my server needs. The following is a walk-through that shows you how you can create your own Facebook Messenger Bot.

Setup Facebook App and Page

To get started, setup a new app in Facebook under the developers site:

Add New App for Messenger

Be sure to select Apps for Messenger under the Category dropdown before clicking Create App ID. 

Once the app gets created, scroll down to the Token Generation section and click on Create a new pageYou’ll be re-directed to the Facebook site to create a new page.

Create a new page

Select any category (I selected Brand or Product > Software) and give it a name (e.g. MuleSoft Demos). Skip through the setup steps and jump back to the Facebook Developer site. Once you refresh the page, the new Facebook Page with show under the Token Generation section under the Page dropdown.

Select your new page and it will create a Page Access Token. You’ll need this for your Mule app to call the Send API later.

Page Access Token

Deploy App to CloudHub

Next, deploy the example app to CloudHub. You can download a copy of the project from Github here: and import it into Anypoint Studio.

Before you deploy it to CloudHub, open the the file and copy and paste the Page Access Token to the access_token variable.


The example app is built to demonstrate a few of the the required event listeners for the Facebook Messenger platform. It handles the webhooks for messages and postbacks. Messages are parsed for keywords such as ‘template’, ‘button’, etc.. and all other messages are echoed back to the user using the Send API.

When the app is deployed and running in CloudHub, navigate to the app URL (e.g and test out the webhook challenge using the following URL:


You should get the challenge returned like the following screenshot:

Setup Webhooks

With the challenge returned successfully, we can setup the webhooks in Facebook now. Switch back to the Facebook Developers site and click on Setup Webhooks under the Webhooks section.

Enter the app URL under Callback URL with the path webhook. Enter any text under the Verify Token field. Under the Subscription Fields section, check messages and messaging_postbacks and then click on Verify and Save.

Once the webhook setup is complete, select the Facebook Page that was created previously and click on Subscribe in the Webhooks section.

Test Messages

Everything should be setup and running now on both CloudHub and Facebook. Let’s go ahead and test out the messages. Switch back to Facebook and open up the Page you created. You should see a button that says Send Message. Mouseover that button and then click on Test Button.

Go ahead and test out various messages and see their responses. When you type in messages that contain keywords such as button or template, you’ll see specific message formats returned.

You can also review the live console in CloudHub for the app to see the logs for the request and response.


If you go back and look at the flow in Anypoint Studio, it’s a surprisingly simple flow that handles the requests and response from the Facebook Messenger platform. The benefit of using MuleSoft is the ability to route messages to other platforms without having to write code. A keyword for ‘orders’ can be routed to Salesforce to retrieve a list of orders for the user using the Salesforce Connector. Try out the code and let me know if you run into any issues or create a useful Bot for Facebook.

Using the DropBox API in the MuleSoft Anypoint Platform


MuleSoft has hundreds of connectors to numerous platforms which take away the need to write custom code. You can find them along with templates and examples on the Anypoint Exchange. The connectors generally wrap the platform APIs and expose all the available operations making it easier to setup integrations without any custom code. But what if the architecture of the flow prevents you from leveraging the connectors?

I recently ran into this scenario with a project that involved uploading files from a headless server to DropBox. Using the MuleSoft DropBox connector which wraps the REST API seemed like a no-brainer at first; but I ran into an architectural issue with the deployment. The initial design below shows how simple the project is, especially without the need to write any code.

The first flow (1) provides an HTTP endpoint where the user can be authenticated through OAuth to retrieve the access token from DropBox. Once authenticated, the second flow (2) can begin to upload files to DropBox. More information about how MuleSoft handles the OAuth process and storage of the token can found on this blog post. The design of this project works as expected but the issue is around the deployment. When deployed to a headless server, there’s no easy way to authenticate the application through the OAuth flow. Additionally, each re-deployment requires the application to be authenticated again which takes away the autonomy of the solution.

So I took a different approach to my project and started looking at the calling the DropBox REST API directly for uploading files. When setting up your application in DropBox, you can generate an access token which you can store and use without going through the OAuth flow.

Within my original project design, I removed the flow to authenticate the user through OAuth. I also replaced the DropBox connector in the second flow with the HTTP connector which allows me to easily make a REST API call. You can see the new flow below.

Setting up the HTTP Connector

For the HTTP connector to make the REST API call, setup requires configuring the request to make an HTTPS call to DropBox. The URL structure looks like the following:

So the request configuration should look like the image below on the General tab. Be sure to set the Protocol: field to HTTPS and set the Port: to 443

The TLS/SSL tab also needs to be configured when making an HTTPS call. More information on how to set that up can be found here.

Back on the configuration screen, set the Method field to PUT and the Path field to the following:


The last step is setting up the header parameters to pass the access token to the API. Under the Parameters section, click on Add Parameter and select header from the drop-down. Set the Name: to Authorization and set the Value: field to Bearer<YOUR_ACCESS_TOKEN_HERE>.

I’ve included a snipped of the Mule XML below:


As you can see, making a REST API call is just as easy as using the pre-built connectors in the Anypoint Platform. Architectural and functional design considerations generally drive the decision on which approach is better for the scenario.

I had the option of storing the DropBox access token into an Object Store and referencing the token from the DropBox connector using the accessTokenId but since my project only needs to upload files, I took the route of using the HTTP connector and making the REST API call directly.

Download all Objects from an Amazon S3 Bucket using MuleSoft

Here’s a quick example on how you can download all the Objects from an Amazon S3 Bucket using the Amazon S3 Connector.  You can see the flow and the corresponding code (minus the HTTP and Amazon S3 Configuration) below.

Mule Flow Screenshot
Flow to download all files from an S3 Bucket

Some prerequisites before you begin. You’ll need the following:

Let me highlight some of the key areas in the code.

  • When you run the project, you’ll kick off the process by navigating to http://localhost:8081/bucket. This is set in the HTTP Listener.
  • If you’ve pasted my code above into your project, you’ll need to configure the Amazon S3 configuration and the HTTP configuration
  • For the S3 Connector, you’ll need your Amazon AWS Access Key and Secret Key when you set up the configuration. Be sure to click on Test Connection before hitting OK.

    Amazon S3 Global Configuration
    Amazon S3 Global Configuration
  • The first Amazon S3 Connector grabs a list of all the Objects with in a Bucket. The field Bucket Name is hard-coded. You can change this to be dynamic and use a message.inboundProperties value instead.
  • You’ll need to open the properties for the File Connector and set the Path to a local directory before running the project.

With this example, you can see how easy it is to integrate with Amazon S3 without having to write any custom code. The connector provides a wrapper around the Amazon SDK to allow you the ability easily click-and-configure a project.

Debugging DataWeave output in Studio

One of the greatest features of Anypoint Studio is the Visual Design Debugger and often times you want to see the payload as a string during debugging. Generally using #[payload:java.lang.String] or #[message.payloadAs(java.lang.String)] does the trick. But when it comes to DataWeave, those methods don’t work if the output is anything other than ‘application/java‘ due to the nature of the payload data type:

DataWeave Output Payload

So the question is, “How do you debug the output coming out of DataWeave?” In this post, I’ll cover that question with a couple scenarios. Keep in mind that these scenarios are just workarounds. It would be ideal to just debug the flow without adding in additional components.


If your output is set to application/json, you can drop in a JSON to Object transformer after the Transform Message component before you start debugging. In doing so, you can view the JSON that’s returned by DataWeave as you step past the component. You will of course need to remove that component after debugging unless you need the data in that format.

JSON to Object
JSON to Object


If your output is set to application/xml, before you start debugging, you can drop in a XML to DOM transformer after the Transform Message component. Once you step past the component, you can use Evaluate Mule Message (Shift + Command + I) and enter the expression #[payload:java.lang.String] to see the resulting XML from DataWeave. Again, you’ll need to remove that component after debugging unless you need the data in that format.



If your output is set to application/csv, before you start debugging, you can drop in an Object to String transformer after the Transform Message component. Unlike the application/json or application/xml, this will automatically show you the expected data of the payload in the Mule Debug window. Again, you’ll need to remove that component after debugging unless you need the data in that format.


As you can see, the general workaround is drop in a component that converts the DataWeave output to an alternative format. By doing so, you can view the payload in Mule Debugger or use the #[payload:java.lang.String] method. Hopefully the next version of DataWeave and the Transform Component will make this easier.