Connect to Azure Data Lake Store using MuleSoft


Azure Data Lake Store is an enterprise-wide hyper-scale repository for big data analytic workloads. Azure Data Lake enables you to capture data of any size, type, and ingestion speed in one single place for operational and exploratory analytics.

This guide shows how easy it is to connect to Azure Data Lake Store using MuleSoft by using the webHDFS REST APIs that are exposed. Azure Data Lake Store is essentially a version of HDFS. While MuleSoft has a connector for HDFS, Azure requires authentication through Active Directory using OAuth. The guide below will show you how to setup an Azure Data Lake Store, setup an Azure Active Directory app, and connect to Azure using a pre-configured project in MuleSoft Anypoint Studio.


  • Microsoft Azure account
  • MuleSoft Anypoint Studio

Create an Azure Data Lake Store

If you don’t already have an Azure Data Lake Store, these are the steps to setup one yourself. It does require that you have an Azure account. You can sign up for a 30 day account or just use the Pay-As-You-Go account which is what I used.

  1. Login to the Azure portal and click on All services on the left hand navigation bar.
  2. In the search field, type in data lake and click on Data Lake Store

  3. In Data Lake Store, click on Create Data Lake Store in the center or click on Add in the top left.
  4. In the New Data Lake Store window, enter the following information:
    • Name: mulesoft
    • Subscription: This can depend on your account. If you recently signed up, you should have a 30 day subscription. I selected Pay-As-You-Go
    • Resource group:
      Either create a new resource group or use an existing one. (e.g. mulesoft)
    • Location: leave the default East US 2
    • Encryption settings: I set this to Do not enable encryption.
  5. Click on Create
  6. Once the Data Lake Store is created, click on Data explorer in the left hand navigation menu. We want to grant access to all users for this demo, so click on the Access button.
  7. In the Access screen, check the Read, Write, and Execute checkboxes under Everyone else and then click on Save

Create an Azure Active Directory “Web” Application

    1. In the left hand navigation menu, click on Azure Active Directory. If the menu item isn’t there, click on All services and search for it.
    2. In the navigation menu for Azure Active Directory, click on App registrations
    3. Click on New application registration
    4. In the Create window, enter the following data:
      • Name: mule
      • Application type: Keep the default Web app / API
      • Sign-on URL: Just enter http://localhost:8081. This can be changed later and doesn’t affect anything in this demo.
      • Click on Create
        Once the app has been created, copy down the Application ID: e.g. bdcabff5-af3c-4127-b69b-38bcf1792bfd
    5. Next, click on Settings and then click on Required permissions
    6. In the Required permissions screen, click on Add and then click on Select an API. Then select Azure Data Lake from the list of available APIs and then click on Select
    7. In the Enable Access screen, check the Have full access to the Azure Data Lake service checkbox and then click on Select
    8. Next we need to generate a key. Click on Keys on the left hand navigation bar for the app settings.
    9. In the Keys window, enter the following:
      • Description: mule-app-key
      • Expires: Set this to Never expires
      • And then click on Save. A value will appear for the key. Copy that value down. e.g. +zAbZQgXomvqsfHgCH32Yv+VCvkT3ZcxRyw5CWaw4dw=
    10. Next, let’s get your tenant ID. In Azure Active Directory click on Properties in the left hand navigation bar.
    11. Copy down the Directory ID value. That’s your tenant ID that you’ll need to make the OAuth call. e.g. 57744783-79ff-49ab-b27e-26245d4d97ef


Download and Run the Mule Project

  1. Download the project from Github
  2. Import the project into Anypoint Studio
  3. Open the and modify the following properties:
    • <name of your Data Lake Store>
    • oauth.path: <tenant ID from Step 2.11>/oauth2/token
    • adls.client_id: This is the ID from Step 2.5
    • adls.client_secret: This is the key from Step 2.9
  4. The first flow will list the files and folders from the root directory of the Data Lake Store. Once the flow receives the request it creates the parameters to be sent to the OAuth request. If the OAuth request is successful, it returns an access token that is used to make the HDFS request to list the folders.
  5. The second flow shows how to upload data to the Data Lake Store. Similar to the first flow, it makes an OAuth request and passes the access token to make the HDFS request.
  6. Run the project and open up Postman.
  7. Let’s test the first flow. Paste the following into the request URL: http://localhost:8081/liststatus and click Send. The screenshot below shows the results. If you add some folders and files, you’ll receive more data from the API call.
  8. Open another tab and paste the following URL: http://localhost:8081/create?name=list2.txt
  9. Change the method to PUT. Under the Body section, select the binary radio button and select the file name list.txt from the src/main/resources folder from the project. Click on Send.
  10. If successful, when you switch back the Data Lake Store, you should see the file at the root directory. Click to open the file.
  11. You should see the following data in the file if everything was configured successfully.

Update Document with a MongoDB Function using MuleSoft

I recently had to figure out how to use the MuleSoft MongoDB Connector and update a document with a function. This post is a quick walk-through showing how to build an example flow to address the use case.

In MongoDB, the collection we’ll be performing the function against looks like the following. It’s just a set of employee records. We’ll run a query against the emp_no field and update the sequence_value field using the function $inc.

Let’s get started, my environment for this post consisted of the following components all running locally on my Macbook.

  • Anypoint Studio
  • MongoDB – For testing purposes, I use mLab which provides a hosted Database-as-a-Service for MongoDB. You can found out more about the service here:

The first thing of course is to install the MongoDB Connector if you don’t already have it installed in Anypoint Studio. You can find it in Exchange. Here’s a link to the documentation on how to use the connector as well.

Once you have it installed, build the flow below using the following connectors like the screenshot below:

  • HTTP
  • Transform Message
  • MongoDB
  • Object to JSON

Next configure the HTTP connector with the default configuration (e.g. http://localhost:8081/)

Then click on the MongoDB connector and click on the Add button for the Connector Configuration and fill in the Username, Password, Database, and Servers (host:port) fields. You can test the connection and then click on OK.

Back on the Connector properties tab, setup the following:

  1. Select Update documents by function for the Operation field.
  2. Enter in a Collection name
  3. Enter in a Function. In our case we’ll be using $inc. I’m using one of the Field Update Operators that MongoDB provides to increment a field when a query matches a document. We’ll set the query in the Transform Message component after configuring the MongoDB connector.
  4. Set the Element Reference field to #[‘{sequence_value:1}’]. This will increment the sequence_value field by 1 when the query matches a document.
  5. Lastly, set the Query Reference field to #[flowVars.query]. This will contain the query from the DataWeave component.

The configuration for the MongoDB connector should look like the following:

Next, select the Transform Component and change the Output to a variable and set the Variable name to query.

Click OK and then paste the following script

I’ve hardcoded the emp_no to match my record (e.g. 10001) in my collection but you can set it to be dynamic and pass in a value when you make an HTTP request.

The final Mule XML for the flow should look like the following:

Go ahead and run your project now. When you make a request from your browser to http://localhost:8081, you’ll see the following response returned from the MongoDB connector.

The matchedCount will equal 1 showing that it matched a document in your collection. The modifiedCount signifies that a document was updated with the function.

Hopefully this post works for you. If you ran into any issues while setting this up or have any question, feel free to post to the comment section.

Deploy Domain Projects using the Mule Plugin for Maven

Continuous Integration (CI) is a DevOps development practice that requires developers to check code into a shared repository (e.g. Github) after which automated builds and tests are run. This process allows teams to detect problems early and locate them more easily.

The MuleSoft Anypoint Platform provides the flexibility and tools to easily build a CI process into your API Lifecycle or tie into an existing CI process using common tooling like Jenkins, Maven, TeamCity, etc…

The mule-maven-plugin is a tool that we provide that allows you to deploy Mule applications to different kinds of Mule Runtime servers: Standalone (both Community and Enterprise), Clustered, and CloudHub.

Recently I had to figure out how to deploy a Mule Domain project to a standalone Mule Runtime. I documented the process below.

My environment consisted of the following components all running locally on my Macbook.

In Studio, I created a ‘Mule Domain Project’. Be sure to check the Use Maven checkbox when you create the project. Also set the ‘Project Name’ as domain. This is important for the deployment to work for this particular exercise. I’ll go into detail why later.

In the domain, I just created an HTTP listener on port 8082 that would be shared by projects in the domain.

Next, I created a really simple ‘Hello World!’ project as part of the domain. Again, be sure the check the Use Maven checkbox when you create the project. The flow uses the shared HTTP listener resource as you can see in the code below.

For Maven to compile and deploy the project, we need to modify the pom.xml file and add the plugin.

You can use the code below. There are three configurations that are required for this to work.

  • <deploymentType> – This ensures that the project is deployed to a standalone Mule Runtime
  • <muleHome> – Setting this to the base directory of your Mule Runtime ensures that Maven doesn’t download and install a Mule standalone server.
  • <domain> – This is the location of the mule-domain-config.xml file. The plugin uses the name of the folder that the file sits in when it creates the domain on the Runtime server. This name needs to match the domain that the Mule projects are tied to. For this article, I created a domain called domain so I wouldn’t have to move the mule-domain-config.xml to a different folder that matched the Mule project.

Once everything is setup, you can execute the deployment by running the following command:

If configured successfully, Maven will create a domain and compile and deploy the Mule project to the apps folder. If the Mule Runtime is stopped, it will start it as well.

Hopefully you’re successful in setting this project up. Leave a comment if you run into any issues or have any questions. Good Luck!


Create a Facebook Messenger Bot using MuleSoft and the Anypoint Platform.

With over 900 million users a month, the Facebook Messenger platform has a pretty big audience.  The introduction of Messenger Bots earlier this year helped extend this adoption by helping drive personalized, scaled experiences with customers. Instead of building an app, just build the message thread as the place where you connect and interact with your users.

I was curious though to see how easy it was to build my own Bot using MuleSoft and the Anypoint Platform. Instead of writing custom code and running my own dedicated server, I leveraged Anypoint Studio to configure an app and CloudHub for my server needs. The following is a walk-through that shows you how you can create your own Facebook Messenger Bot.

Setup Facebook App and Page

To get started, setup a new app in Facebook under the developers site:

Add New App for Messenger

Be sure to select Apps for Messenger under the Category dropdown before clicking Create App ID. 

Once the app gets created, scroll down to the Token Generation section and click on Create a new pageYou’ll be re-directed to the Facebook site to create a new page.

Create a new page

Select any category (I selected Brand or Product > Software) and give it a name (e.g. MuleSoft Demos). Skip through the setup steps and jump back to the Facebook Developer site. Once you refresh the page, the new Facebook Page with show under the Token Generation section under the Page dropdown.

Select your new page and it will create a Page Access Token. You’ll need this for your Mule app to call the Send API later.

Page Access Token

Deploy App to CloudHub

Next, deploy the example app to CloudHub. You can download a copy of the project from Github here: and import it into Anypoint Studio.

Before you deploy it to CloudHub, open the the file and copy and paste the Page Access Token to the access_token variable.


The example app is built to demonstrate a few of the the required event listeners for the Facebook Messenger platform. It handles the webhooks for messages and postbacks. Messages are parsed for keywords such as ‘template’, ‘button’, etc.. and all other messages are echoed back to the user using the Send API.

When the app is deployed and running in CloudHub, navigate to the app URL (e.g and test out the webhook challenge using the following URL:


You should get the challenge returned like the following screenshot:

Setup Webhooks

With the challenge returned successfully, we can setup the webhooks in Facebook now. Switch back to the Facebook Developers site and click on Setup Webhooks under the Webhooks section.

Enter the app URL under Callback URL with the path webhook. Enter any text under the Verify Token field. Under the Subscription Fields section, check messages and messaging_postbacks and then click on Verify and Save.

Once the webhook setup is complete, select the Facebook Page that was created previously and click on Subscribe in the Webhooks section.

Test Messages

Everything should be setup and running now on both CloudHub and Facebook. Let’s go ahead and test out the messages. Switch back to Facebook and open up the Page you created. You should see a button that says Send Message. Mouseover that button and then click on Test Button.

Go ahead and test out various messages and see their responses. When you type in messages that contain keywords such as button or template, you’ll see specific message formats returned.

You can also review the live console in CloudHub for the app to see the logs for the request and response.


If you go back and look at the flow in Anypoint Studio, it’s a surprisingly simple flow that handles the requests and response from the Facebook Messenger platform. The benefit of using MuleSoft is the ability to route messages to other platforms without having to write code. A keyword for ‘orders’ can be routed to Salesforce to retrieve a list of orders for the user using the Salesforce Connector. Try out the code and let me know if you run into any issues or create a useful Bot for Facebook.

Using the DropBox API in the MuleSoft Anypoint Platform


MuleSoft has hundreds of connectors to numerous platforms which take away the need to write custom code. You can find them along with templates and examples on the Anypoint Exchange. The connectors generally wrap the platform APIs and expose all the available operations making it easier to setup integrations without any custom code. But what if the architecture of the flow prevents you from leveraging the connectors?

I recently ran into this scenario with a project that involved uploading files from a headless server to DropBox. Using the MuleSoft DropBox connector which wraps the REST API seemed like a no-brainer at first; but I ran into an architectural issue with the deployment. The initial design below shows how simple the project is, especially without the need to write any code.

The first flow (1) provides an HTTP endpoint where the user can be authenticated through OAuth to retrieve the access token from DropBox. Once authenticated, the second flow (2) can begin to upload files to DropBox. More information about how MuleSoft handles the OAuth process and storage of the token can found on this blog post. The design of this project works as expected but the issue is around the deployment. When deployed to a headless server, there’s no easy way to authenticate the application through the OAuth flow. Additionally, each re-deployment requires the application to be authenticated again which takes away the autonomy of the solution.

So I took a different approach to my project and started looking at the calling the DropBox REST API directly for uploading files. When setting up your application in DropBox, you can generate an access token which you can store and use without going through the OAuth flow.

Within my original project design, I removed the flow to authenticate the user through OAuth. I also replaced the DropBox connector in the second flow with the HTTP connector which allows me to easily make a REST API call. You can see the new flow below.

Setting up the HTTP Connector

For the HTTP connector to make the REST API call, setup requires configuring the request to make an HTTPS call to DropBox. The URL structure looks like the following:

So the request configuration should look like the image below on the General tab. Be sure to set the Protocol: field to HTTPS and set the Port: to 443

The TLS/SSL tab also needs to be configured when making an HTTPS call. More information on how to set that up can be found here.

Back on the configuration screen, set the Method field to PUT and the Path field to the following:


The last step is setting up the header parameters to pass the access token to the API. Under the Parameters section, click on Add Parameter and select header from the drop-down. Set the Name: to Authorization and set the Value: field to Bearer<YOUR_ACCESS_TOKEN_HERE>.

I’ve included a snipped of the Mule XML below:


As you can see, making a REST API call is just as easy as using the pre-built connectors in the Anypoint Platform. Architectural and functional design considerations generally drive the decision on which approach is better for the scenario.

I had the option of storing the DropBox access token into an Object Store and referencing the token from the DropBox connector using the accessTokenId but since my project only needs to upload files, I took the route of using the HTTP connector and making the REST API call directly.

Download all Objects from an Amazon S3 Bucket using MuleSoft

Here’s a quick example on how you can download all the Objects from an Amazon S3 Bucket using the Amazon S3 Connector.  You can see the flow and the corresponding code (minus the HTTP and Amazon S3 Configuration) below.

Mule Flow Screenshot
Flow to download all files from an S3 Bucket

Some prerequisites before you begin. You’ll need the following:

Let me highlight some of the key areas in the code.

  • When you run the project, you’ll kick off the process by navigating to http://localhost:8081/bucket. This is set in the HTTP Listener.
  • If you’ve pasted my code above into your project, you’ll need to configure the Amazon S3 configuration and the HTTP configuration
  • For the S3 Connector, you’ll need your Amazon AWS Access Key and Secret Key when you set up the configuration. Be sure to click on Test Connection before hitting OK.

    Amazon S3 Global Configuration
    Amazon S3 Global Configuration
  • The first Amazon S3 Connector grabs a list of all the Objects with in a Bucket. The field Bucket Name is hard-coded. You can change this to be dynamic and use a message.inboundProperties value instead.
  • You’ll need to open the properties for the File Connector and set the Path to a local directory before running the project.

With this example, you can see how easy it is to integrate with Amazon S3 without having to write any custom code. The connector provides a wrapper around the Amazon SDK to allow you the ability easily click-and-configure a project.

Debugging DataWeave output in Studio

One of the greatest features of Anypoint Studio is the Visual Design Debugger and often times you want to see the payload as a string during debugging. Generally using #[payload:java.lang.String] or #[message.payloadAs(java.lang.String)] does the trick. But when it comes to DataWeave, those methods don’t work if the output is anything other than ‘application/java‘ due to the nature of the payload data type:

DataWeave Output Payload

So the question is, “How do you debug the output coming out of DataWeave?” In this post, I’ll cover that question with a couple scenarios. Keep in mind that these scenarios are just workarounds. It would be ideal to just debug the flow without adding in additional components.


If your output is set to application/json, you can drop in a JSON to Object transformer after the Transform Message component before you start debugging. In doing so, you can view the JSON that’s returned by DataWeave as you step past the component. You will of course need to remove that component after debugging unless you need the data in that format.

JSON to Object
JSON to Object


If your output is set to application/xml, before you start debugging, you can drop in a XML to DOM transformer after the Transform Message component. Once you step past the component, you can use Evaluate Mule Message (Shift + Command + I) and enter the expression #[payload:java.lang.String] to see the resulting XML from DataWeave. Again, you’ll need to remove that component after debugging unless you need the data in that format.



If your output is set to application/csv, before you start debugging, you can drop in an Object to String transformer after the Transform Message component. Unlike the application/json or application/xml, this will automatically show you the expected data of the payload in the Mule Debug window. Again, you’ll need to remove that component after debugging unless you need the data in that format.


As you can see, the general workaround is drop in a component that converts the DataWeave output to an alternative format. By doing so, you can view the payload in Mule Debugger or use the #[payload:java.lang.String] method. Hopefully the next version of DataWeave and the Transform Component will make this easier.

Google Chrome CR-48

I came home the other day to find a box on my doorstep. It was a pretty nondescript except for a UPS label so I had no idea where it came from or who sent it. I was pleasantly surprised to find a Google Chrome CR-48, the predecessor to the upcoming Chromebooks which are slated to be released in mid-June. I signed up back in December but never expected to get one but apparently Google has recently started sending them out to according to news reports. Anyway, here are my initial thoughts from playing around with the device for the past couple days.


Keeping in mind that it’s only for beta testing, I didn’t expect much around the form factor and build quality of the… let’s call it a Chromebook. I use a MacBook Pro regularly which sets a pretty high standard around build quality. The Chromebook does take a lot of design stylings from the MacBook surprisingly. The chiclet keyboard, the size, trackpad (though usability isn’t even close). Entirely black, the case has a rubberized feel to it and doesn’t feel very sturdy, but again, it is a beta test unit. There are a minimal number of ports… VGA out, USB, headphone jack, and a SD card slot. Supposedly it has bluetooth, GPS, and support for Verizon but I haven’t gotten around to testing those features yet. The full hardware specs can be found here.

Operating System

So the main reason Google has been distributing these Chromebooks is to have users test Chrome OS. Ubiquitous wireless access, cheaper manufacturing costs, and most importantly the transition of traditional native applications to web-based applications that provide equal functionality has lead Google to believe that the traditional Operating System (Windows, Mac OS X, Linux, etc…) can be replaced with just a browser. Chrome OS is essentially a Linux variant that boots directly into the Google Chrome browser. Chrome has gained significant mainstream adoption in the past couple years and provides Google a stepping stone to increased adoption of the Chromebooks once they’re released in June. So essentially if you use Chrome today, the transition to using a Chromebook is seamless. And if you spend a considerable amount of time using Gmail, Google Docs, Google Calendar… you don’t really notice that the OS is missing. Boot time is fast and switching between users is quick and easy. With the Google Web Store finding and adding apps to your home tab gives you easy one-click access to your everyday apps.


While I spend a considerable amount of time using web applications today, my work requires me to use desktop applications to do software design and development. I haven’t found any web based application that can replace the ease and efficiency of their desktop counterparts so shifting my day-to-day use to just a Chromebook is not an option for me.

For a standard user, one that just reads e-mails, writes documents, and browses the web, the Chromebook should be an easy transition. Mainstream adoption will take some time though as people get used to the paradigm shift of moving their data into the cloud and learning how to navigate new apps.

What should be interesting in the next year is the response from competition. From Apple, it should be interesting to see what comes after iOS 5.0 and Mac OS X Lion. Supposedly the two operating systems are on a collision course. Throw in what people are speculating about the Apple iCloud and 2012 could be an innovative year for Apple. Microsoft on the other hand really needs to figure things out. If Google is successful with Chrome OS, it’ll greatly accelerate the decline of WIndows in the consumer marketplace even further. Microsoft has been touting Azure and the recently released Office 365 but I personally don’t know anyone that uses or raves about any of the Microsoft web applications.

Anyways, look for the new Chromebooks June 15th from Acer and Samsung. I think the Samsung version looks to be the better version of the two. It’s a little heaver but it has a longer battery life and a larger screen though the price may be higher from what I’ve heard.

Document object to String object (Java Code)

Converting a Document object to a String object in Java is an issue I’ve run into a couple times at work. I wanted to see the output of a WebService call response that was in XML in the console. Here’s a quick code snippet I found that handles this: