Connect to Azure Data Lake Store using MuleSoft

Overview

Azure Data Lake Store is an enterprise-wide hyper-scale repository for big data analytic workloads. Azure Data Lake enables you to capture data of any size, type, and ingestion speed in one single place for operational and exploratory analytics.

This guide shows how easy it is to connect to Azure Data Lake Store using MuleSoft by using the webHDFS REST APIs that are exposed. Azure Data Lake Store is essentially a version of HDFS. While MuleSoft has a connector for HDFS, Azure requires authentication through Active Directory using OAuth. The guide below will show you how to setup an Azure Data Lake Store, setup an Azure Active Directory app, and connect to Azure using a pre-configured project in MuleSoft Anypoint Studio.

Requirements

  • Microsoft Azure account
  • MuleSoft Anypoint Studio

Create an Azure Data Lake Store

If you don’t already have an Azure Data Lake Store, these are the steps to setup one yourself. It does require that you have an Azure account. You can sign up for a 30 day account or just use the Pay-As-You-Go account which is what I used.

  1. Login to the Azure portal and click on All services on the left hand navigation bar.
  2. In the search field, type in data lake and click on Data Lake Store

  3. In Data Lake Store, click on Create Data Lake Store in the center or click on Add in the top left.
  4. In the New Data Lake Store window, enter the following information:
    • Name: mulesoft
    • Subscription: This can depend on your account. If you recently signed up, you should have a 30 day subscription. I selected Pay-As-You-Go
    • Resource group:
      Either create a new resource group or use an existing one. (e.g. mulesoft)
    • Location: leave the default East US 2
    • Encryption settings: I set this to Do not enable encryption.
  5. Click on Create
  6. Once the Data Lake Store is created, click on Data explorer in the left hand navigation menu. We want to grant access to all users for this demo, so click on the Access button.
  7. In the Access screen, check the Read, Write, and Execute checkboxes under Everyone else and then click on Save

Create an Azure Active Directory “Web” Application

    1. In the left hand navigation menu, click on Azure Active Directory. If the menu item isn’t there, click on All services and search for it.
    2. In the navigation menu for Azure Active Directory, click on App registrations
    3. Click on New application registration
    4. In the Create window, enter the following data:
      • Name: mule
      • Application type: Keep the default Web app / API
      • Sign-on URL: Just enter http://localhost:8081. This can be changed later and doesn’t affect anything in this demo.
      • Click on Create
        Once the app has been created, copy down the Application ID: e.g. bdcabff5-af3c-4127-b69b-38bcf1792bfd
    5. Next, click on Settings and then click on Required permissions
    6. In the Required permissions screen, click on Add and then click on Select an API. Then select Azure Data Lake from the list of available APIs and then click on Select
    7. In the Enable Access screen, check the Have full access to the Azure Data Lake service checkbox and then click on Select
    8. Next we need to generate a key. Click on Keys on the left hand navigation bar for the app settings.
    9. In the Keys window, enter the following:
      • Description: mule-app-key
      • Expires: Set this to Never expires
      • And then click on Save. A value will appear for the key. Copy that value down. e.g. +zAbZQgXomvqsfHgCH32Yv+VCvkT3ZcxRyw5CWaw4dw=
    10. Next, let’s get your tenant ID. In Azure Active Directory click on Properties in the left hand navigation bar.
    11. Copy down the Directory ID value. That’s your tenant ID that you’ll need to make the OAuth call. e.g. 57744783-79ff-49ab-b27e-26245d4d97ef

 

Download and Run the Mule Project

  1. Download the project from Github https://github.com/djuang1/azure-data-lake-store
  2. Import the project into Anypoint Studio
  3. Open the mule-app.properties and modify the following properties:
    • adls.host: <name of your Data Lake Store>.azuredatalakestore.net
    • oauth.path: <tenant ID from Step 2.11>/oauth2/token
    • adls.client_id: This is the ID from Step 2.5
    • adls.client_secret: This is the key from Step 2.9
  4. The first flow will list the files and folders from the root directory of the Data Lake Store. Once the flow receives the request it creates the parameters to be sent to the OAuth request. If the OAuth request is successful, it returns an access token that is used to make the HDFS request to list the folders.
  5. The second flow shows how to upload data to the Data Lake Store. Similar to the first flow, it makes an OAuth request and passes the access token to make the HDFS request.
  6. Run the project and open up Postman.
  7. Let’s test the first flow. Paste the following into the request URL: http://localhost:8081/liststatus and click Send. The screenshot below shows the results. If you add some folders and files, you’ll receive more data from the API call.
  8. Open another tab and paste the following URL: http://localhost:8081/create?name=list2.txt
  9. Change the method to PUT. Under the Body section, select the binary radio button and select the file name list.txt from the src/main/resources folder from the project. Click on Send.
  10. If successful, when you switch back the Data Lake Store, you should see the file at the root directory. Click to open the file.
  11. You should see the following data in the file if everything was configured successfully.

Create a Facebook Messenger Bot using MuleSoft and the Anypoint Platform.

With over 900 million users a month, the Facebook Messenger platform has a pretty big audience.  The introduction of Messenger Bots earlier this year helped extend this adoption by helping drive personalized, scaled experiences with customers. Instead of building an app, just build the message thread as the place where you connect and interact with your users.

I was curious though to see how easy it was to build my own Bot using MuleSoft and the Anypoint Platform. Instead of writing custom code and running my own dedicated server, I leveraged Anypoint Studio to configure an app and CloudHub for my server needs. The following is a walk-through that shows you how you can create your own Facebook Messenger Bot.

Setup Facebook App and Page

To get started, setup a new app in Facebook under the developers site: https://developers.facebook.com/apps/

Add New App for Messenger

Be sure to select Apps for Messenger under the Category dropdown before clicking Create App ID. 

Once the app gets created, scroll down to the Token Generation section and click on Create a new pageYou’ll be re-directed to the Facebook site to create a new page.

Create a new page

Select any category (I selected Brand or Product > Software) and give it a name (e.g. MuleSoft Demos). Skip through the setup steps and jump back to the Facebook Developer site. Once you refresh the page, the new Facebook Page with show under the Token Generation section under the Page dropdown.

Select your new page and it will create a Page Access Token. You’ll need this for your Mule app to call the Send API later.

Page Access Token

Deploy App to CloudHub

Next, deploy the example app to CloudHub. You can download a copy of the project from Github here: https://github.com/djuang1/mule-facebook-bot and import it into Anypoint Studio.

Before you deploy it to CloudHub, open the the mule-app.properties file and copy and paste the Page Access Token to the access_token variable.

access_token

The example app is built to demonstrate a few of the the required event listeners for the Facebook Messenger platform. It handles the webhooks for messages and postbacks. Messages are parsed for keywords such as ‘template’, ‘button’, etc.. and all other messages are echoed back to the user using the Send API.

When the app is deployed and running in CloudHub, navigate to the app URL (e.g http://facebook-bot.cloudhub.io/) and test out the webhook challenge using the following URL:

https://<app_name>.cloudhub.io/webhook?hub.challenge=1234

You should get the challenge returned like the following screenshot:

Setup Webhooks

With the challenge returned successfully, we can setup the webhooks in Facebook now. Switch back to the Facebook Developers site and click on Setup Webhooks under the Webhooks section.

Enter the app URL under Callback URL with the path webhook. Enter any text under the Verify Token field. Under the Subscription Fields section, check messages and messaging_postbacks and then click on Verify and Save.

Once the webhook setup is complete, select the Facebook Page that was created previously and click on Subscribe in the Webhooks section.

Test Messages

Everything should be setup and running now on both CloudHub and Facebook. Let’s go ahead and test out the messages. Switch back to Facebook and open up the Page you created. You should see a button that says Send Message. Mouseover that button and then click on Test Button.

Go ahead and test out various messages and see their responses. When you type in messages that contain keywords such as button or template, you’ll see specific message formats returned.

You can also review the live console in CloudHub for the app to see the logs for the request and response.

Summary

If you go back and look at the flow in Anypoint Studio, it’s a surprisingly simple flow that handles the requests and response from the Facebook Messenger platform. The benefit of using MuleSoft is the ability to route messages to other platforms without having to write code. A keyword for ‘orders’ can be routed to Salesforce to retrieve a list of orders for the user using the Salesforce Connector. Try out the code and let me know if you run into any issues or create a useful Bot for Facebook.