Azure Databricks Integration Steps¶
Fire Insights integrates with Databricks. It submits jobs to the Databricks clusters using the REST API of Databricks and have the results displayed back in Fire Insights.
Fire also fetches the list of Databases and Tables from Databricks, making it easier for the user to build their workflows and execute them. In addition Fire displays the list of Databricks clusters running for the user.
- Running Databricks on Azure : https://docs.microsoft.com/en-us/azure/azure-databricks/quickstart-create-databricks-workspace-portal
Below are the steps for Integrating Fire Insights with your Databricks Clusters.
Install Fire Insights¶
Install Fire Insights on any machine. The machine has to be reachable from the Databricks cluster.
Upload Fire Core Jar to Databricks¶
Upload Fire Insights jar to Databricks. Fire Insights jobs running on Databricks make use of this jar file.
fire-x.y.z/fire-core-lib/fire-spark_2_4-core-3.1.0-jar-with-dependencies.jar to Databricks. Upload it under Workspace as a Library on to Databricks.
1. Login to
2. Click on
workspace in the left side pane¶
3. Create a new Library¶
fire-spark_2_4-core-3.1.0-jar-with-dependencies.jar from your machine by Clicking on
Drop JAR here¶
fire-spark_2_4-core-3.1.0-jar-with-dependencies.jar is uploaded, click on
- Check the box with
Install automatically on all clusters, in order to avoid installing it manually to every cluster.
Configure the Uploaded Library in Fire Insights¶
Configure the path of the uploaded fire core jar library in Databricks in Fire Insights.
This has to be done under Administration/Configuration.
Configure app.postMessageURL in Fire Insights¶
app.postMessageURL to be the IP of the machine on which Fire Insights is installed. Jobs running on Databricks would post back results to Fire Insights using this URL.
Install Databricks JDBC Driver¶
Fire needs the Databricks JDBC Driver to be installed. Install it in the
fire-server-lib folder of the Fire installation.
You can download the Databricks JDBC Driver from the Databricks site :
The driver is available as a zip file. eg:
- Unzip the downloaded file. It will create a directory like
- Copy the jdbc jar file named
Create your REST API token in Databricks¶
Create your token in Databricks. It would be used in making REST API calls to Databricks from Fire Insights.
1. Login to your Databricks Account¶
2. Click on
Account icon in right corner top¶
3. Click on
4. Click on
Generate New Token¶
Lifetime(days) for token expiry & Click on
6. Copy the token generated. Click on
Create Databricks Connection in Fire Insights¶
Create a connection in Fire Insights to Databricks.
It can be created by the Administrator under Administration/Global Connections. These connections are available for everyone to use.
It can also be created by any user with their Application. In this case, it is only available to the Application and its users.
- Specify your Databricks Token.
- Specify the Databricks JDBC URL of your cluster in Databricks.
Now we are ready to start using the Databricks Connection in Fire Insights to:
- Browse DBFS
- View your Databricks Clusters
- Browse your Databricks Databases & Tables
- Create Workflows which Read from and Write to Databricks