Fire Insights integrates with Databricks. It submits jobs to the Databricks clusters using the REST API of Databricks and have the results displayed back in Fire Insights.
Fire also fetches the list of Databases and Tables from Databricks, making it easier for the user to build their workflows and execute them. In addition fire displays the list of Databricks clusters running for the user.
Databricks can be running on Azure or on AWS.
- Running Databricks on Azure : https://docs.microsoft.com/en-us/azure/azure-databricks/quickstart-create-databricks-workspace-portal
- Running Databricks on AWS : https://databricks.com/aws
Install Fire Insights on any machine. The machine has to be reachable from the web via a public IP or DNS entry. Then connect Fire Insights to your Databricks Cluster
Upload Fire Core Jar to Databricks¶
Fire Insights jar has to be uploaded to Databricks. Fire Insights jobs running on Databricks make use of this jar file.
fire-x.y.z/fire-core-lib/fire-spark_2_3-core-3.1.0-jar-with-dependencies.jar to Databricks. Upload it under Workspace as a Library on to Databricks.
workspace in the left side pane¶
workspace, Click on
Create & Library¶
fire-spark_2_3-core-3.1.0-jar-with-dependencies.jar from localmachine on Clicking on
Drop JAR here¶
fire-spark_2_3-core-3.1.0-jar-with-dependencies.jar uploaded, Click on
- Check the box with
Install automatically on all clusters, Inorder to avoid uploading manually to every cluster.
Configure the Uploaded Library in Fire Insights¶
Configure the path of the uploaded fire core jar library in Databricks in Fire Insights.
This has to be done under Administration/Configuration.
app.postMessageURL to be the public IP of the machine. It has to be accessible from the Databricks Cluster.
Install Databricks JDBC Driver¶
Fire needs the Databricks JDBC Driver to be installed. Install it in the
fire-user-lib folder of the Fire installation.
You can download the Databricks JDBC Driver from the Databricks site :
The driver is available as a zip file. eg:
- Unzip the downloaded file. It will create a directory like
- Copy the jdbc jar file named
Create your REST API token in Databricks¶
Create your token in Databricks. It would be used in making REST API calls to Databricks from Fire Insights.
Login to your Databricks Account¶
Account icon in right corner top¶
Generate New Token¶
Lifetime(days) for token expiry & Click on
Copy the token generated and Click on
Create Databricks Connection¶
Create a connection in Fire Insights under your Application for connecting to Databricks.
- Specify your Databricks Token.
- Specify the Databricks JDBC URL of your cluster in Databricks.
Now we are ready to start using the Databricks Connection in workflows etc.