Databricks Archives - The Data Guy

Category: Databricks

Bulk load to Azure SQL Database with Structured Streaming

Posted on April 9, 2020April 9, 2020 by benjaminleroux

As you may have noticed, my go to language is Scala when it comes to Azure Databricks and Spark. With that said, Scala has a great library to read and write data to Azure SQL Database. Along with some great...Read More

Connect Azure Databricks to SQL Database & Azure SQL Data Warehouse using a Service Principal

Posted on May 3, 2019May 8, 2019 by benjaminleroux

Azure has recently added the ability to authenticate to Azure SQL Database and Azure SQL Data Warehouse using Azure Active Directory. This allows you to centrally manage identity to your database. Azure Active Directory (AAD) authentication As per Microsoft documentation,...Read More

Running an Azure Databricks notebook in a CI/CD release stage

Posted on April 23, 2019May 8, 2019 by benjaminleroux

What happens if you have multiple environment? Or, what happens if you need to make sure your notebook passes unit testing before making its way to production? In my previous post we saw how you can use Azure DevOps to...Read More

CI/CD with Databricks and Azure DevOps

Posted on January 18, 2019May 8, 2019 by benjaminleroux

So you've created notebooks in your Databricks workspace, collaborated with your peers and now you're ready to operationalize your work. This is a simple process if you only need to copy to another folder within the same workspace. But what...Read More

Auto Process files added to Azure Storage Account using Databricks

Posted on December 5, 2018May 8, 2019 by benjaminleroux

Have you ever wanted to process in near real time new files added to your Azure Storage account (BLOB)? Have you tried using Azure EventHub but files are too large to make this a practical solution? Let me present you...Read More

Managing passwords and secrets in Azure Databricks with KeyVault

Posted on December 5, 2018May 8, 2019 by benjaminleroux

Continuing on my previous post, changes in the integration between Databricks and Azure have made things even easier... I'm referring to secrets within Databricks and Azure KeyVault. First, you will need to create a KeyVault resource within Azure. The following...Read More