Skip to content

The Data Guy

Data is the new bacon

Bulk load to Azure SQL Database with Structured Streaming

Posted on April 9, 2020April 9, 2020 by benjaminleroux
As you may have noticed, my go to language is Scala when it comes to Azure Databricks and Spark. With that said, Scala has a great library to read and write data to Azure SQL Database. Along with some great...Read More
Posted in Azure SQL, Big Data, DatabricksLeave a comment

Connect Azure Databricks to SQL Database & Azure SQL Data Warehouse using a Service Principal

Posted on May 3, 2019May 8, 2019 by benjaminleroux
Azure has recently added the ability to authenticate to Azure SQL Database and Azure SQL Data Warehouse using Azure Active Directory. This allows you to centrally manage identity to your database. Azure Active Directory (AAD) authentication As per Microsoft documentation,...Read More
Posted in Azure SQL, Databricks, Uncategorized22 Comments

Running an Azure Databricks notebook in a CI/CD release stage

Posted on April 23, 2019May 8, 2019 by benjaminleroux
What happens if you have multiple environment? Or, what happens if you need to make sure your notebook passes unit testing before making its way to production? In my previous post we saw how you can use Azure DevOps to...Read More
Posted in Big Data, Databricks2 Comments

CI/CD with Databricks and Azure DevOps

Posted on January 18, 2019May 8, 2019 by benjaminleroux
So you've created notebooks in your Databricks workspace, collaborated with your peers and now you're ready to operationalize your work. This is a simple process if you only need to copy to another folder within the same workspace. But what...Read More
Posted in Big Data, DatabricksTagged Azure, Databricks, DevOps31 Comments

Auto Process files added to Azure Storage Account using Databricks

Posted on December 5, 2018May 8, 2019 by benjaminleroux
Have you ever wanted to process in near real time new files added to your Azure Storage account (BLOB)? Have you tried using Azure EventHub but files are too large to make this a practical solution?  Let me present you...Read More
Posted in Big Data, DatabricksTagged Azure, BLOB, Databricks, Streaming6 Comments

Managing passwords and secrets in Azure Databricks with KeyVault

Posted on December 5, 2018May 8, 2019 by benjaminleroux
Continuing on my previous post, changes in the integration between Databricks and Azure have made things even easier... I'm referring to secrets within Databricks and Azure KeyVault. First, you will need to create a KeyVault resource within Azure. The following...Read More
Posted in Big Data, DatabricksTagged Azure, Databricks, Keyvault, passwords, secrets2 Comments

About Me

Data Platform Solution Architect at Microsoft, I help companies define their roadmaps via solution architecture bridging big data, advanced analytics and business intelligence in the Azure cloud, on-premises or in a hybrid approach.

Recent Posts

  • Bulk load to Azure SQL Database with Structured Streaming April 9, 2020
  • Connect Azure Databricks to SQL Database & Azure SQL Data Warehouse using a Service Principal May 3, 2019
  • Running an Azure Databricks notebook in a CI/CD release stage April 23, 2019
  • CI/CD with Databricks and Azure DevOps January 18, 2019
  • Auto Process files added to Azure Storage Account using Databricks December 5, 2018

Admin stuff!

  • Log in
  • Entries feed
  • Comments feed
  • WordPress.org

  • Archives

    • April 2020 (1)
    • May 2019 (1)
    • April 2019 (1)
    • January 2019 (1)
    • December 2018 (2)

  • Categories

    • Azure SQL (2)
    • Big Data (5)
    • Databricks (6)
    • Uncategorized (1)

  • Search

- Avant theme, by Kaira