Introduction : What did we have till now on Azure? As part of auto-deployment or long-running clusters, we are habituated to work with azure-VMs that have HDP installed, aka HDInsight clusters. These provide more control for configuration and compute capacity management. The cluster is fairly static and auto-scaling feature if rudimentary where only data-nodes of a pre-specified size can be added/removed manually. Quick overview of azure offerings and the scale for ease-of-use and reduced administration (read cluster control) What is this Azure-Databricks now? -Imagine a world with no hadoop and a holistic data-compute architecture which decouples storage and compute for cloud based applications. I think, you are now imagining azure-databricks. Built for spark-on-cloud and just that, azure-databricks serves as a good start for compute-only (ephemeral, if you may) clusters. A snippet from the azure-blob on the benefits of using this offering: It differs from HDI