Databricks Launches Private Network Gateway for Secure Serverless Connectivity

Details: Simplifying Network Access

The core problem PNG solves is the high-friction networking step required to connect Databricks serverless compute to on-premises databases, applications, and other internal systems. Previously, enabling such connectivity often necessitated intricate, per-resource configurations involving separate private endpoints, load balancers, and proxy virtual machines. This process demanded extensive coordination between networking, security, and administration teams, leading to significant delays and increased complexity. Databricks reports that over 75% of customers cited network configuration complexity as a primary reason for keeping specific workloads on classic compute rather than migrating to serverless.

With Private Network Gateway, users can now establish one connection into their private network. From this single point, existing network configurations, including routes, DNS, and firewall rules, dictate what serverless workloads can access. This approach scales by network, not by individual endpoint, significantly streamlining operations. Databricks serverless compute itself is designed to be fully managed, intelligent, and versionless, removing the need for users to manage clusters and infrastructure. PNG complements this by abstracting away the complexities of private network connectivity.

How PNG Works and Its Benefits

Setting up PNG involves delegating a single subnet within the user's virtual network to Databricks. The PNG then establishes a managed, encrypted gateway from the serverless compute environment into this designated subnet. This gateway is configured through a Network Connectivity Configuration (NCC), an existing account-level object used for serverless networking. Once attached to workspaces, supported serverless workloads can securely reach private destinations. Crucially, Databricks manages the gateway infrastructure, eliminating the need for users to deploy, patch, or scale gateway virtual machines within their own networks.

PNG offers two primary modes for traffic routing, providing flexibility and control. The specific_destinations mode allows users to precisely define fully qualified domain names (FQDNs) that traffic should traverse through PNG, while all other traffic follows its normal path. This is ideal for scenarios requiring granular control over resource access. Alternatively, the all_traffic mode routes all outbound traffic through the user's network, except for traffic that matches more specific Private Link or service endpoint paths. This mode is beneficial for comprehensive egress traffic inspection through existing firewalls or for routing traffic through customer-owned IP addresses. This additive approach ensures that PNG works alongside existing connectivity and egress controls without disruption.

Addressing the Serverless Networking Challenge

Classic compute environments, running within a customer's virtual network, inherently simplify access to private resources through standard network peering. Serverless, however, operates in a Databricks-managed network, necessitating explicit pathways for private resource access. While Serverless Private Link remains suitable for cloud-managed services like AWS S3 or Azure Data Lake Storage, accessing self-hosted resources such as databases, Kafka clusters, or internal APIs traditionally required a more complex, resource-specific setup. This often involved multiple deployments and significant cross-team coordination, creating a bottleneck for teams needing to connect serverless to numerous internal services across diverse network environments, including on-premises setups.

The impact of PNG is substantial for organizations looking to fully leverage Databricks serverless capabilities. It enables simplified ingestion from private databases using tools like Lakeflow Connect, allows AI agents to securely interact with private services, and permits serverless environments to access private package repositories like PyPI, Artifactory, or Nexus. Furthermore, it ensures a consistent, customer-owned source IP address for external SaaS applications and allows egress traffic to be routed through existing customer firewalls for native filtering and logging. This significantly reduces the operational overhead and accelerates the integration of serverless workloads into existing enterprise architectures.

Availability and Future Outlook

Private Network Gateway is currently available in Public Preview on Azure and in Private Preview on AWS. Databricks has indicated that support for Google Cloud is under active development and will be available soon. This phased rollout allows Databricks to gather feedback and refine the offering before a broader general availability. Organizations on Azure can begin exploring PNG immediately, while AWS customers can inquire about private preview enrollment through their Databricks account teams. The introduction of PNG marks a significant step towards making Databricks serverless compute a more accessible and integrated solution for a wider range of enterprise use cases, particularly those with stringent network security and connectivity requirements.

Customer testimonials highlight the transformative impact of PNG. Fabiano Seki, Tech Lead at Banco Bradesco, noted that PNG has "significantly simplified how we connect serverless compute to data sources on Azure and on-premises — no more standing up a private link service and load balancer for every connection." Similarly, Primož Gale, Lead Technical Architect at NLB, stated that PNG "simplified the whole process, reduced operational overhead, and helped us run jobs faster." These endorsements underscore the practical benefits and ease of integration that PNG brings to Databricks serverless operations, making it a compelling solution for modern data platforms.