In Cloudera deployments on public cloud, one of many key configuration components is the DNS. Get it incorrect and your deployment might turn out to be wholly unusable with customers unable to entry and use the Cloudera knowledge providers. If the DNS is ready up much less supreme than it may very well be, connectivity and efficiency points might come up. On this weblog, we’ll take you thru our tried and examined greatest practices for establishing your DNS to be used with Cloudera on Azure.
To get began and provide you with a really feel for the dependencies for the DNS, in an Azure deployment for Cloudera, these are the Azure managed providers getting used:
- AKS cluster: knowledge warehouse, knowledge engineering, machine studying, and Information movement
- MySQL database: knowledge engineering
- Storage account: all providers
- Azure database for PostgreSQL DB: knowledge lake and knowledge hub clusters
- Key vault: all providers
Typical buyer governance restrictions and the impression
Most Azure customers use non-public networks with a firewall as egress management. Most customers have restrictions on firewalls for wildcard guidelines. Cloudera sources are created on the fly, which implies wildcard guidelines could also be declined by the safety staff.
Most Azure customers use hub-spoke community topology. DNS servers are normally deployed within the hub digital community or an on-prem knowledge middle as a substitute of within the Cloudera VNET. Which means if DNS just isn’t configured appropriately, the deployment will fail.
Most Cloudera prospects deploying on Azure enable the usage of service endpoints; there’s a smaller set of organizations that don’t enable the usage of service endpoints. Service endpoint is a less complicated implementation to permit sources on a non-public community to entry managed providers on Azure Cloud. If service endpoints should not allowed, firewall and personal endpoints would be the different two choices. Most cloud customers don’t like opening firewall guidelines as a result of that may introduce the danger of exposing non-public knowledge on the web. That leaves non-public endpoints the one choice, which may also introduce extra DNS configuration for the non-public endpoints.
Connectivity from non-public community to Azure managed providers
Firewall to Web
Route from firewall to Azure managed service endpoint on the web straight.
Service endpoint
Azure offers service endpoints for sources on non-public networks to entry the managed providers on the web with out going by way of the firewall. That may be configured at a subnet stage. Since Cloudera sources are deployed in numerous subnets, this configuration should be enabled on all subnets.
The DNS data of the managed providers utilizing service endpoints shall be on the web and managed by Microsoft. The IP handle of this service shall be a public IP, and routable from the subnet. Please check with the Microsoft documentation for element.
Not all managed providers assist providers endpoint. In a Cloudera deployment state of affairs, solely storage accounts, PostgreSQL DB, and Key Vault assist service endpoints.
Fortuitously, most customers enable service endpoints. If a buyer doesn’t enable service endpoints, they should go together with a non-public endpoint, which has similarities to what must be configured within the following content material.
Personal Endpoint
There’s a community interface with a non-public IP handle created with a non-public endpoint, and there’s a non-public hyperlink service related to a selected community interface, in order that different sources within the non-public community can entry this service by way of the non-public community IP handle.
The important thing right here is for the non-public sources to discover a DNS resolve for that non-public IP handle. There are two choices to retailer the DNS document:
- Azure managed public DNS zones will at all times be there, however they retailer several types of IP addresses for the non-public endpoint. For instance:
- Storage account non-public endpoint—the general public DNS zone shops the general public IP handle of that service.
- AKS API server non-public endpoint—the general public DNS zone shops the non-public IP of that service.
- Azure Personal DNS zone: The DNS data shall be synchronized to the Azure Default DNS of LINKED VNET.
Personal endpoint is eligible to all Azure managed providers which can be utilized in Cloudera deployments.
As a consequence, for storage accounts, customers both use service endpoints or non-public endpoints. As a result of the general public DNS zone will at all times return a public IP, the non-public DNS zone turns into a compulsory configuration.
For AKS, these two DNS options are each appropriate. The challenges of personal DNS zones shall be mentioned subsequent.
Challenges of personal DNS zone on Azure non-public community
Essential Assumptions
As talked about above for the standard state of affairs, most Azure customers are utilizing a hub-and-spoke community structure, and deploy customized non-public DNS on hub VNET.
The DNS data shall be synchronized to Azure default DNS of linked VNET.
Easy Structure Use Instances
One VNET state of affairs with non-public DNS zone:
When a non-public endpoint is created, Cloudera on Azure will register the non-public endpoint to the non-public DNS zone. The DNS document shall be synchronized to Azure Default DNS of linked VNET.
If customers use customized non-public DNS, they will configure conditional ahead to Azure Default DNS for the area suffix of the FQDN.
Hub-and-spoke VNET with Azure default DNS:
With hub-spoke VNET and Azure default DNS, that’s nonetheless acceptable. The one drawback is that the sources on the un-linked VNET will be unable to entry the AKS. However since AKS is utilized by Cloudera, that doesn’t pose any main points.
The Problem Half
The preferred community structure amongst Azure shoppers is hub-spoke community with customized non-public DNS servers deployed both on hub-VNET or on-premises community.
Since DNS data should not synchronized to the Azure Default DNS of the hub VNET, the customized non-public DNS server can’t discover the DNS document for the non-public endpoint. And since the Cloudera VNET is utilizing the customized non-public DNS server on hub VNET, the Cloudera sources on Cloudera VNET will go to a customized non-public DNS server for DNS decision of the FQDN of the non-public endpoint. The provisioning will fail.
With the DNS server deployed within the on-prem community, there isn’t Azure default DNS related to the on-prem community, so the DNS server couldn’t discover the DNS document of the FQDN of the non-public endpoint.
Configuration greatest practices
In opposition to the background
Choice 1: Disable Personal DNS Zone
Use Azure managed public DNS zone as a substitute of a non-public DNS zone.
- For knowledge warehouse: create knowledge warehouses by way of the Cloudera command line interface with the parameter “privateDNSZoneAKS”: set to”None.”
- For Liftie-based knowledge providers: the entitlement “LIFTIE_AKS_DISABLE_PRIVATE_DNS_ZONE” should be set. Clients can request this entitlement to be set both by way of a JIRA ticket or have their Cloudera answer engineer to make the request on their behalf.
The only real downside of this feature is that it doesn’t apply to knowledge engineering, since that knowledge service will create and use a MySQL non-public DNS zone on the fly. There may be at current no choice to disable non-public DNS zones for knowledge engineering.
Choice 2: Pre-create Personal DNS Zones
Pre-create non-public DNS zones and hyperlink each Cloudera and hub VNETs to them.
The benefit of this method is that each knowledge warehouse and Liftie-based knowledge providers assist pre-created non-public DNS zones. There are nevertheless additionally a number of drawbacks:
- For Liftie, the non-public DNS zone must be configured when registering the atmosphere. As soon as previous the atmosphere registration stage, it can’t be configured.
- DE will want a non-public DNS zone for MySQL and it doesn’t assist pre-configured non-public DNS zones.
- On-premises networks can’t be linked to a non-public DNS zone. If the DNS server is on an on-prem community, there aren’t any workable options.
Choice 3: Create DNS Server as a Forwarder.
Create a few DNS servers (for HA consideration) with load balancer in Cloudera VNET, and configure conditional ahead to Azure Default DNS of the Cloudera VNET. Configure conditional ahead from the corporate customized non-public DNS server to the DNS server within the Cloudera subnet.
The downside of this feature is that extra DNS servers are required, which results in extra administration overhead for the DNS staff.
Choice 4: Azure-Managed DNS Resolve
Create a devoted /28 subnet in Cloudera VNET for Azure non-public DNS resolver inbound endpoint. Configure conditional ahead from customized non-public DNS to the Azure non-public DNS resolver inbound endpoint.
Abstract
Bringing all issues collectively, contemplate these greatest practices for establishing your DNS with Cloudera on Azure:
- For the storage account, key vault, postgres DB
- Use service endpoints as the primary selection.
- If service endpoint just isn’t allowed, pre-create non-public DNS zones and hyperlink to the VNET the place the DNS server is deployed. Configure conditional forwards from customized non-public DNS to Azure default DNS.
- If the customized non-public DNS is deployed within the on-premises community, use Azure DNS resolver or one other DNS server as DNS forwarder on the Cloudera VNET. Conditional ahead the DNS lookup from the non-public DNS to the resolver endpoint.
- For the information warehouse, DataFlow, or machine studying knowledge providers
- Disable the non-public DNS zone and use the general public DNS zone as a substitute.
- For the information engineering knowledge service
- Configure the Azure DNS resolver or one other DNS server as a DNS forwarder on the Cloudera VNET. Conditional ahead the DNS lookup from the non-public DNS to the resolver endpoint. Please check with Microsoft documentation for the main points of establishing an Azure DNS Personal Resolver.
For extra background studying on community and DNS specifics for Azure, take a look at our documentation for the assorted knowledge providers: DataFlow, Information Engineering, Information Warehouse, and Machine Studying. We’re additionally joyful to debate your particular wants; in that case please attain out to your Cloudera account supervisor or get in contact.