Virtual machines often use friendly names rather than just IP addresses, which requires the creation of a DNS record. This is primarily because many virtual machines are treated as "pets" rather than "cattle," which requires that a unique identity, including a network identity, be assigned.
Organizations operating a fleet of virtual machines (VMs) often establish a naming convention for virtual machines along with a process for assigning DNS records to new instances. These actions are typically one component of a broader VM lifecycle automation, encompassing both provisioning and decommissioning.
Red Hat OpenShift Virtualization lets you manage virtual machines declaratively, often through GitOps. This article explores methods for declaratively assigning DNS records to these virtual machines.
Requirements
To manage DNS records that are associated with VMs in a declarative fashion, there are several items you must consider. These virtual machines are typically exposed to the network on VLANs. Aside from VLANs, externally addressable user defined networks (UDNs), which use BGP to announce routes, are another option for providing virtual machines access to the network.
The IPs that these virtual machines receive might or might not be known in advance, but this solution should work with either approach.
Finally, virtual machines can be exposed on multiple networks, and therefore be available at multiple IPs and consequently require multiple DNS records.
So, in summary:
- Virtual machines can be exposed on multiple networks.
- IPs can be assigned statically or dynamically.
- Some of the assigned IPs to a virtual machine might need to have associated DNS records.
Design
We want the user experience to be as simple as possible. Ideally, adding labels and annotations on the VirtualMachine custom resource should be enough to declaratively create DNS records. The following is an approach from the user experience point of view that can be applied to achieve the envisioned goal:
A label, such as vm.redhat-cop.dns-name: "true", can be used to declare that DNS records should be assigned to the VM.
Annotations can then be added to specify FQDNs that should be associated with this virtual machine. The annotations will take this form:
vm.redhat-cop.dns-name/<interface-name>: <FQDN>In this case, interface-name is the name of the network interface whose IP needs to be associated with the specified FQDN. This combination allows for assigning distinct FQDNs to multiple network interfaces.
Now that the user experience side of the design is defined, let’s focus on how the system will actually work. The diagram in Figure 1 depicts the steps and components involved.
First, a VirtualMachine with the previously described labels and annotations is created in the spec.template section of the custom resource. The instantiation of a VirtualMachine generates a VirtualMachineInstance custom resource with the same labels and annotations.
Given the way the virtual machine API works, the actual values of the IPs configured on the network interfaces are known only at runtime within the status of the VirtualMachineInstance. This also requires the qemu-guest-agent to be correctly installed in the virtual machine OS.
A policy controller monitors VirtualMachineInstance for certain attributes. When there is a match between the IP record on the status and the value of the annotation a DNSEndpoint Custom Resource is created.
The DNSEnpoint itself is monitored by the ExternalDNS controller, which, if correctly configured (out of scope for this article), will update the DNS server with the desired DNS records.
As an example, let’s suppose we have a VirtualMachine with this snippet:
spec:
template:
metadata:
annotations:
vm.redhat-cop.dns-name/enp1s0: vm.example.com
labels:
vm.redhat-cop.dn-name: "true"Assuming that at runtime the VirtualMachineInstance receives an IP, this should generate the following DSNEndpoint:
apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
name: fedora-moccasin-tahr-60-enp1s0
namespace: test-vm
spec:
endpoints:
- dnsName: vm.example.com
recordTTL: 180
recordType: A
targets:
- 10.136.0.178The final decision left unanswered to this point is the specific policy controller that will be used. Two frameworks will be shown: Kyverno and Red Hat Advanced Cluster Management for Kubernetes, which in the end, will achieve the same result.
Kyverno-based approach
Kyverno is a cloud-native policy management framework that has been donated to the Cloud Native Computing Foundation (CNCF). It is lightweight and relatively easy to use. However, as of the time of this article, Kyverno is not supported by Red Hat.
Our policy will need to react based on the existence of certain conditions (i.e., VirtualMachineInstance with certain labels and annotations) and when this situation occurs, one or more objects will be created in response (i.e., DNSEndpoints for this use case). Such a policy can be written similar to the following:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: create-vm-dns-endpoints
annotations:
policies.kyverno.io/title: DNS Entries for Virtual Machines
policies.kyverno.io/category: external-dns, virtual-machines, dns
policies.kyverno.io/subject: DNSEndpoints, VirtualMachinesInstance
policies.kyverno.io/minversion: 1.6.0
policies.kyverno.io/description: >-This policy creates DNSEndpoints for external-dns to manage DNS entries for virtual machines.
spec:
rules:
- name: generate-dns-endpoints
match:
any:
- resources:
kinds:
- kubevirt.io/v1/VirtualMachineInstance
selector:
matchLabels:
vm.redhat-cop.dns-name: "true"
generate:
generateExisting: true
synchronize: true
orphanDownstreamOnPolicyDelete: true
foreach:
- list: request.object.status.interfaces
preconditions:
any:
- key: '{{ request.object.metadata.annotations."vm.redhat-cop.dns-name/{{ element.name }}" || "" }}'
operator: NotEquals
value: ""
apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
name: '{{request.object.metadata.name}}-{{element.name}}'
namespace: "{{request.object.metadata.namespace}}"
data:
spec:
endpoints:
- dnsName: '{{ request.object.metadata.annotations."vm.redhat-cop.dns-name/{{ element.name }}" }}'
recordTTL: 180
recordType: A
targets:
- '{{element.ipAddress}}'This policy will generate a DNSEndpoint for each interface for which a DNS address has been requested.
Red Hat Advanced Cluster Management policies-based approach
Red Hat Advanced Cluster Management for Kubernetes is OpenShift's fleet management tool. It operates in a hub-and-spoke architecture, so it requires that a hub cluster be established in order to enforce policies on managed clusters. Advanced Cluster Management for Kubernetes contains a very expressive policy definition language primarily based on Golang templates.
An Advanced Cluster Management policy to satisfy our use case is shown here (this is a raw policy; the full policy definition, Placement, and Placement bindings have been omitted for brevity):
object-templates-raw: |-
{{ range (lookup "kubevirt.io/v1" "VirtualMachineInstance" "" "" "vm.redhat-cop.dns-name=true").items }}
{{ $vm := . -}}
- complianceType: mustonlyhave
objectDefinition:
apiVersion: externaldns.k8s.io/v1alpha1
kind: DNSEndpoint
metadata:
name: "{{ .metadata.name }}"
namespace: "{{ .metadata.namespace }}"
spec:
{{- if $vm.status.interfaces }}
endpoints:
{{- range $vm.status.interfaces -}}
{{ $annotationName := printf "%v%v" "vm.redhat-cop.dns-name/" .name -}}
{{ if get $vm.metadata.annotations $annotationName -}}
- dnsName: "{{ get $vm.metadata.annotations $annotationName }}"
recordTTL: 180
recordType: A
targets:
- "{{.ipAddress}}"
{{ end }}
{{ end }}
{{ else }}
endpoints: []
{{ end }}
{{ end }}Conclusion
This article explored several methods for how to declaratively assign FQDNs to virtual machine running in OpenShift Virtualization by using a combination of policy controllers and operators.
We started off by providing details related to how many organizations apply naming conventions (FQDNs) to virtual machines. We have not implemented this logic in our solution, but you can easily write more policies to allow only virtual machines with well-formed FQDNs. Admission policies are relatively easy to write. This exercise is left to the reader.
