At Altenar, we often use RabbitMQ as an entry point for our products. In this specific case, we will discuss a system that calculates sports data and provides it to other systems within the organization as well as outside the perimeter. When designing the system, the RabbitMQ component had the following requirements:
The endpoint is secured with an SSL certificate
RabbitMQ is hosted in Kubernetes. We use the RabbitMQ cluster operator and RabbitMQ messaging topology operator
For this project, we are using Google Cloud. Like Let’s Encrypt, you can use a Google Managed SSL certificate for any public HTTP load balancer in Google Cloud. However, it has several limitations, with the key ones being:
It only works for HTTP Load balancers.
It only works for public Load balancers.
According to RabbitMQ documentation, there are two common approaches for client connections:
Configure RabbitMQ to handle TLS connections
Use a proxy or load balancer (such as HAproxy) to perform TLS termination of client connections and use plain TCP connections to RabbitMQ nodes.
Our idea was to use a Google Managed certificate for TCP/SSL proxy load balancer in Google Cloud. However, due to the aforementioned limitations, a managed certificate is not supported for TCP load balancers. On the other hand, Google allows you to use an SSL certificate for several load balancers, which we decided to explore. Overall the blueprint would look like this:
We wanted to have a default dummy HTTP service that we would expose over port 443 and use for the sake of managing the SSL certificate and automatic renewals.
We would have a separate endpoint reusing the same IP address and SSL certificate.
Let’s examine each part of the solution in detail.
GKE Cluster
As the main hosting platform, we use GKE. We use the Google Terraform module for the private GKE cluster, configure Workload Identity and install an AutoNEG controller to the cluster.
In this article, we won’t delve into the specifics of how Workload Identity works with AutoNEG, as they are used as described in the official documentation.
With all preparations in place, we deploy a dummy “Hello World” service to the cluster and attach it to the Google Load balancer via AutoNEG. Below is the YAML configuration for that:
apiVersion: apps/v1
kind: Deployment
metadata:
name: gcp-tls-certificate-issuer
labels:
app: gcp-tls-certificate-issuer
annotations:
deployment.kubernetes.io/revision: '1'
spec:
replicas: 2
selector:
matchLabels:
app: gcp-tls-certificate-issuer
template:
metadata:
labels:
app: gcp-tls-certificate-issuer
spec:
containers:
- name: ok
image: assemblyline/ok:latest
ports:
- containerPort: 8888
protocol: TCP
imagePullPolicy: Always
securityContext:
capabilities:
drop:
- ALL
runAsUser: 1000
runAsGroup: 3000
runAsNonRoot: true
readOnlyRootFilesystem: true
restartPolicy: Always
terminationGracePeriodSeconds: 30
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 25%
maxSurge: 25%
revisionHistoryLimit: 10
progressDeadlineSeconds: 600
---
apiVersion: v1
kind: Service
metadata:
name: gcp-tls-certificate-issuer
labels:
app: gcp-tls-certificate-issuer
annotations:
cloud.google.com/neg: '{"exposed_ports": {"8888":{"name": "gcp-tls-certificate-issuer"}}}'
controller.autoneg.dev/neg: '{"backend_services":{"8888":[{"name":"envcode-rabbit-https-backend-service","max_connections_per_endpoint":10000}]}}'
spec:
ports:
- name: http
protocol: TCP
port: 8888
targetPort: 8888
selector:
app: gcp-tls-certificate-issuer
clusterIP: 10.10.12.130
clusterIPs:
- 10.10.12.130
type: ClusterIPNotice the service's annotation. This is how AutoNEG adds it to the Load balancer as a backend.
Google load balancer
The next part is managed outside GKE and created separately. Google Load balancer is not a single object but rather a set of different objects combined. Below is the Terraform code with comments:
resource "google_compute_managed_ssl_certificate" "rabbitmq" {
project = var.project
name = "${var.environment_name}-google-managed-certificate-rabbitmq"
managed {
domains = ["rabitmq.example.com."] # Replace with your domain
}
}
# reserved IP address
resource "google_compute_global_address" "default" {
project = var.project
name = "tcp-proxy-xlb-ip"
}
output "rabbitmq-ip" {
value = google_compute_global_address.default.address
}
# forwarding rule for TCP Loadbalanser
resource "google_compute_global_forwarding_rule" "default" {
project = var.project
name = "${var.environment_name}-tcp-global-loadbalancer"
provider = google
ip_protocol = "TCP"
load_balancing_scheme = "EXTERNAL"
port_range = "5671"
target = google_compute_target_ssl_proxy.default.id
ip_address = google_compute_global_address.default.id
}
# https://cloud.google.com/load-balancing/docs/ssl
# When you use Google-managed SSL certificates with SSL Proxy Load Balancing, the frontend port for traffic must be 443 to enable the Google-managed SSL certificates to be provisioned and renewed.
# forwarding rule for HTTPS Loadbalanser
resource "google_compute_global_forwarding_rule" "https" {
project = var.project
name = "${var.environment_name}-https-global-loadbalancer"
provider = google
ip_protocol = "TCP"
load_balancing_scheme = "EXTERNAL"
port_range = "443"
target = google_compute_target_ssl_proxy.https.id
ip_address = google_compute_global_address.default.id
}
resource "google_compute_target_ssl_proxy" "default" {
project = var.project
name = "${var.environment_name}-global-loadbalancer-tcp-proxy"
backend_service = google_compute_backend_service.default.id
ssl_certificates = [google_compute_managed_ssl_certificate.rabbitmq.id]
}
resource "google_compute_target_ssl_proxy" "https" {
project = var.project
name = "${var.environment_name}-global-loadbalancer-https-proxy"
backend_service = google_compute_backend_service.https.id
ssl_certificates = [google_compute_managed_ssl_certificate.rabbitmq.id]
}
# backend service For RabbitMQ Autoneg
resource "google_compute_backend_service" "default" {
project = var.project
name = "${var.environment_name}-tcp-backend-service"
protocol = "TCP"
port_name = "tcp"
load_balancing_scheme = "EXTERNAL"
timeout_sec = 10
health_checks = [google_compute_health_check.default.id]
session_affinity = "CLIENT_IP"
# We don't want TF to remove whatever was configured by AutoNEG
lifecycle {
ignore_changes = [backend]
}
}
# backend service For HTTPS Autoneg
resource "google_compute_backend_service" "https" {
project = var.project
#that's what you use in the service annotations
name = "${var.environment_name}-https-backend-service"
protocol = "TCP"
port_name = "tcp"
load_balancing_scheme = "EXTERNAL"
timeout_sec = 10
health_checks = [google_compute_health_check.https.id]
# We don't want TF to remove whatever was configured by AutoNEG
lifecycle {
ignore_changes = [backend]
}
}
resource "google_compute_health_check" "default" {
project = var.project
name = "tcp-proxy-health-check"
description = "Backend service for AutoNEG"
timeout_sec = 1
check_interval_sec = 1
tcp_health_check {
port = "5672" #use container port
}
}
resource "google_compute_health_check" "https" {
project = var.project
name = "https-proxy-health-check"
description = "Backend service for AutoNEG"
timeout_sec = 1
check_interval_sec = 1
tcp_health_check {
port = "8888" #use container port
}
}]
As you can see, we created one IP address and one SSL certificate and then used them in two forwarding rules. This allowed us to have a managed SSL certificate used for the TCP Load Balancer.
Don’t forget to configure DNS and point the IP to the right hostname for the whole thing to work.
Tip: GKE has thel7-default-backend deployment. Perhaps it would’ve been enough just creating a service with AutoNEG annotations and pointing it to that deployment’s pods. Try that and let me know in the comments if it works.
