How to configure HTTPS for your applications on the Kubernetes cluster? How to manage TLS certificates using traefik? Traefik proxy is a very popular ingress controller in kubernetes environments. It supports both HTTPS (router) and TLS connections. This article explains how to configure TLS connection and obtain TLS certificates dynamically using pebble (Only for the test environment). In the real world, you will be using Let’s Encrypt (ACME). ACME stands for Automatic Certificate Management environment.
Pebble is used to simulate the let’s encrypt on the LAB environment. The following diagram explains what we are going to achieve at the end of this article.
How to Pebble works?
Pebble is a small ACME test server and not suited for production use. It provides a simplified ACME testing front end. In my LAB environment, the pebble will listen on the HTTPS port and traefik will try to establish an HTTPS connection to obtain the TLS certs from the pebble. To establish the connection between traefik and pebble, the root certificate needs to be shared with traefik pod.
Installing Pebble:
Pebble can be installed quickly using the helm charts. For detailed information about pebble and deployment, kindly check in Pebble GitHub page.
- Add the pebble jupyterhub helm repo.
root@kmaster1:~# helm repo add jupyterhub https://jupyterhub.github.io/helm-chart "jupyterhub" has been added to your repositories root@kmaster1:~#
2. Search the jypyterhub repo to know the pebble version.
root@kmaster1:~# helm search repo jupyterhub NAME CHART VERSION APP VERSION DESCRIPTION jupyterhub/jupyterhub 2.0.0 3.0.0 Multi-user Jupyter installation jupyterhub/pebble 1.0.1 v2.3.1 This Helm chart bootstraps Pebble: an ACME serv... root@kmaster1:~#
3. Download the pebble value files to edit a few parameters.
root@kmaster1:~# helm show values jupyterhub/pebble > /tmp/pebble_vaules.yaml
4. Update the environment values. Enable “PEBBLE_VA_ALWAYS_VALID ” and change the value to “1” .
env:
## ref: https://github.com/letsencrypt/pebble#testing-at-full-speed
- name: PEBBLE_VA_NOSLEEP
value: "1"
## ref: https://github.com/letsencrypt/pebble#invalid-anti-replay-nonce-errors
- name: PEBBLE_WFE_NONCEREJECT
value: "0"
## ref: https://github.com/letsencrypt/pebble#authorization-reuse
- name: PEBBLE_AUTHZREUSE
value: "100"
# ## ref: https://github.com/letsencrypt/pebble#skipping-validation
- name: PEBBLE_VA_ALWAYS_VALID
value: "1"
5. Disable the coredns which is optional.
## coredns is an optional DNS server that can for example point anything.test ## to your-acme-client.your-namespace.svc.cluster.local. coredns: enabled: false
6. Deploy the pebble using the modified values file.
root@kmaster1:~# helm install pebble jupyterhub/pebble --values /tmp/pebble_vaules.yaml -n traefik
NAME: pebble
LAST DEPLOYED: Tue Oct 18 05:27:40 2022
NAMESPACE: traefik
STATUS: deployed
REVISION: 1
NOTES:
The ACME server is available at:
https://pebble.traefik/dir
https://localhost:32443/dir
The ACME server generates leaf certificates to ACME clients,
and signs them with an insecure root cert, available at:
https://pebble.traefik:8444/roots/0
https://localhost:32444/roots/0
Communication with the ACME server itself requires
accepting a root certificate in configmap/pebble:
kubectl get configmap/pebble -o jsonpath="{.data['root-cert\.pem']}"
root@kmaster1:~#
7. Verify the pebble installation.
root@kmaster1:~# kubectl get all -n traefik NAME READY STATUS RESTARTS AGE pod/pebble-5dfccc497b-tmz7f 1/1 Running 0 20m pod/traefik-fc9c99df4-pc6ps 1/1 Running 0 13s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/pebble NodePort 10.102.40.179 <none> 443:32443/TCP,8444:32444/TCP 20m service/traefik LoadBalancer 10.111.162.31 172.16.16.9 80:30170/TCP,443:32044/TCP 13s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/pebble 1/1 1 1 20m deployment.apps/traefik 1/1 1 1 13s NAME DESIRED CURRENT READY AGE replicaset.apps/pebble-5dfccc497b 1 1 1 20m replicaset.apps/traefik-fc9c99df4 1 1 1 13s root@kmaster1:~#
8. Pebble deployment would have created the configmap to store the root certificate. List the configmap in the traefik namespace.
root@kmaster1:~# kubectl get cm -n traefik NAME DATA AGE kube-root-ca.crt 1 37h pebble 3 21m root@kmaster1:~#
We can see that configmap has been created in the name of “pebble” which has the root certificate and root keys.
Configuring traefik to use Pebble ACME:
Traefik value files need a few changes to use pebble as ACME server. If you do not have the traefik value file, you can extract using the following command.
root@kmaster1:~# helm get values traefik -n traefik > /tmp/traefik_vaules.current.yaml
1. I have an existing value file that was used to deploy traefik. Navigate to “additionalArguments” section.
#
# Configure Traefik static configuration
# Additional arguments to be passed at Traefik's binary
# All available options available on https://docs.traefik.io/reference/static-configuration/cli/
## Use curly braces to pass values: `helm install --set="additionalArguments={--providers.kubernetesingress.ingressclass=traefik-internal,--log.level=DEBUG}"`
additionalArguments: []
# - "--providers.kubernetesingress.ingressclass=traefik-internal"
# - "--log.level=DEBUG"
2. Remove the empty list from “additionalArguments” and update the following.
additionalArguments: - --certificatesresolvers.pebble.acme.tlschallenge=true - --certificatesresolvers.pebble.acme.email=test@hello.com - --certificatesresolvers.pebble.acme.storage=/data/acme.json - --certificatesresolvers.pebble.acme.caserver=https://pebble.traefik/dir
3. Mount the pebble configmap as volume on the traefik pod.
Existing entries:
volumes: [] # - name: public-cert # mountPath: "/certs" # type: secret
Updated entries
volumes:
- name: pebble
mountPath: "/cacerts"
type: configMap
# - name: public-cert
# mountPath: "/certs"
# type: secret
4. update the environment variable to make the traefik pod to aware that the pebble root certificate is placed under mount path “/cacerts”. Search for “env” and update the environment variable.
Existing entries:
# Environment variables to be passed to Traefik's binary env: [] # - name: SOME_VAR # value: some-var-value # - name: SOME_VAR_FROM_CONFIG_MAP # valueFrom:
Updated entries:
# Environment variables to be passed to Traefik's binary
env:
- name: LEGO_CA_CERTIFICATES
vaule: "/cacerts/root-cert.pem"
5. We have successfully updated all the required values. Let’s upgrade the traefik deployment using helm.
root@kmaster1:~# helm ls -n traefik NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION pebble traefik 1 2022-10-18 05:27:40.845619796 +0000 UTC deployed pebble-1.0.1 v2.3.1 traefik traefik 1 2022-10-18 05:47:58.708384764 +0000 UTC deployed traefik-15.2.2 2.9.1 root@kmaster1:~#
root@kmaster1:~# helm upgrade traefik traefik/traefik --values traefik-values.yaml -n traefik Release "traefik" has been upgraded. Happy Helming! NAME: traefik LAST DEPLOYED: Tue Oct 18 06:14:54 2022 NAMESPACE: traefik STATUS: deployed REVISION: 2 TEST SUITE: None root@kmaster1:~#
Test our work:
To test the certificate resolver, will deploy nginx and expose the service.
1. Deploy the nginx service on a new namespace called “testpebble”
root@kmaster1:~# kubectl apply -f https://k8s.io/examples/application/deployment.yaml -n testpebble --createnamespace deployment.apps/nginx-deployment created root@kmaster1:~# kubectl get all -n testpebble NAME READY STATUS RESTARTS AGE pod/nginx-deployment-66b6c48dd5-fmpmb 1/1 Running 0 11s pod/nginx-deployment-66b6c48dd5-ltwpm 1/1 Running 0 11s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-deployment 2/2 2 2 11s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-deployment-66b6c48dd5 2 2 2 11s root@kmaster1:~#
2. Expose the nginx-deployment to ClusterIP.
root@kmaster1:~# kubectl expose deployment nginx-deployment -n testpebble service/nginx-deployment exposed root@kmaster1:~# kubectl get all -n testpebble NAME READY STATUS RESTARTS AGE pod/nginx-deployment-66b6c48dd5-fmpmb 1/1 Running 0 92s pod/nginx-deployment-66b6c48dd5-ltwpm 1/1 Running 0 92s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/nginx-deployment ClusterIP 10.110.176.233 <none> 80/TCP 4s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/nginx-deployment 2/2 2 2 92s NAME DESIRED CURRENT READY AGE replicaset.apps/nginx-deployment-66b6c48dd5 2 2 2 92s root@kmaster1:~#
3. Create an ingressroute for nginx-deployment.
root@kmaster1:~# cat ingress.yaml
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
name: nginx-deployment
namespace: testpebble
spec:
entryPoints:
- websecure
routes:
- match: Host(`nginxtls.ua.com`)
kind: Rule
services:
- name: nginx-deployment
port: 80
tls:
certResolver: pebble
root@kmaster1:~#
root@kmaster1:~# kubectl create -f ingress.yaml -n testpebble ingressroute.traefik.containo.us/nginx-deployment created root@kmaster1:~#
4. Ensure that you have the following host entry from where you are trying to access this nginx deployment.
172.16.16.9 nginxtls.ua.com
172.16.16.9 is the traefik IP address.
root@kmaster1:~# kubectl get svc -n traefik |grep LoadBalancer traefik LoadBalancer 10.111.162.31 172.16.16.9 80:30170/TCP,443:32044/TCP 57m root@kmaster1:~#
5. Try to access “https://nginxtls.ua.com” from your host.
Click on advanced and proceed. You are getting this warning because pebble provides fake certificates for the LAB environment.
6. Here is the nginx welcome page.
7. Click on “Not Secure” and click on the “Certificate not valid” tab.
Check the pebble logs to know how the valid certificate has been assigned to the microservice.
root@kmaster1:~# kubectl logs pebble-5dfccc497b-tmz7f -n traefik
Pebble 2022/10/18 06:26:12 There are now 1 accounts in memory
Pebble 2022/10/18 06:26:12 POST /order-plz -> calling handler()
Pebble 2022/10/18 06:31:19 POST /order-plz -> calling handler()
Pebble 2022/10/18 06:55:46 POST /order-plz -> calling handler()
Pebble 2022/10/18 06:55:46 There are now 1 authorizations in the db
Pebble 2022/10/18 06:55:46 Added order "OFjipa4HBL9PqiCC89nOQZqkdGbcZMThwStzf_98Jws" to the db
Pebble 2022/10/18 06:55:46 There are now 1 orders in the db
Pebble 2022/10/18 06:55:46 POST /authZ/ -> calling handler()
Pebble 2022/10/18 06:55:46 POST /chalZ/ -> calling handler()
Pebble 2022/10/18 06:55:46 Pulled a task from the Tasks queue: &va.vaTask{Identifier:acme.Identifier{Type:"dns", Value:"nginxtls.ua.com"}, Challenge:(*core.Challenge)(0xc000079720), Account:(*core.Account)(0xc000226900)}
Pebble 2022/10/18 06:55:46 Starting 3 validations.
Pebble 2022/10/18 06:55:46 PEBBLE_VA_ALWAYS_VALID is enabled. Skipping real validation of challenge gEz7JpwqaomkhZuV9fnxvHbN6wXLlAVCCh6y-0IK0PA
Pebble 2022/10/18 06:55:46 POST /authZ/ -> calling handler()
Pebble 2022/10/18 06:55:52 POST /finalize-order/ -> calling handler()
Pebble 2022/10/18 06:55:52 Order OFjipa4HBL9PqiCC89nOQZqkdGbcZMThwStzf_98Jws is fully authorized. Processing finalization
Pebble 2022/10/18 06:55:52 Issued certificate serial 1ee532509beda353 for order OFjipa4HBL9PqiCC89nOQZqkdGbcZMThwStzf_98Jws
Pebble 2022/10/18 06:55:52 POST /my-order/ -> calling handler()
Pebble 2022/10/18 06:55:52 POST /certZ/ -> calling handler()
root@kmaster1:~#
On the traefik dashboard, you can see the TLS cert resolver entry like below.
We have successfully configured the TLS connection and obtained TLS certificates dynamically using pebble on the k8s traefik environment.