kubernetes traefik - Managing TLS Certificates - Part 4

How to configure HTTPS for your applications on the Kubernetes cluster? How to manage TLS certificates using traefik? Traefik proxy is a very popular ingress controller in kubernetes environments. It supports both HTTPS (router) and TLS connections. This article explains how to configure TLS connection and obtain TLS certificates dynamically using pebble (Only for the test environment). In the real world, you will be using Let’s Encrypt (ACME). ACME stands for Automatic Certificate Management environment.

Kubernetes cluster - TLS - Lets encrypt — Kubernetes cluster – TLS – Lets encrypt

Pebble is used to simulate the let’s encrypt on the LAB environment. The following diagram explains what we are going to achieve at the end of this article.

Pebble - K8s - traefik - TLS connection — Pebble – K8s – traefik – TLS connection

How to Pebble works?

Pebble is a small ACME test server and not suited for production use. It provides a simplified ACME testing front end. In my LAB environment, the pebble will listen on the HTTPS port and traefik will try to establish an HTTPS connection to obtain the TLS certs from the pebble. To establish the connection between traefik and pebble, the root certificate needs to be shared with traefik pod.

Pebble - Configmap - traefik volume — Pebble – Configmap – traefik volume

Installing Pebble:

Pebble can be installed quickly using the helm charts. For detailed information about pebble and deployment, kindly check in Pebble GitHub page.

Add the pebble jupyterhub helm repo.

root@kmaster1:~# helm repo add jupyterhub https://jupyterhub.github.io/helm-chart
"jupyterhub" has been added to your repositories
root@kmaster1:~#

2. Search the jypyterhub repo to know the pebble version.

root@kmaster1:~# helm search repo jupyterhub
NAME                    CHART VERSION   APP VERSION     DESCRIPTION
jupyterhub/jupyterhub   2.0.0           3.0.0           Multi-user Jupyter installation
jupyterhub/pebble       1.0.1           v2.3.1          This Helm chart bootstraps Pebble: an ACME serv...
root@kmaster1:~#

3. Download the pebble value files to edit a few parameters.

root@kmaster1:~# helm show values jupyterhub/pebble > /tmp/pebble_vaules.yaml

4. Update the environment values. Enable “PEBBLE_VA_ALWAYS_VALID ” and change the value to “1” .

  env:
    ## ref: https://github.com/letsencrypt/pebble#testing-at-full-speed
    - name: PEBBLE_VA_NOSLEEP
      value: "1"
    ## ref: https://github.com/letsencrypt/pebble#invalid-anti-replay-nonce-errors
    - name: PEBBLE_WFE_NONCEREJECT
      value: "0"
    ## ref: https://github.com/letsencrypt/pebble#authorization-reuse
    - name: PEBBLE_AUTHZREUSE
      value: "100"
    # ## ref: https://github.com/letsencrypt/pebble#skipping-validation
    - name: PEBBLE_VA_ALWAYS_VALID
      value: "1"

5. Disable the coredns which is optional.

## coredns is an optional DNS server that can for example point anything.test
## to your-acme-client.your-namespace.svc.cluster.local.
coredns:
  enabled: false

6. Deploy the pebble using the modified values file.

root@kmaster1:~# helm install pebble jupyterhub/pebble --values /tmp/pebble_vaules.yaml -n traefik
NAME: pebble
LAST DEPLOYED: Tue Oct 18 05:27:40 2022
NAMESPACE: traefik
STATUS: deployed
REVISION: 1
NOTES:
The ACME server is available at:

    https://pebble.traefik/dir
    https://localhost:32443/dir

The ACME server generates leaf certificates to ACME clients,
and signs them with an insecure root cert, available at:

    https://pebble.traefik:8444/roots/0
    https://localhost:32444/roots/0

Communication with the ACME server itself requires
accepting a root certificate in configmap/pebble:

    kubectl get configmap/pebble -o jsonpath="{.data['root-cert\.pem']}"
root@kmaster1:~#

7. Verify the pebble installation.

root@kmaster1:~# kubectl get all -n traefik
NAME                          READY   STATUS    RESTARTS   AGE
pod/pebble-5dfccc497b-tmz7f   1/1     Running   0          20m
pod/traefik-fc9c99df4-pc6ps   1/1     Running   0          13s

NAME              TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                        AGE
service/pebble    NodePort       10.102.40.179   <none>        443:32443/TCP,8444:32444/TCP   20m
service/traefik   LoadBalancer   10.111.162.31   172.16.16.9   80:30170/TCP,443:32044/TCP     13s

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/pebble    1/1     1            1           20m
deployment.apps/traefik   1/1     1            1           13s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/pebble-5dfccc497b   1         1         1       20m
replicaset.apps/traefik-fc9c99df4   1         1         1       13s
root@kmaster1:~#

8. Pebble deployment would have created the configmap to store the root certificate. List the configmap in the traefik namespace.

root@kmaster1:~# kubectl get cm -n traefik
NAME               DATA   AGE
kube-root-ca.crt   1      37h
pebble             3      21m
root@kmaster1:~#

We can see that configmap has been created in the name of “pebble” which has the root certificate and root keys.

Configuring traefik to use Pebble ACME:

Traefik value files need a few changes to use pebble as ACME server. If you do not have the traefik value file, you can extract using the following command.

root@kmaster1:~#  helm get values traefik -n traefik > /tmp/traefik_vaules.current.yaml

1. I have an existing value file that was used to deploy traefik. Navigate to “additionalArguments” section.

#
# Configure Traefik static configuration
# Additional arguments to be passed at Traefik's binary
# All available options available on https://docs.traefik.io/reference/static-configuration/cli/
## Use curly braces to pass values: `helm install --set="additionalArguments={--providers.kubernetesingress.ingressclass=traefik-internal,--log.level=DEBUG}"`
additionalArguments: []
#  - "--providers.kubernetesingress.ingressclass=traefik-internal"
#  - "--log.level=DEBUG"

2. Remove the empty list from “additionalArguments” and update the following.

additionalArguments:
  - --certificatesresolvers.pebble.acme.tlschallenge=true
  - --certificatesresolvers.pebble.acme.email=test@hello.com
  - --certificatesresolvers.pebble.acme.storage=/data/acme.json
  - --certificatesresolvers.pebble.acme.caserver=https://pebble.traefik/dir

3. Mount the pebble configmap as volume on the traefik pod.

Existing entries:

volumes: []
# - name: public-cert
#   mountPath: "/certs"
#   type: secret

Updated entries

volumes:
  - name: pebble
    mountPath: "/cacerts"
    type: configMap
# - name: public-cert
#   mountPath: "/certs"
#   type: secret

4. update the environment variable to make the traefik pod to aware that the pebble root certificate is placed under mount path “/cacerts”. Search for “env” and update the environment variable.

Existing entries:

# Environment variables to be passed to Traefik's binary
env: []
# - name: SOME_VAR
#   value: some-var-value
# - name: SOME_VAR_FROM_CONFIG_MAP
#   valueFrom:

Updated entries:

# Environment variables to be passed to Traefik's binary
env:
  - name: LEGO_CA_CERTIFICATES
    vaule: "/cacerts/root-cert.pem"

5. We have successfully updated all the required values. Let’s upgrade the traefik deployment using helm.

root@kmaster1:~# helm ls -n traefik
NAME    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
pebble  traefik         1               2022-10-18 05:27:40.845619796 +0000 UTC deployed        pebble-1.0.1    v2.3.1
traefik traefik         1               2022-10-18 05:47:58.708384764 +0000 UTC deployed        traefik-15.2.2  2.9.1
root@kmaster1:~#

root@kmaster1:~# helm upgrade traefik traefik/traefik --values traefik-values.yaml -n traefik
Release "traefik" has been upgraded. Happy Helming!
NAME: traefik
LAST DEPLOYED: Tue Oct 18 06:14:54 2022
NAMESPACE: traefik
STATUS: deployed
REVISION: 2
TEST SUITE: None
root@kmaster1:~#

Test our work:

To test the certificate resolver, will deploy nginx and expose the service.

1. Deploy the nginx service on a new namespace called “testpebble”

root@kmaster1:~# kubectl apply -f https://k8s.io/examples/application/deployment.yaml -n testpebble --createnamespace
deployment.apps/nginx-deployment created

root@kmaster1:~# kubectl  get all -n testpebble
NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-66b6c48dd5-fmpmb   1/1     Running   0          11s
pod/nginx-deployment-66b6c48dd5-ltwpm   1/1     Running   0          11s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   2/2     2            2           11s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-66b6c48dd5   2         2         2       11s
root@kmaster1:~#

2. Expose the nginx-deployment to ClusterIP.

root@kmaster1:~# kubectl expose deployment nginx-deployment -n testpebble
service/nginx-deployment exposed
root@kmaster1:~# kubectl  get all -n testpebble
NAME                                    READY   STATUS    RESTARTS   AGE
pod/nginx-deployment-66b6c48dd5-fmpmb   1/1     Running   0          92s
pod/nginx-deployment-66b6c48dd5-ltwpm   1/1     Running   0          92s

NAME                       TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)   AGE
service/nginx-deployment   ClusterIP   10.110.176.233   <none>        80/TCP    4s

NAME                               READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nginx-deployment   2/2     2            2           92s

NAME                                          DESIRED   CURRENT   READY   AGE
replicaset.apps/nginx-deployment-66b6c48dd5   2         2         2       92s
root@kmaster1:~#

3. Create an ingressroute for nginx-deployment.

root@kmaster1:~# cat ingress.yaml
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: nginx-deployment
  namespace: testpebble
spec:
  entryPoints:
    - websecure
  routes:
    - match: Host(`nginxtls.ua.com`)
      kind: Rule
      services:
        - name: nginx-deployment
          port: 80
  tls:
    certResolver: pebble
root@kmaster1:~#

root@kmaster1:~# kubectl create -f ingress.yaml -n testpebble
ingressroute.traefik.containo.us/nginx-deployment created
root@kmaster1:~#

4. Ensure that you have the following host entry from where you are trying to access this nginx deployment.

172.16.16.9        nginxtls.ua.com

172.16.16.9 is the traefik IP address.

root@kmaster1:~# kubectl get svc -n traefik |grep LoadBalancer
traefik   LoadBalancer   10.111.162.31   172.16.16.9   80:30170/TCP,443:32044/TCP     57m
root@kmaster1:~#

5. Try to access “https://nginxtls.ua.com” from your host.

nginx webpage - TLS_v1 — nginx webpage – TLS_v1

Click on advanced and proceed. You are getting this warning because pebble provides fake certificates for the LAB environment.

6. Here is the nginx welcome page.

nginx welcome page - TLS — nginx welcome page – TLS

7. Click on “Not Secure” and click on the “Certificate not valid” tab.

Check the pebble logs to know how the valid certificate has been assigned to the microservice.

root@kmaster1:~# kubectl logs pebble-5dfccc497b-tmz7f  -n traefik

Pebble 2022/10/18 06:26:12 There are now 1 accounts in memory
Pebble 2022/10/18 06:26:12 POST /order-plz -> calling handler()
Pebble 2022/10/18 06:31:19 POST /order-plz -> calling handler()
Pebble 2022/10/18 06:55:46 POST /order-plz -> calling handler()
Pebble 2022/10/18 06:55:46 There are now 1 authorizations in the db
Pebble 2022/10/18 06:55:46 Added order "OFjipa4HBL9PqiCC89nOQZqkdGbcZMThwStzf_98Jws" to the db
Pebble 2022/10/18 06:55:46 There are now 1 orders in the db
Pebble 2022/10/18 06:55:46 POST /authZ/ -> calling handler()
Pebble 2022/10/18 06:55:46 POST /chalZ/ -> calling handler()
Pebble 2022/10/18 06:55:46 Pulled a task from the Tasks queue: &va.vaTask{Identifier:acme.Identifier{Type:"dns", Value:"nginxtls.ua.com"}, Challenge:(*core.Challenge)(0xc000079720), Account:(*core.Account)(0xc000226900)}
Pebble 2022/10/18 06:55:46 Starting 3 validations.
Pebble 2022/10/18 06:55:46 PEBBLE_VA_ALWAYS_VALID is enabled. Skipping real validation of challenge gEz7JpwqaomkhZuV9fnxvHbN6wXLlAVCCh6y-0IK0PA

Pebble 2022/10/18 06:55:46 POST /authZ/ -> calling handler()
Pebble 2022/10/18 06:55:52 POST /finalize-order/ -> calling handler()
Pebble 2022/10/18 06:55:52 Order OFjipa4HBL9PqiCC89nOQZqkdGbcZMThwStzf_98Jws is fully authorized. Processing finalization
Pebble 2022/10/18 06:55:52 Issued certificate serial 1ee532509beda353 for order OFjipa4HBL9PqiCC89nOQZqkdGbcZMThwStzf_98Jws
Pebble 2022/10/18 06:55:52 POST /my-order/ -> calling handler()
Pebble 2022/10/18 06:55:52 POST /certZ/ -> calling handler()
root@kmaster1:~#

On the traefik dashboard, you can see the TLS cert resolver entry like below.

traefik - Dashboard - TLS - Pebble cert resolver — traefik – Dashboard – TLS – Pebble cert resolver

We have successfully configured the TLS connection and obtained TLS certificates dynamically using pebble on the k8s traefik environment.

kubernetes traefik – Managing TLS Certificates – Part 4

How to Pebble works?

Installing Pebble:

Configuring traefik to use Pebble ACME:

Test our work:

Leave a Reply Cancel reply

How to Pebble works?

Installing Pebble:

Configuring traefik to use Pebble ACME:

Test our work:

Reader Interactions

Leave a Reply Cancel reply