A large Dutch government client was facing challenges to manage certificates for the various infrastructure components. As a trusted partner, Devoteam was asked to help out with these challenges. In this blog, you will read about my experience solving this problem of PKI Certificates renewal using Ansible and Venafi, in a fully automated way.

Introduction to PKI certificates

The PKI certificate is the digital signature when sending data and messages. PKI stands for Public Key Infrastructure. This is the international standard for securing data and messages.

This last function is what we discuss here since we often face the necessity to enrich data from one log to another. Below is a schematic visualization of the data enrichment process.

The company has a lot of systems and applications which use certificates that are only valid for one year. An internal team is dedicated to renewing or creating new certificates and signing them with the correct CA.

The process for requesting and installing a new certificate looks like this:

1. Create a private key with a password and create a CSR with this private key
2. Send the CSR to the certificate team and wait for them to return a certificate. This step could take days if they are really busy.
3. Install the certificate on your server.
4. Document the new certificate with its expiration date.

Within the systems support team, approximately 50 certificates are managed. Within the whole company, thousands of certificates are managed.

A time-consuming and error-prone process

This whole process of getting and installing a new certificate takes a lot of time and mistakes will be made. In some cases, a server or application also needs to have a keystore, this takes more time than the basic process. And if an application needs to be available outside of the network, an F5 Network (BigIP) load balancer holds the certificate.

The time-consuming process doesn’t stop here, there is also the documentation that needs to be up to date. In our team, we have a confluence page with all the certificates and their expiration dates. And we also have a password manager with all the certificates, private keys, chains, and keystores with their passwords, which need to be up to date.

To get a sense of the situation, let’s take one basic Certificate, so no keystore or F5 Network, to renew. In a time sense, this takes about 45 minutes, if you know what you are doing. And this doesn’t include the waiting period for the certificate team to send you the new certificate.

So, you have requested and installed the new certificate, but there is something wrong.  When you check the certificate, it has the wrong altname. Oh no, the whole process starts all over again! You can imagine this is a real pain for the company, but also for the people administering the certificates.

When I started working for this company, one of my first questions was: “Why is this not automated?” The common answer was: “It can’t be done, because we don’t have something like a certificate management tool.” I couldn’t believe it and called the Certificate team to discuss the opportunities for a certificate management tool. Fortunately, they also saw this opportunity and were already building a certificate management tool with Venafi. They were happy to have me on board to help out with the solution.

Putting the solution together with Venafi and Ansible

As soon as I was part of the project I started making a playbook in Ansible. Lucky for me, the Ansible community created a Venafi module. With this module, I can create a CSR to request a new certificate. With some trial and error, I managed to authenticate to Venafi and request a new certificate. This request only takes 2 seconds now, that’s a big win!

The process of requesting a new certificate is now automated, but what about the installation and documentation of it all? I came up with a solution for this too. Because I needed to talk to several systems, F5 Network for external certificates, the managed servers, and confluence for the documentation, there was more to this playbook than just the Venafi module. There needed to be a check to see if a certificate was to expire.

This is what I came up with:

1. A pipeline that would run every day and invoke the Ansible playbook
2. The playbook would check the following systems:
   • BigIp, for certificates that would expire in the upcoming month with the bigip device info module.
   • All the servers, if there were certificates that would pass in the upcoming month with the shell module and the command: # openssl x509 -enddate -noout -in
3. If there would be certificates to renew, the playbook would: 
   1. Request a new certificate with the Venafi module.
   2. For BigIp, install the certificate with the bigip ssl certificate module.
   3. For a server, replace the existing certificate with the new one and restart the application. If the application needs a keystore, create the keystore with the new certificate with the openssl pkcs12 module.
4. After checking and installing the certificates, the playbook would update the confluence page with the REST API and the Ansible uri module

Organization-wide impact

As a result of the solution mentioned above, we don’t ever need to worry about certificates anymore. Mistakes are off the table, and naturally, the teams were really happy with the result. I delivered workshops to explain the solution in depth so that other teams can also follow along and automate this for themselves. Now several teams within the organization reuse this playbook adjusted to their needs.

Working on this solution was quite gratifying due to not only the technical challenge it posed but also the significant positive impact it had on the company.

Working at Devoteam

We, as Devoteam, are simply the sum of 9,000 Tech Enthusiasts full of innovative energy. To increase our tech stronghold & density, we are constantly looking for talent and potential, from young professionals to experienced legends. Have a look at our vibrant culture and discover our company!