Build your own CA with Ansible

2020-11-28 ansible

Securing your Kafka, Elasticsearch, Cassandra, or whatever distributed software requires configuring using SSL (also known as TLS) to encrypt communications:

Node to node communication
Client to node communication

Setting up SSL means providing SSL certificates for each node. But generating SSL certificates is a cumbersome task:

The Kafka documentation describes extensively the process.
Elasticsearch brings its own elasticsearch-certutil tool.
Datastax also documents a similar process for Cassandra

I will describe here how to generate an SSL certificate for each node using Ansible. It makes sense as I am also deploying Kafka, Elasticsearch and the like with Ansible.

There are several important rules to know when generating certificates:

The name present in the certificate must match the public name of the host. We can not share the same certificate on all nodes unless using star certificates. Any TLS client connecting to a node will check that certificate name and hostname matches unless disabling hostname verification.
The name present in the certificate, should match the reverse DNS name corresponding to the IP of the host. Java clients connecting to a node, will do a reverse DNS lookup to get the public name of the host they are connecting to.

These two rules are meant to prevent Man in the middle attacks. A TLS certificate allows checking you’re talking to the wanted target, not something in between which could spy and steal information.

When a machine has multiple names (think about DNS aliases, virtual hosts), a certificate can contain multiple names. The main name is called CN (Common Name), while other names are called SAN (Subject Alt Names).

The certificate authority

As Kafka or Elasticsearch clusters should never be publicly exposed, using a public certificate authority (Thawte, Verisign and the like) is not necessary. A self-signed certificate authority local to the cluster or the environment (Dev, Q/A) should be enough.

So the first step is to create a certificate authority that will be used to sign the certificates of all hosts belonging to our cluster. As this step will be done only once, I won’t automate it.

$ mkdir ownca
$ openssl req -new -x509 \
    -days 1825 \ (1)
    -extensions v3_ca \ (2)
    -keyout ownca/root.key -out ownca/root.crt (3)

Generating a RSA private key
......+++++
....+++++
writing new private key to 'ownca/root.key'
Enter PEM pass phrase: (4)
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:FR (5)
State or Province Name (full name) [Some-State]:.
Locality Name (eg, city) []:.
Organization Name (eg, company) [Internet Widgits Pty Ltd]:eNova Conseil
Organizational Unit Name (eg, section) []:.
Common Name (e.g. server FQDN or YOUR name) []:Root
Email Address []:rootca@enova-conseil.com

1	The CA root certificate will last 5 years
2	This certificate will be used as a CA
3	Generate both key and self-signed certificate
4	The key is protected with a password
5	Information describing the Root certificate

For safety reasons, the generated key should be kept secret and stored in a secure place:

It must not be transfered to target Kafka servers
It must not be kept in source control (Git) unless hidden in Ansible Vault password file

The nodes certificates

This is where Ansible comes in. As your cluster might have many nodes, automating certificate generation makes sense. For each target host, I will repeat the same process:

On the target host, generate a key target.key and a CSR (Certificate signing request) target.csr
Pull the CSR on the control host.
Sign the CSR with the CA key. This will generate a certificate target.crt.
Push the generated certificate target.crt on the target host. The CA certificate root.crt is also pushed.

As the TLS keys .key are sensitive, they do not travel, they stay where they were generated. On the contrary, certificates .crt and CSRs .csr only contain public information.

# Step 1
- name: Generate private key
  become: true
  openssl_privatekey:
    path: "/etc/pki/tls/private/{{ openssl_name }}.key"

- name: Generate CSR
  become: true
  openssl_csr:
    path: "/etc/pki/tls/private/{{ openssl_name }}.csr"
    privatekey_path: "/etc/pki/tls/private/{{ openssl_name }}.key"
    country_name: FR
    organization_name: "eNova Conseil"
    common_name: "{{ openssl_name }}"
    subject_alt_name: "DNS:{{ ansible_host }},DNS:{{ ansible_fqdn }}"

# Step 2
- name: Pull CSR
  become: true
  fetch: 
    src: "/etc/pki/tls/private/{{ openssl_name }}.csr"
    dest: "{{ openssl_ownca_dir }}/{{ openssl_name }}.csr"
    flat: true

# Step 3
- name: Sign CSR with CA key
  connection: local
  delegate_to: localhost
  openssl_certificate:
    path: "{{ openssl_ownca_dir }}/{{ openssl_name }}.crt"
    csr_path: "{{ openssl_ownca_dir }}/{{ openssl_name }}.csr"
    ownca_path: "{{ openssl_ownca_dir }}/root.crt"
    ownca_privatekey_path: "{{ openssl_ownca_dir }}/root.key"
    provider: ownca

# Step 4
- name: Push certificate
  become: true
  copy:
    src: "{{ openssl_ownca_dir }}/{{ openssl_name }}.crt"
    dest: "/etc/pki/tls/private/{{ openssl_name }}.crt"

- name: Push CA
  become: true
  copy: 
    src: "{{ openssl_ownca_dir }}/root.crt"
    dest: "/etc/pki/ca-trust/source/anchors/root.pem"

Once you have the key, the certificate and CA certificate chain on the target host, you can start using them:

- name: Update CA Trust
  become: true
  command: "update-ca-trust extract"

- name: Build PKCS12 file containing key and cert
  become: true
  openssl_pkcs12:
    action: export
    path: "/etc/pki/tls/private/{{ openssl_name }}.p12"
    friendly_name: "{{ openssl_name }}"
    privatekey_path: "/etc/pki/tls/private/{{ openssl_name }}.key"
    certificate_path: "/etc/pki/tls/private/{{ openssl_name }}.crt"
    other_certificates: "/etc/pki/ca-trust/source/anchors/root.pem"
    state: present

The produced PKCS12 file can be used as a Java Keystore. The java_keystore Ansible module can be used to create a JKS file instead.

The attentive reader has noticed I am using a bunch of openssl_xxx Ansible modules (namely openssl_privatekey, openssl_csr, openssl_certificate and openssl_pkcs12). These modules require to have openssl and PyOpenSSL installed on each host.

- name: Python OpenSSL package
  become: true
  yum:
    name: 
      - pyOpenSSL
      - python2-pip
      - ca-certificates

- name: Upgrade Python OpenSSL
  become: true
  pip:
    name: pyOpenSSL>=0.15

Build your own CA with Ansible

2020-11-28 ansible

The certificate authority

The nodes certificates

Other posts