How-to guides

Secure with mutual TLS

Authenticate every node with a certificate from the cluster's own certificate authority, bound to its address.

The cluster-dev transport checks no certificates, so anyone who can reach a node's port can talk to it. A production cluster uses mutual TLS instead: every node presents a certificate from the cluster's own certificate authority, and accepts only peers that do the same. The examples on this page come from two nodes that form a cluster over mutual TLS. The first one creates the certificates, builds its transport in code and starts the second one with a config file, the way a deployed node reads its certificates.

Issue the certificates

tellus/examples/how_to/secure_with_mutual_tls.rs

struct CertificateAuthority {
    cert: Certificate,
    key: KeyPair,
}

impl CertificateAuthority {
    fn new() -> anyhow::Result<Self> {
        let key = KeyPair::generate().context("generating the CA key")?;
        let mut params = CertificateParams::new(Vec::new()).context("CA parameters")?;
        params.is_ca = IsCa::Ca(BasicConstraints::Unconstrained);
        let cert = params
            .self_signed(&key)
            .context("signing the CA certificate")?;
        Ok(Self { cert, key })
    }

    fn issue(&self, addr: SocketAddr) -> anyhow::Result<(Certificate, KeyPair)> {
        let key = KeyPair::generate().context("generating a node key")?;
        let mut params =
            CertificateParams::new(vec![SERVER_NAME.to_string()]).context("node parameters")?;
        let uri = format!("tellus://{addr}");
        params.subject_alt_names.push(SanType::URI(
            uri.try_into().context("the URI SAN is not ASCII")?,
        ));
        let cert = params
            .signed_by(&key, &self.cert, &self.key)
            .context("signing a node certificate")?;
        Ok((cert, key))
    }
}

A node's certificate needs two things:

The name every node validates its peers against, here tellus as a DNS name. All nodes share it.
The address the node advertises, as a tellus://<ip:port> URI SAN. A node is admitted to the cluster only at an address its certificate covers, so an authenticated node cannot claim another member's address. Without such a URI, the advertised IP has to be among the certificate's IP address SANs, which then covers every port of that IP. That is enough as long as every IP hosts at most one node, as with one pod per IP on Kubernetes.

The example uses rcgen. In production, your certificate authority issues them, e.g. through cert-manager, whose Certificate resource requests uris and ipAddresses. SPIRE cannot issue either form.

Build the transport in code

tellus/examples/how_to/secure_with_mutual_tls.rs

async fn first_node() -> anyhow::Result<()> {
    let seeds = reserved_addrs(NODES)?;
    let ca = CertificateAuthority::new()?;
    let dir = tempfile::tempdir().context("creating a directory for the PEM files")?;
    let config = write_node_config(dir.path(), &ca, seeds[1], &seeds)?;
    let _second_node = spawn_second_node(&config)?;

    let (cert, key) = ca.issue(seeds[0])?;
    let mut roots = RootCertStore::empty();
    roots
        .add(ca.cert.der().clone())
        .context("adding the certificate authority")?;
    let key = PrivateKeyDer::from(PrivatePkcs8KeyDer::from(key.serialize_der()));
    let transport =
        QuicTransport::mutual_tls(seeds[0], vec![cert.der().clone()], key, roots, SERVER_NAME)
            .context("mutual TLS transport")?;
    join_cluster(seeds[0], seeds, transport).await?;

    let mut state = cluster::cluster_state().context("subscribing to the cluster state")?;
    let state = timeout(
        BOOTSTRAP_TIMEOUT,
        state.wait_for(|state| state.up().count() == NODES),
    )
    .await
    .context("the second node did not come up within the timeout")?
    .context("the cluster state stopped")?;
    println!("Members up over mutual TLS: {}", state.up().count());
    Ok(())
}

QuicTransport::mutual_tls takes the node's certificate chain and key, the roots its peers' certificates must verify against, and the shared name. The node presents its certificate as both server and client, so both sides of every connection are authenticated. A node without a certificate from the cluster's authority cannot complete a connection, so it reaches neither the protocol nor anybody's failure detection.

Read the certificates from a config file

With the serde feature, QuicConfig names the PEM files, so it can come from your configuration:

tellus/examples/how_to/secure_with_mutual_tls.rs

async fn second_node() -> anyhow::Result<()> {
    let path = env::args().nth(2).context("config path argument")?;
    let config = config::Config::builder()
        .add_source(config::File::with_name(&path))
        .build()
        .context("loading the configuration")?
        .try_deserialize::<NodeConfig>()
        .context("deserializing the configuration")?;

    let transport =
        QuicTransport::from_config(&config.transport).context("mutual TLS transport")?;
    join_cluster(config.transport.bind_addr, config.seeds, transport).await?;

    future::pending().await
}

#[derive(Deserialize)]
struct NodeConfig {
    seeds: Vec<SocketAddr>,
    transport: QuicConfig,
}

The second node reads a TOML file like this one, which the example writes next to the PEM files:

seeds = ["127.0.0.1:41000", "127.0.0.1:41001"]

[transport]
bind_addr   = "127.0.0.1:41001"
cert_chain  = "/etc/tellus/tls/node-cert.pem"
key         = "/etc/tellus/tls/node-key.pem"
roots       = "/etc/tellus/tls/roots.pem"
server_name = "tellus"

QuicTransport::from_config reads and validates the files. It reads them once, so renewing a certificate takes a restart of the node.

Run it

From a clone of the tellus repository:

cargo run --quiet -p tellus --features cluster,serde --example how_to_secure_with_mutual_tls

Members up over mutual TLS: 2