Shut down gracefully

A node that is stopped should neither fail the requests a load balancer still sends it nor leave the other nodes to detect its absence. So on a shutdown signal it reports itself unhealthy, keeps serving until the load balancer has taken it out of rotation, drains the requests still in flight, and only then stops its actors and leaves the cluster. The examples on this page come from a node that serves a health endpoint and an application endpoint backed by an actor.

Serve requests and health

struct App {
    healthy: AtomicBool,
    counter: ActorRef<Count>,
}

fn router(app: Arc<App>) -> Router {
    Router::new()
        .route("/health", get(health))
        .route("/count", get(count))
        .with_state(app)
}

async fn health(State(app): State<Arc<App>>) -> StatusCode {
    if app.healthy.load(Ordering::Relaxed) {
        StatusCode::OK
    } else {
        StatusCode::SERVICE_UNAVAILABLE
    }
}

async fn count(State(app): State<Arc<App>>) -> Result<String, StatusCode> {
    app.counter
        .ask(ASK_TIMEOUT, Count)
        .await
        .map(|count| format!("{count}\n"))
        .map_err(|_| StatusCode::SERVICE_UNAVAILABLE)
}

/count asks the counter actor, so it only works while the actors run. /health answers 200 while the node is healthy and 503 once it is not. A load balancer, or a Kubernetes readiness probe, checks it periodically and stops sending requests to a node that fails it.

Wait for the signal

async fn shutdown_signal() -> anyhow::Result<()> {
    let mut terminate = signal(SignalKind::terminate()).context("SIGTERM handler")?;
    tokio::select! {
        _ = terminate.recv() => Ok(()),
        interrupt = signal::ctrl_c() => interrupt.context("Ctrl-C handler"),
    }
}

Kubernetes, Docker and systemd stop a process with SIGTERM; Ctrl-C sends SIGINT. The node treats both alike.

Shut down in order

async fn shut_down(
    app: &App,
    drain: &Notify,
    server: tokio::task::JoinHandle<io::Result<()>>,
    system: ActorSystem<Count>,
    leave_timeout: Duration,
) -> anyhow::Result<()> {
    app.healthy.store(false, Ordering::Relaxed);
    println!("Unhealthy, still serving");
    sleep(UNHEALTHY_LINGER).await;

    drain.notify_one();
    timeout(DRAIN_TIMEOUT, server)
        .await
        .context("the requests in flight did not drain within the timeout")?
        .context("the HTTP server panicked")?
        .context("serving HTTP")?;
    println!("Drained, leaving the cluster");

    system.stop();
    let departure = timeout(
        TERMINATION_TIMEOUT + leave_timeout,
        cluster::leave_on_terminated(system),
    )
    .await
    .context("the actors did not stop and leave within the timeout")?
    .context("leaving the cluster")?;
    if departure.is_acknowledged() {
        println!("Left the cluster");
    } else {
        println!(
            "Left the cluster, unacknowledged by {:?}, timed out for {:?}",
            departure.unacknowledged(),
            departure.timed_out()
        );
    }
    Ok(())
}

1. The node reports itself unhealthy, so the load balancer starts taking it out of rotation.
2. It lingers, still serving every request with its actors running. The linger has to exceed the time the load balancer needs to drop a node that fails its health check. For a load balancer that checks every two seconds and drops a node after one failure, that is two seconds; a Kubernetes readiness probe with periodSeconds: 2 and failureThreshold: 2 takes up to four. The example lingers for five.
3. It tells the HTTP server to shut down gracefully: the server accepts no new connections and waits for the requests in flight, bounded by a drain timeout.
4. It stops its actor system and leaves the cluster. leave_on_terminated first waits for the actor system to terminate, without a deadline of its own, and then tries to drain what the node still has queued for the other members, within EndpointConfig::leave_timeout. The other members list the node as left as soon as they receive the departure, and the Departure tells which of them acknowledged the drain. The outer timeout bounds both parts: a termination timeout for the actors plus the leave timeout.

The HTTP server runs in a task of its own, and the drain signal shuts it down:

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    init_tracing();
    let leave_timeout = start_endpoint()?;
    cluster::form().context("forming the cluster")?;

    let system = ActorSystem::new(Counter);
    let app = Arc::new(App {
        healthy: AtomicBool::new(true),
        counter: system.root().clone(),
    });
    let listener = TcpListener::bind(HTTP_ADDR)
        .await
        .context("binding the HTTP port")?;
    let drain = Arc::new(Notify::new());
    let drained = drain.clone();
    let server = tokio::spawn(
        axum::serve(listener, router(app.clone()))
            .with_graceful_shutdown(async move { drained.notified().await })
            .into_future(),
    );
    println!("Serving, press Ctrl-C to shut down");

    shutdown_signal().await?;
    shut_down(&app, &drain, server, system, leave_timeout).await
}

Budget the shutdown

Whatever stops the process waits only so long before it kills it: Kubernetes waits terminationGracePeriodSeconds, thirty by default. That has to cover all four steps: the linger, the drain timeout, the termination timeout and the leave timeout, five, five, five and three seconds in the example, eighteen in all. A node killed before it is done has not left: the others detect its absence and down it, as for a crash.

Run it

From a clone of the tellus repository:

cargo run --quiet -p tellus --features cluster-dev --example how_to_shut_down_gracefully

In a second terminal, curl -i localhost:8080/health answers 200 and curl localhost:8080/count counts. Press Ctrl-C in the first one. For the five seconds of the linger, /health answers 503, while /count keeps counting. Then the node drains, leaves and exits:

Serving, press Ctrl-C to shut down
Unhealthy, still serving
Drained, leaving the cluster
Left the cluster