Getting started

When devices fail

Let failing actors restart with backoff, and stop them when they keep failing.

Sensors are flaky. Sometimes a device reports a reading that is not a number at all. In this step such a reading makes the device fail, and tellus restarts it.

Failing

tellus/examples/tutorials/getting_started/06_supervision.rs

impl Actor for Device {
    type Message = DeviceMessage;
    type State = Option<f64>;
    type Error = InvalidReading;

    fn init(&self, _: &ActorContext<Self::Message>) -> Result<Self::State, Self::Error> {
        Ok(None)
    }

    fn receive(
        &self,
        _: &ActorContext<Self::Message>,
        incoming: Incoming<Self::Message>,
        last_reading: Self::State,
    ) -> Result<Control<Self::State>, Self::Error> {
        let Incoming::Message(message) = incoming else {
            unreachable!("the device watches no actor, hence never gets a terminated signal")
        };

        match message {
            DeviceMessage::Record(degrees) if degrees.is_nan() => Err(InvalidReading),

            DeviceMessage::Record(degrees) => Ok(Control::Continue(Some(degrees))),

            DeviceMessage::Read(reply_to) => {
                reply_to.reply(last_reading);
                Ok(Control::Continue(last_reading))
            }

            DeviceMessage::Retire => Ok(Control::Stop),
        }
    }
}

The device's Error type is now InvalidReading instead of Infallible. For a reading that is not a number, receive returns Err(InvalidReading). You can also use ? to escalate a failure the same way. A panic in receive counts as a failure, too.

What happens next is up to the device's supervision strategy.

Restarting

tellus/examples/tutorials/getting_started/06_supervision.rs

let device = devices.entry(device_name).or_insert_with(|| {
    let config = ActorConfig::default().with_supervision_strategy(
        SupervisionStrategy::Restart(RestartPolicy::new(MAX_RESTARTS)),
    );
    let device = context.spawn_with_config(Device, config);
    context.watch(&device);
    device
});

The group spawns every device with the Restart strategy. When a device fails, tellus runs init again for a fresh state. The device keeps its ActorRef and its mailbox, so messages that are already queued are still handled after the restart. Only the state is lost: a restarted device has no last reading.

Restarts back off exponentially, starting at 250 ms by default. A device failing more than MAX_RESTARTS times in a row stops for good. Since the group watches its devices, it gets Terminated and prunes the device, just as for a retired one.

Run it

tellus/examples/tutorials/getting_started/06_supervision.rs

#[tokio::main]
async fn main() -> anyhow::Result<()> {
    init_tracing();

    let system = ActorSystem::new(Group);
    let group = system.root();

    let readings = [
        ("left", 21.5),
        ("middle", 20.4),
        ("right", 19.1),
        ("left", f64::NAN),
        ("broken", f64::NAN),
        ("broken", f64::NAN),
        ("broken", f64::NAN),
        ("broken", f64::NAN),
    ];
    for (device_name, degrees) in readings {
        group.tell(GroupMessage::Record {
            device_name,
            degrees,
        });
    }
    group.tell(GroupMessage::Retire {
        device_name: "right",
    });

    let average = group
        .ask(Duration::from_secs(5), GroupMessage::Average)
        .await
        .context("asking for the average temperature")?;
    match average {
        Some(degrees) => println!("Average temperature: {degrees:.2}"),
        None => println!("No readings"),
    }

    system.stop();
    system
        .terminated()
        .await
        .context("awaiting actor system termination")
}

RUST_LOG=tellus=debug cargo run --quiet -p tellus --example getting_started_06_supervision

On stdout it prints:

Average temperature: 20.40

left failed and restarted, so it lost its 21.5 and has no reading. broken failed four times and stopped. right retired. Only middle is left with a reading.

On stderr tellus logs every failure and restart, as JSON because the example configures the tracing subscriber that way. The answer takes about two seconds, because the query waits until broken has used up its restarts and stopped. That is why main asks with a timeout of five seconds here.

What you learned

An actor fails by returning an error or panicking. Its supervision strategy decides whether it restarts or stops. A restart rebuilds the state but keeps the address and the mailbox. Together with watch, the group copes with devices failing for good.

In the last step you put a manager on top of the groups.