The death watch ordering guarantee

An actor can watch another actor and gets a signal once that actor has terminated. tellus adds a guarantee to that signal: it arrives behind every message the terminated actor delivered to the watcher. This page explains what the guarantee says, why it holds and what you can build on it.

Watching

context.watch(&actor) registers interest in another actor's termination. Once that actor has terminated, the watcher receives Incoming::Terminated with the terminated actor's ActorId, in its ordinary mailbox, handled by receive like any message.

Watching is race-free: if the actor has already terminated, the signal is delivered right away. A terminated signal is delivered even when the watcher's bounded mailbox is full, and it proves that the terminated actor's destructors have run.

The guarantee

Receiving Terminated proves that the watcher has seen every message from that actor it will ever see. Each of them arrived before the signal or was dropped as a dead letter. None can arrive after it.

Why it holds

Messages from one sender arrive in the order they were sent, because a mailbox is a single FIFO queue. A watch hands the watched actor a way to send into exactly that queue. When the watched actor terminates, after its destructors have run, it sends the terminated signal through it, into the same queue as all the messages it delivered to the watcher before. The signal therefore queues up behind them. Nothing more is needed than the shared queue.

The guarantee covers delivered messages only. tell delivers at most once: a message dropped as a dead letter, for example because the watcher's bounded mailbox was full, never entered the queue. That is why the guarantee says "seen every message it will ever see" and not "received every message it sent".

What you can build on it

Waiting for answers without timeouts. An actor that asks several others and waits for their answers can watch each of them. When Terminated arrives for one, its answer cannot come anymore, so the actor stops waiting for it. The query actor in the tutorial works this way. It never hangs, and the only timeout is the one on ask at the boundary to the outside world.

Handovers. An actor can send a final message before it stops, and its watcher knows that this message comes before Terminated. In the tutorial, retiring devices hand over their last reading this way.

Registries. A parent that keeps its children in a map can watch them and remove an entry on Terminated. Since nothing from that child can arrive afterwards, the map never refers to an actor that might still send something.

Unwatching

context.unwatch(&actor) reverts a watch, and its contract is absolute: after it returns, the watcher never receives a terminated signal for that actor, even if the signal is already waiting in its mailbox. tellus enforces this on the watcher's side and drops such a signal before receive sees it. Unwatch as soon as you no longer care, for example once an actor has answered.

Across nodes

The upcoming cluster feature extends watch to actors on other nodes, through the same API. A real termination, reported over the wire, keeps the full local guarantee. If instead a node is declared down, for example after a network partition, its actors cannot report their own termination. The watcher then gets a synthesized signal, which guarantees that no message from that actor is ever delivered through this node again.

For the implementation, see Death watch in the internals documentation.