Notifications

A notification is an asynchronous signaling primitive. It stores a 64-bit word of pending signal bits. Signaling a notification performs an atomic OR on the bits and optionally wakes a waiting thread. Notifications never block the signaler.

Notification Structure

#[repr(C)]
pub struct Notification {
    pub header: KernelObject,
    lock: AtomicU8,           // per-notification spinlock
    pub bits: AtomicU64,      // pending signal bits
    waiting: *mut Tcb,        // thread blocked in Wait (at most one)
    pub bound_tcb: *mut Tcb,  // TCB with this notification bound
}

Each notification has its own spinlock with the same IRQ-disable pattern as endpoints.

Operations

Signal (syscall 5)

Signal a notification by OR-ing bits into the pending word. Never blocks.

  1. Acquire ntfn_lock.

  2. bits.fetch_or(signal_bits, SeqCst).

  3. If a thread is directly waiting (waiting != null): clear the waiter, wake it.

  4. Otherwise, if a bound TCB exists and is blocked on an endpoint recv: wake it via the bound notification mechanism (see Bound Notifications).

  5. Release ntfn_lock.

The signal caller’s rights must include SEND.

Wait (syscall 6)

Wait for notification bits. Blocks if no bits are pending.

  1. Atomically read and clear bits.

  2. If non-zero: return the bits to the caller immediately.

  3. If zero: set waiting = current_tcb, block the thread, release lock, reschedule.

When the thread is eventually woken by a Signal, it reads and clears the accumulated bits.

The waiter’s rights must include RECV.

Poll (syscall 7)

Non-blocking check for notification bits.

  1. Atomically swap bits with 0.

  2. Return the previous value (may be zero).

No blocking occurs regardless of the result.

Bound Notifications

A notification can be bound to a TCB via TCB_BIND_NOTIFICATION (invoke label 0x49). This creates a bidirectional link: notification.bound_tcb <→ tcb.bound_notification.

When a signal arrives on a bound notification and the bound TCB is blocked on an endpoint Recv, the kernel wakes the thread and sets woken_by_notification = true. The thread’s recv operation returns with the notification bits instead of an IPC message.

This allows a server to wait on both its endpoint (for client requests) and its notification (for IRQ or async events) in a single blocking call.

Combined Wait Behavior

During Recv on an endpoint:

  1. The kernel first checks the bound notification for pending bits (notification pre-check).

  2. If bits are pending, they are consumed and returned immediately — the thread never enters the endpoint’s recv queue.

  3. If no bits are pending, the thread blocks on the endpoint. A subsequent Signal on the bound notification wakes the thread from the endpoint queue.

Notification Dispatcher

If the bound TCB has a notification_dispatcher address set, signals can also interrupt Call-blocked threads:

  • A thread blocked in ReplyWait (waiting for a server reply after Call) is woken and redirected to the notification dispatcher.

  • A thread blocked in CallSendBlocked (waiting in an endpoint send queue during Call) is removed from the queue and woken.

This enables a thread to handle urgent notifications (IRQs, timer events) even while waiting for a server reply.

NotifReturn (syscall 27)

NotifReturn allows a thread to return from a notification dispatcher back to the interrupted IPC operation. It is the counterpart to the notification dispatcher mechanism: the dispatcher handles the event, then calls NotifReturn to resume the original blocking IPC.

IRQ Delivery

Hardware interrupts are delivered to userspace through notifications via IrqHandler objects.

IrqHandler Structure

#[repr(C)]
pub struct IrqHandler {
    pub header: KernelObject,
    pub irq_num: u32,
    pub notification: AtomicPtr<Notification>,
    pub acknowledged: AtomicBool,
    pub active: bool,
    pub level_triggered: bool,
    pub next: *mut IrqHandler,  // shared IRQ linked list
}

IRQ Delivery Path

Diagram

  1. When a hardware interrupt fires, the architecture-specific handler calls dispatch_irq(irq_num).

  2. dispatch_irq acquires IRQ_LOCK and walks the linked list of handlers registered for that IRQ.

  3. For each handler that is acknowledged and has a bound notification, the handler’s acknowledged flag is cleared and the notification is signaled with 1 << (irq_num % 64).

  4. The userspace driver, waiting on the notification via Wait, is woken and handles the interrupt.

  5. The driver acknowledges the interrupt by invoking IRQ_ACK (label 0x61) on the IRQ handler capability.

Shared IRQs

Multiple IrqHandler objects can be registered for the same IRQ line. They form a singly-linked list per IRQ number. dispatch_irq signals every acknowledged handler on the chain. Each handler is independently acknowledged.

IRQ Handler Operations

Label Name Description

0x60

IRQ_CONTROL_GET

Create a new IRQ handler for a given IRQ number (requires IRQ control capability).

0x61

IRQ_HANDLER_ACK

Acknowledge the interrupt, re-enabling delivery.

0x62

IRQ_HANDLER_SET_NOTIFICATION

Bind a notification to this IRQ handler.

0x63

IRQ_HANDLER_CLEAR

Unbind the notification.

0x64

DEVICE_UNTYPED_CREATE

Create a device untyped capability for an MMIO region.

Futex (syscall 18)

The futex syscall provides a userspace synchronization primitive. Three operations:

FUTEX_WAIT

If the value at the given address equals the expected value, block the calling thread. Used to implement userspace mutexes and condition variables.

FUTEX_WAKE

Wake up to N threads blocked on the given address.

FUTEX_WAIT_TIMEOUT

Block the calling thread until the futex value changes or the specified timeout expires. Returns TRONA_TIMEOUT if the timeout elapses.

Futex waiters are stored in a global hash table protected by FUTEX_LOCK.

Error Conditions and Edge Cases

Notification Destruction While Thread Is Waiting

When the last capability to a notification is deleted, Notification::cleanup() wakes the waiting thread (if any). The woken thread sees zero bits (the notification word was cleared).

If a TCB has this notification bound (bound_tcb), the binding is cleared during cleanup. Subsequent Recv on endpoints no longer perform the notification pre-check.

Signal During Lock-Order Inversion Window

Notification::signal() must release ntfn_lock before acquiring endpoint.lock when waking a bound TCB blocked on an endpoint. During this gap, the bound TCB’s state may change (another CPU wakes it, or it times out). The code re-validates the TCB’s state after reacquiring the endpoint lock (TOCTOU guard). If the TCB is no longer blocked, the signal bits remain in the notification word for the next Recv pre-check to consume.

Bound Notification + Notification Dispatcher Interaction

When a signal arrives on a bound notification and the bound TCB is in CallSendBlocked or ReplyWait:

  • If the TCB has a notification_dispatcher set: the TCB is woken and redirected to the dispatcher. PIP donations are cleaned up.

  • If the TCB does not have a dispatcher: the signal bits accumulate in the notification word. The TCB is not interrupted.

This distinction prevents servers that use bound notifications for IRQ delivery (without a dispatcher) from having their outgoing Call operations interrupted.

Futex Spurious Wakeup

FUTEX_WAIT can return without the value at the address having changed (spurious wakeup). Userspace must always re-check the condition in a loop:

while *addr == expected {
    futex_wait(addr, expected)
}

Futex Address Aliasing

The futex hash table keys on the virtual address within a single VSpace. Two processes sharing physical memory (via shared MO) at different virtual addresses see different futex keys. Cross-process futex requires mapping the shared region at the same virtual address in both VSpaces.

Related Pages

  • Endpoints — synchronous IPC

  • IPC Fastpath — assembly-optimized paths for Call and ReplyRecv

  • Design Patterns — combined server and IRQ driver patterns

  • Threads — TCB bound_notification field, notification dispatcher