unshell/unshell-protocol/src/protocol/PROTOCOL_CHANGES.md

# Protocol Change Pressure

This document records protocol-spec changes that are worth considering after the
runtime rewrite in `src/protocol`.

The current rewrite intentionally keeps the existing wire model from
`/home/astatin3/Documents/GitHub/unshell/PROTOCOL.md` wherever possible. The main
goal was to remove avoidable runtime work without silently drifting the protocol.

The implementation now does the following:

- compiles child routing prefixes once instead of scanning child paths on every packet
- routes from the header first, then decodes payloads only on local delivery
- keeps pending hook state minimal and active hook state directly indexed
- separates local typed send paths from framed transport-facing send paths

Those are implementation changes. They do not require a protocol update.

## Implemented Deviation

The current scratch rewrite **does** deviate from the frame format described in
`PROTOCOL.md` Section 8.

The old format used one `u32` length prefix immediately before each archived
section. The new implementation uses one aligned two-section frame:

- `u32 header_len`
- `u32 payload_len`
- aligned archived header bytes
- aligned archived payload bytes

The payload start is padded up to the canonical archive alignment boundary.

This deviation was made explicitly because the prior layout baked in alignment
repair complexity and extra decode copies even in an otherwise clean runtime.

## No Immediate Semantic Change Required

Aside from the framing change above, the current runtime rewrite does **not**
require a semantic protocol break.

The following parts of `PROTOCOL.md` remain worth keeping as-is:

- path-based routing remains the canonical behavior
- pending call context remains distinct from active hook state
- `Fault` remains upstream-only
- unknown or expired `hook_id` still drops returned traffic
- hook closure still requires both sides to send `end_hook = true`, or one `Fault`

Those rules keep the protocol boring and interoperable.

## Change 1: Framing That Guarantees Archive Alignment

### Current problem

`PROTOCOL.md` Section 8 fixes a framed format with a 4-byte big-endian length
prefix before each archived section.

That is simple, but it has one hard performance downside in the current Rust
implementation:

- the start of the archived section is not guaranteed to satisfy `rkyv` alignment
- the decoder therefore has to copy header bytes into an `AlignedVec` before safe access
- local payload decode also copies the payload bytes into another `AlignedVec`

This means the runtime still performs unavoidable memory copies during decode even
after the architectural cleanup.

### Recommended protocol change

Revise the framing rules so each archived section begins at a guaranteed aligned
offset.

Two viable options:

1. Add explicit padding after each length field so the archived section begins at
   the required alignment boundary.
2. Replace the current two-section frame with one canonical aligned envelope type
   whose internal layout already satisfies the archive alignment rules.

### Why this is objectively better

- removes the forced alignment-copy step on decode
- makes zero-copy or near-zero-copy archived access actually achievable
- reduces local delivery latency for all packet types
- reduces transient allocation pressure in the decoder

### Tradeoff

This is a wire-format change. Every compliant implementation would need to adopt
the new framing.

### Status

Implemented in the current rewrite.

## Change 2: Compact Path Representation for a Future v2

### Current problem

`PROTOCOL.md` Sections 5, 6, 10, 11, and 13 make paths canonical on the wire as
`Vec<String>` values.

That is easy to understand and debug, but it imposes real cost:

- path routing requires segment-wise string comparison
- hook state keys carry owned path vectors
- packets repeat full path strings over and over
- the runtime must repeatedly compare or clone path structures at boundaries

The new implementation minimizes those costs internally, but it cannot eliminate
them while the wire format remains path-string based.

### Recommended protocol change

For a future protocol version, consider separating:

- the canonical human-readable control/discovery layer
- the compact transport/runtime layer

The compact transport/runtime layer would use stable numeric endpoint IDs instead
of repeated `Vec<String>` path payloads.

### Why this is objectively better

- routing becomes integer-based instead of string-prefix based
- hook keys become compact and cheap to index
- packets shrink
- path comparisons and many path clones disappear from the hot path

### Tradeoff

This is a full protocol-versioning decision, not a local cleanup.

It adds coordination costs:

- peers must agree on endpoint IDs
- topology updates become more structured
- the protocol becomes less self-describing on the wire

### Recommendation

Do **not** make this change as a silent update to the current protocol.

If pursued, it should be introduced explicitly as a `v2` protocol, because it is
no longer behaviorally equivalent to the current path-based wire model.

## Change 3: Clarify Caller-Side Hook Activation Semantics

### Current problem

`PROTOCOL.md` Section 13 is explicit about callee-side pending call context, but
it leaves more room for interpretation on the caller side after a `Call` is sent.

The current runtime keeps caller-side hook state available immediately after send
so it can validate returned traffic efficiently.

That is practical, but the spec could be clearer about whether the caller's local
hook record is considered active immediately, or merely reserved until the callee
accepts.

### Recommended protocol change

Clarify caller-side wording in Section 13 so implementations know whether the
caller may allocate directly into active host state after sending a `Call`, as
long as early returned `Data` for an actually inactive hook is still discarded per
Section 14.1.

### Why this is objectively better

- removes ambiguity for optimized runtimes
- makes caller-side hook bookkeeping more consistent across implementations
- avoids accidental spec drift through inference

### Tradeoff

This is a clarification change, not necessarily a wire-format change.

## Summary

The runtime rewrite shows that most of the original performance problems were
architectural, not inherent to the protocol.

The current protocol can support a much lower-loop implementation than before.

The main remaining protocol-level blocker is the framing/alignment rule. That is
the one change most worth making if the next goal is to reduce unavoidable memory
copies further.