astatin3/unshell
1
0
Fork 0
mirror of https://github.com/Astatin3/unshell.git synced 2026-06-08 22:38:01 -06:00
Code Issues Packages Projects Releases Wiki Activity

Move protocol to workspace root.

Browse Source
This commit is contained in:
Michael Mikovsky
2026-05-31 08:58:08 -06:00
parent ca1daedebe
commit 0a44bc93de
29 changed files with 844 additions and 71 deletions
Download Patch File Download Diff File Expand all files Collapse all files
LEAF_MACRO_INTERFACE.md
+810
View File
@@ -0,0 +1,810 @@
# Macro-Generated Leaf Interface Design
**Status:** Draft
**Last updated:** 2026-05-28
**Primary use case:** Remote PTY sessions over hook-backed UnShell packets
## Summary
This document proposes a generated leaf interface for UnShell. The goal is to make
stateful leaves, such as a remote PTY leaf, easy to write without forcing every leaf
author to hand-code packet draining, procedure dispatch, session lookup, hook
lifetime handling, and retry-safe final-frame cleanup.
The user writes the application logic:
- the leaf state struct
- one or more session types for long-running hook-backed conversations
- one or more procedure types for one-packet operations
- payload encoding and decoding
- OS-specific behavior, such as spawning or polling a PTY
The macro generates the plumbing:
- the `Leaf` implementation
- the generated wrapper that owns session stores and retry queues
- inbound packet filtering for the leaf's procedure ids
- per-packet procedure dispatch
- per-hook session dispatch
- retry-safe outbound flushing
- final-frame session removal only after routing succeeds
The macro should generate ordinary Rust. No runtime registry, no boxed procedure
objects, and no hidden dynamic dispatch in the hot path.
## Problem
The current `Leaf` trait is deliberately small:
```rust
pub trait Leaf {
fn get_id(&self) -> u32;
fn update(&mut self, endpoint: &mut Endpoint);
}
```
That makes the protocol runtime flexible, but it also means every non-trivial leaf
has to solve the same set of problems by hand:
- select only the inbound packets that belong to this leaf
- distinguish one-shot procedures from long-running sessions
- group session packets by `hook_id`
- keep per-session application state
- build response packets with the right `hook_id`, path, procedure id, and `end_hook`
- retry failed outbound packets without losing stream progress
- remove session state only after final packets route successfully
- avoid consuming packets intended for other leaves
A remote PTY leaf makes this pain obvious. A PTY session is bidirectional and long
lived. The same hook carries `Open`, `Input`, `Resize`, `StdinEof`, `Output`, `Exit`,
and errors. The endpoint already owns hook authorization, but the leaf still needs a
safe session state machine.
## Goals
- Automate the repetitive `Leaf::update` machinery.
- Keep application logic explicit and testable.
- Keep `unshell-protocol` minimal and no_std-friendly.
- Keep OS-specific PTY code outside `unshell-protocol`.
- Preserve the new endpoint hook model: downward packets pave hooks, final packets close hooks.
- Make final-frame retries hard to get wrong.
- Keep generated runtime code static and size-conscious.
- Let sessions and procedures mutate shared leaf state without directly accessing each other.
- Make multiple sessions of the same type cheap and predictable.
## Non-Goals
- Do not define a full actor framework.
- Do not add async requirements to the protocol layer.
- Do not make sessions discover or mutate other sessions directly.
- Do not introduce `Vec<Box<dyn Procedure>>` or `Vec<Box<dyn Session>>` runtime registries.
- Do not hide PTY business logic inside the macro.
- Do not add source-path fields to `Packet` just for PTY. The PTY `Open` payload can carry the reply path.
## Current Protocol Assumptions
The endpoint routing layer now owns hook lifetime:
```text
validated downward packet, end_hook=false -> open or refresh peer-bound hook
validated downward packet, end_hook=true -> close hook after successful route or delivery
validated upward packet -> require matching hook
validated upward packet, end_hook=true -> close hook after successful route or delivery
```
For PTY, this means:
- `Open` uses `end_hook = false` because it expects returned output.
- `Input`, `Resize`, `StdinEof`, and `Terminate` use `end_hook = false`.
- `StdinEof` is not `end_hook`. EOF closes stdin, not the whole PTY session.
- `Abort` may use `end_hook = true` if no acknowledgement is expected.
- `Output` uses `end_hook = false`.
- `Exit` or fatal `Error` uses `end_hook = true`.
## Crate Boundary
```text
unshell-protocol
Endpoint
Packet
Leaf
hook routing rules
filtered inbound drain API
unshell-runtime or unshell-leaves
real PTY worker implementation
std-only integrations
portable-pty adapter
unshell-macros
tiny proc-macro shim
unshell-macros-core
syn, quote, proc-macro2, deluxe or darling
parser and code generator tests
```
The generated code runs in the final binary. Macro parsing dependencies do not.
That means `syn`, `quote`, `deluxe`, and `darling` are acceptable in the macro crates,
but not in the protocol runtime.
## Required Endpoint Addition
The generated leaf must be able to drain only the packets it owns. The current
`take_inbound_clear` drains a whole local queue, which is unsafe once multiple
application leaves share an endpoint.
Add a filtered drain API:
```rust
impl Endpoint {
/// Drains packets from `local_id` that match `predicate`, preserving all other
/// packets in their original relative order.
pub fn take_inbound_matching<P, F>(&mut self, local_id: u32, predicate: P, f: F)
where
P: FnMut(&Packet) -> bool,
F: FnMut(&Packet);
}
```
For generated leaves, matching usually means:
```rust
packet.procedure_id == PROC_PTY
|| packet.procedure_id == PROC_PING
|| packet.procedure_id == PROC_CAPABILITIES
```
This is the one endpoint API the macro needs before it can be safe in mixed-leaf endpoints.
## User-Facing Macro
Use an attribute macro that wraps a user-owned state struct and generates a leaf type.
A derive macro alone cannot add storage fields to the original struct, so the macro
must generate a companion wrapper.
```rust
#[unshell_leaf(
leaf = RemotePtyLeaf,
id = LEAF_REMOTE_PTY,
sessions(PtySession),
procedures(PingProcedure, CapabilitiesProcedure)
)]
pub struct RemotePtyState {
max_sessions: usize,
default_rows: u16,
default_cols: u16,
}
```
The macro emits roughly:
```rust
pub struct RemotePtyLeaf {
state: RemotePtyState,
pty_sessions: SessionStore<PtySession>,
retry: PacketQueue,
}
impl RemotePtyLeaf {
pub fn new(state: RemotePtyState) -> Self { ... }
}
impl Leaf for RemotePtyLeaf {
fn get_id(&self) -> u32 {
LEAF_REMOTE_PTY
}
fn update(&mut self, endpoint: &mut Endpoint) {
... generated dispatch ...
}
}
```
The leaf wrapper is the object stored in `Endpoint::leaves`. The state struct stays
small and owned by the user.
## Core Model
```text
Endpoint inbound queue
|
v
+-------------------------------+
| generated RemotePtyLeaf |
| |
| state: RemotePtyState |
| sessions: SessionStore |
| retry: PacketQueue |
+-------------------------------+
|
+--> Procedure packet -> Procedure::handle(...)
|
+--> Session packet -> by hook_id
create or update session
session queues outbound frames
generated flush handles retry
```
Procedures and sessions can both mutate `RemotePtyState`. They cannot directly
borrow or mutate each other. Communication between them must happen through leaf
state or through packets.
## Sessions
A session is a long-running hook-backed conversation. PTY is the main example.
```rust
pub trait Session<L> {
/// All packets for this session type use this outer procedure id.
const PROCEDURE_ID: u32;
/// State owned for one active hook/session.
type State;
/// Attempts to create a new session from an incoming packet.
fn init(
leaf: &mut L,
packet: Packet,
ctx: &mut SessionInit,
) -> SessionInitResult<Self::State>;
/// Advances one session. The generated leaf passes all queued packets for this
/// hook and one context that can enqueue outbound frames.
fn update(
leaf: &mut L,
session: &mut Self::State,
incoming: &mut PacketQueue,
ctx: &mut SessionCtx,
) -> SessionStatus;
}
```
`L` is the user state type, for example `RemotePtyState`.
### Session Init Context
```rust
pub struct SessionInit {
hook_id: HookID,
packet_path: Vec<u32>,
}
pub enum SessionInitResult<S> {
Created(S),
Rejected,
RejectedWith(Packet),
}
```
`RejectedWith(Packet)` is intended for cases where the initializer can build a
protocol-level failure response, such as "too many PTY sessions". The generated leaf
still owns routing and retry for that packet.
The PTY `Open` payload should include the caller reply path because `Packet` does
not currently carry source path.
```text
Open payload:
opcode
reply_path_len
reply_path segments
rows
cols
command/env/options
```
The session stores that reply path and uses it for upward output packets.
### Session Update Context
Sessions should use a context wrapper rather than directly constructing packets.
The context can still carry restricted endpoint access when absolutely necessary,
but the normal output path should be helper methods.
```rust
pub struct SessionCtx<'a> {
endpoint: &'a mut Endpoint,
hook_id: HookID,
reply_path: &'a [u32],
procedure_id: u32,
outbox: &'a mut PacketQueue,
}
```
Helpers:
```rust
impl<'a> SessionCtx<'a> {
pub fn send(&mut self, opcode: u8, data: &[u8]);
pub fn send_final(&mut self, opcode: u8, data: &[u8]);
pub fn error(&mut self, code: u8, data: &[u8]);
pub fn error_final(&mut self, code: u8, data: &[u8]);
}
```
These helpers build packets like:
```rust
Packet {
hook_id: self.hook_id,
end_hook,
path: self.reply_path.to_vec(),
procedure_id: self.procedure_id,
data: encode_frame(opcode, payload),
}
```
The helper only queues packets. It does not route them immediately. The generated
leaf owns flushing and retry.
### Session Status
```rust
pub enum SessionStatus {
Running,
Closing,
Closed,
}
```
`Closed` means the session has no more application work. The generated leaf still
must keep the session until any final packet has routed successfully.
## Procedures
A procedure is a one-packet operation. It is appropriate for introspection, ping,
capabilities, and simple state queries.
```rust
pub trait Procedure<L> {
const PROCEDURE_ID: u32;
fn handle(
leaf: &mut L,
endpoint: &mut Endpoint,
packet: Packet,
out: &mut ProcedureOut,
);
}
```
Procedure helpers mirror session helpers but operate on one incoming packet:
```rust
pub struct ProcedureOut {
hook_id: HookID,
reply_path: Vec<u32>,
procedure_id: u32,
outbox: PacketQueue,
}
impl ProcedureOut {
pub fn send(&mut self, data: &[u8]);
pub fn send_final(&mut self, data: &[u8]);
}
```
Procedures do not directly access sessions. If a procedure needs information about
sessions, that information should be mirrored into leaf state by the session code.
## PTY Binary Protocol
One PTY session uses one hook and one outer procedure id. The inner PTY protocol is
a tiny binary frame in `Packet::data`.
If API names use `end_state` at the session layer, it maps directly to
`Packet::end_hook` at the protocol layer.
```text
Packet {
hook_id: session hook,
procedure_id: PROC_PTY,
data: [opcode, payload...],
end_hook: session lifetime marker,
}
```
Suggested opcodes:
| Opcode | Direction | Meaning | end_hook |
|---:|---|---|---|
| 0 | Downward | Open PTY | false |
| 1 | Upward | Opened | false |
| 2 | Downward | Input bytes | false |
| 3 | Downward | Resize | false |
| 4 | Downward | Stdin EOF | false |
| 5 | Downward | Terminate process | false |
| 6 | Downward | Abort session without acknowledgement | true |
| 7 | Upward | Output bytes | false |
| 8 | Upward | Exit status | true |
| 9 | Upward | Fatal error | true |
`StdinEof` must not set `end_hook = true`. The remote process may still emit output
after stdin closes.
## Generated Update Loop
The generated `Leaf::update` should follow this shape:
```text
update(endpoint)
1. flush retry queue first
2. drain matching inbound packets only
3. dispatch procedure packets
4. group session packets by hook_id
5. create sessions for open packets
6. update active sessions
7. flush outbound frames
8. remove sessions whose final packet routed successfully
```
More concrete flow:
```text
+----------------------------+
| generated update |
+----------------------------+
|
v
flush retry packets
|
v
take_inbound_matching(local_id, owns_packet)
|
+--> procedure id match -> Procedure::handle
|
+--> session id match -> session inbox by hook_id
|
v
for each session inbox
|
+--> existing hook -> Session::update
|
+--> no hook -> Session::init, then Session::update
|
v
flush session/procedure outbox
|
+--> route success -> advance/remove when safe
|
+--> route failure -> keep retry packet and keep state
```
The retry rule is the most important generated invariant:
```text
If endpoint.add_outbound(packet) fails, the generated leaf must not:
- drop the packet
- advance a final frame
- remove the session
- consume state that cannot be reconstructed
```
## Generated Dispatch
The macro should emit static matches, not runtime registries.
Good:
```rust
match packet.procedure_id {
PtySession::PROCEDURE_ID => self.dispatch_pty_session(packet),
PingProcedure::PROCEDURE_ID => PingProcedure::handle(...),
CapabilitiesProcedure::PROCEDURE_ID => CapabilitiesProcedure::handle(...),
_ => self.handle_unknown(packet),
}
```
Avoid:
```rust
Vec<Box<dyn Procedure>>
Vec<Box<dyn Session>>
```
Static dispatch keeps the generated code visible to LLVM and avoids registry setup
costs in constrained binaries.
## Session Store
The first implementation can use a small `Vec` or `VecDeque` store.
```rust
pub struct SessionEntry<S> {
hook_id: HookID,
state: S,
inbox: PacketQueue,
outbox: PacketQueue,
retry: Option<Packet>,
closing: bool,
}
```
For minimal binaries, a later implementation can replace this with a fixed-capacity
table under a feature flag.
Required operations:
- find by `hook_id`
- insert if capacity allows
- push incoming packet to inbox
- enqueue outbound packet
- retain session while retry packet exists
- remove only after session is closed and outbox/retry are empty
## Unknown Packets
The generated leaf should have configurable unknown-packet behavior.
Default behavior:
- unknown procedure id remains in the endpoint queue because `take_inbound_matching` does not drain it
- unknown session opcode for a known session produces a fatal error frame and closes the session
- packet for unknown hook under a session procedure produces a fatal error frame if the payload is not a valid `Open`
For PTY:
```text
unknown hook + Open opcode -> create session
unknown hook + non-Open opcode -> Error end_hook=true
known hook + known opcode -> session update
known hook + unknown opcode -> Error end_hook=true
```
## Access Boundaries
Sessions and procedures can mutate leaf state:
```rust
fn update(leaf: &mut RemotePtyState, ...)
```
They should not directly access each other:
```text
Procedure -> no direct SessionStore access
Session -> no direct Procedure access
```
If cross-cutting data is needed, mirror it into `RemotePtyState`:
```rust
pub struct RemotePtyState {
active_count: usize,
total_spawned: u64,
max_sessions: usize,
}
```
This keeps borrowing simple and avoids turning the generated leaf into a shared
mutable object graph.
## Endpoint Access
The user proposed passing `&mut Endpoint` to sessions and procedures. The safest
design is slightly narrower:
- procedures may receive `&mut Endpoint` because they handle one packet and return immediately
- sessions should receive `SessionCtx`, which can expose narrow endpoint helpers
- a raw endpoint escape hatch can be added later if a real leaf proves it needs one
Rationale: sessions are long-lived and retry-sensitive. If session code calls
`endpoint.add_outbound` directly, it can bypass generated retry handling and lose a
final packet. The helper path keeps the dangerous part centralized.
## Remote PTY Example
User code:
```rust
#[unshell_leaf(
leaf = RemotePtyLeaf,
id = LEAF_REMOTE_PTY,
sessions(PtySession),
procedures(PtyCapabilities)
)]
pub struct RemotePtyState {
max_sessions: usize,
default_rows: u16,
default_cols: u16,
}
pub struct PtySession;
pub struct PtyState {
hook_id: HookID,
reply_path: Vec<u32>,
worker: PtyWorker,
saw_stdin_eof: bool,
}
impl Session<RemotePtyState> for PtySession {
const PROCEDURE_ID: u32 = PROC_PTY;
type State = PtyState;
fn init(
leaf: &mut RemotePtyState,
packet: Packet,
ctx: &mut SessionInit,
) -> SessionInitResult<Self::State> {
let open = decode_open(&packet.data)?;
if leaf.active_count >= leaf.max_sessions {
return SessionInitResult::RejectedWith(pty_error_busy(packet.hook_id));
}
let worker = PtyWorker::spawn(open.command, open.rows, open.cols)?;
SessionInitResult::Created(PtyState {
hook_id: ctx.hook_id(),
reply_path: open.reply_path,
worker,
saw_stdin_eof: false,
})
}
fn update(
leaf: &mut RemotePtyState,
session: &mut Self::State,
incoming: &mut PacketQueue,
ctx: &mut SessionCtx,
) -> SessionStatus {
while let Some(packet) = incoming.pop_front() {
match decode_pty_frame(&packet.data) {
PtyFrame::Input(bytes) => session.worker.write(&bytes),
PtyFrame::Resize { rows, cols } => session.worker.resize(rows, cols),
PtyFrame::StdinEof => {
session.saw_stdin_eof = true;
session.worker.close_stdin();
}
PtyFrame::Terminate => session.worker.terminate(),
PtyFrame::Abort => return SessionStatus::Closed,
_ => ctx.error_final(ERR_BAD_OPCODE, b"bad pty opcode"),
}
}
while let Some(bytes) = session.worker.poll_output() {
ctx.send(OP_OUTPUT, &bytes);
}
if let Some(status) = session.worker.poll_exit() {
ctx.send_final(OP_EXIT, &encode_exit(status));
leaf.active_count -= 1;
return SessionStatus::Closed;
}
SessionStatus::Running
}
}
```
The exact error handling syntax above is illustrative. The final API should avoid
`?` unless `SessionInitResult` has a clear conversion story.
## Generated Remote PTY Flow
```text
Caller Remote endpoint RemotePtyLeaf
------ --------------- -------------
allocate hook
Open end=false -------------> downward route opens hook ----> init PtyState
Input false -------------> hook remains active -----------> write PTY stdin
Resize false -------------> hook remains active -----------> resize PTY
StdinEof false -------------> hook remains active -----------> close PTY stdin
Output false <------------- upward route requires hook <---- ctx.send
Output false <------------- upward route requires hook <---- ctx.send
Exit true <------------- upward route closes hook <---- ctx.send_final
caller cleanup remove session
```
## Proc Macro Implementation
Rust still requires a proc-macro crate for real attribute macros. Keep that crate
thin and put the real logic in a normal testable crate.
```text
unshell-macros
#[proc_macro_attribute]
pub fn unshell_leaf(attr, item) -> TokenStream
unshell-macros-core
parse UnshellLeafArgs
parse state struct
generate wrapper
generate Leaf impl
generate dispatch methods
tests over proc_macro2::TokenStream
```
Recommended parser helper:
- `deluxe` for attribute parsing if the desired syntax uses richer Rust tokens
- `darling` if the syntax stays close to normal Rust meta attributes
Current recommendation: use `deluxe` for attribute parsing and `syn`/`quote` for
code generation.
## Generated Code Requirements
- Must compile without requiring std in `unshell-protocol`.
- Must not allocate handler trait objects.
- Must preserve generic parameters and where clauses on the state struct.
- Must emit useful compile errors for duplicate procedure ids.
- Must emit useful compile errors for duplicate session procedure ids.
- Must reject session and procedure id collisions unless explicitly allowed.
- Must preserve user attributes that are not consumed by the macro.
- Must not require users to manually implement `Leaf` for the generated wrapper.
## Testing Strategy
Phase 1 tests should use a fake PTY session, not `portable-pty`.
Protocol tests:
- `open_pty_paves_hook_and_creates_session`
- `input_and_output_share_one_hook`
- `stdin_eof_keeps_hook_until_exit`
- `exit_end_hook_cleans_route_and_session`
- `failed_final_exit_route_retries_without_losing_session`
- `abort_downward_end_hook_closes_without_ack`
- `unknown_session_input_returns_error_end_hook`
- `two_pty_sessions_interleave_without_crossing_hooks`
- `pty_leaf_does_not_consume_other_leaf_packets`
Macro tests:
- generated leaf implements `Leaf`
- duplicate procedure ids fail at compile time
- duplicate session procedure ids fail at compile time
- generic state structs expand correctly
- generated code routes final frames retry-safely
- generated code preserves unmatched inbound packets
Use `trybuild` for compile-fail macro tests once `unshell-macros` exists.
## Rollout Plan
1. Add `Endpoint::take_inbound_matching`.
2. Write a manual fake PTY leaf that follows the exact generated shape.
3. Add PTY session tests against the manual fake leaf.
4. Create `unshell-macros` and `unshell-macros-core`.
5. Generate the same shape as the manual fake leaf.
6. Port fake PTY tests to the generated leaf.
7. Add compile-fail macro tests.
8. Implement the real std-only PTY worker in `unshell-leaves` or `unshell-runtime`.
9. Add integration tests for the real worker where the platform supports PTYs.
This avoids writing a macro against an unproven design. First make the generated
shape real by hand, then teach the macro to emit it.
## Open Questions
- Should `SessionCtx` expose any raw `Endpoint` access, or only narrow helpers?
- Should session stores be `Vec` first, or should fixed-capacity storage be designed immediately?
- Should unknown opcodes produce an error packet by default, or should each session type decide?
- Should `Open` always carry `reply_path`, or should the packet format eventually add source path?
- Should `ProcedureOut` be retry-safe like session output, or are procedures allowed to fail fast?
- Should macro-generated leaves expose counters for active sessions and retry queue depth?
## Recommendation
Implement this as static generated code with a narrow context API.
Do not build a runtime plugin system. Do not give sessions raw access to session
stores. Do not let session code bypass generated output flushing. The macro should
make the correct routing and hook behavior the easiest path, especially for final
frames.
The first proof point should be fake PTY. If fake PTY works cleanly, real PTY is
mostly OS plumbing.