> ## Documentation Index
> Fetch the complete documentation index at: https://docs.declaw.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# sandbox MicroVM

> How Declaw uses sandbox microVMs for sandbox isolation: rootfs, TAP networking, boot process, and the envd in-VM daemon.

Declaw uses [sandbox](https://firecracker-sandbox.github.io/) — an open-source sandbox monitor developed by AWS — to provide hardware-level isolation for each sandbox. Each sandbox is a separate VM with its own Linux kernel, memory, and I/O devices. The boot time is approximately 125 milliseconds.

## VM structure

```mermaid theme={null}
graph LR
    subgraph host [Host Machine]
        Orch["Orchestrator\n(Go)"]
        HostNs["Host network\nnamespace"]
        NS["Sandbox namespace\n(ns-sbx-abc123)"]
        Veth1["veth0 (host side)\n172.20.0.1/30"]
        Veth2["veth0 (ns side)\n172.20.0.2/30"]
        TAP["TAP device\n(ns side)\n172.16.0.1/24"]
    end

    subgraph vm [sandbox MicroVM]
        Kernel["Linux kernel\n(vmlinux)"]
        Rootfs["OverlayFS rootfs\n(base + overlay)"]
        EnvdProc["envd\nPID 1 | :49983"]
        SecProxyProc["security-proxy\n:8443"]
        Workload["Agent workload"]
        GuestNIC["eth0\n172.16.0.2/24"]
    end

    Orch -->|"Socket API"| sandbox
    Veth1 --- HostNs
    Veth2 --- NS
    TAP --- NS
    GuestNIC <-->|"TAP bridge"| TAP
    EnvdProc --> Kernel
    SecProxyProc --> Kernel
    Workload --> Kernel
```

## Rootfs isolation

Each sandbox gets an independent rootfs through Linux OverlayFS:

1. **Base image** (`/opt/declaw/rootfs/rootfs.ext4`): Read-only ext4 image shared across all sandboxes. Contains the OS, packages, and tools from the selected template.

2. **Overlay image** (`/opt/declaw/run/<sandbox_id>/overlay.ext4`): Per-sandbox writable ext4 image. All writes go here. Deleting a sandbox deletes only this file.

3. **OverlayFS mount**: The VM sees the merged view — the base image as the lower layer and the overlay as the upper layer.

```
/opt/declaw/run/sbx-abc123/
├── overlay.ext4       # writable per-sandbox layer (fast-formatted at creation)
├── fc.sock            # sandbox API socket
├── fc.log             # sandbox VM logs
└── ca.pem             # per-sandbox CA cert (if PII/injection enabled)
```

## Sandbox ID validation

Sandbox IDs must match the pattern `sbx-[a-z0-9][a-z0-9-]{0,63}` before they are used in any filesystem path. The orchestrator rejects IDs containing `..`, `/`, or any path separator. This prevents path traversal attacks where a crafted sandbox ID like `sbx-../../../etc` could resolve to an unintended directory.

```go theme={null}
// From infra/orchestrator/internal/sandbox/firecracker.go
var validSandboxID = regexp.MustCompile(`^sbx-[a-z0-9][a-z0-9-]{0,63}$`)

func ValidateSandboxID(id string) error {
    if !validSandboxID.MatchString(id) {
        return fmt.Errorf("invalid sandbox ID %q", id)
    }
    if strings.Contains(id, "..") || strings.Contains(id, "/") {
        return fmt.Errorf("invalid sandbox ID %q: contains path traversal", id)
    }
    return nil
}
```

## Network setup

Each sandbox gets a dedicated Linux network namespace (`ns-sbx-<id>`) with:

* **veth pair**: One end in the host network namespace (for routing), one end in the sandbox namespace.
* **TAP device**: Inside the sandbox namespace, bridges the namespace to the sandbox VM's virtual NIC.
* **iptables rules**: Applied to the veth interface in the sandbox namespace for CIDR-based network policies.

```
Host namespace
└── veth0 (172.20.0.1/30)

Sandbox namespace (ns-sbx-abc123)
├── veth0 (172.20.0.2/30)
├── tap0  (172.16.0.1/24)
└── iptables REDIRECT rules

sandbox VM
└── eth0 (172.16.0.2/24, gateway 172.16.0.1)
```

### Network slot pool

The orchestrator maintains a pool of 256 pre-allocated network slots. Each slot includes a pre-created network namespace, veth pair, and TAP device. When a sandbox is created, a slot is claimed from the pool (instant), and the pool replenishes in the background. This eliminates networking setup time from the sandbox creation hot path.

## Boot process

```mermaid theme={null}
sequenceDiagram
    participant Orch as Orchestrator
    participant FC as sandbox
    participant Envd as envd (PID 1)
    participant SecP as security-proxy

    Orch->>Orch: Validate sandbox ID
    Orch->>Orch: Claim network slot (pool)
    Orch->>Orch: Copy overlay from pool or format new
    Orch->>Orch: Generate per-sandbox CA (if PII/injection enabled)
    Orch->>FC: Start sandbox in sandbox namespace
    FC->>FC: Wait for API socket
    Orch->>FC: Configure via API socket (kernel, rootfs, network, vcpus, mem)
    Orch->>FC: Boot VM
    FC->>Envd: Start envd (PID 1)
    Envd->>SecP: Start security-proxy
    SecP->>SecP: Apply iptables REDIRECT rules
    SecP->>SecP: Load CA cert into trust store
    Envd-->>Orch: Ready signal on :49983
    Orch-->>API: SandboxReady
```

Cold boot time from `CreateVM()` call to envd ready is approximately 125 milliseconds. With snapshot restore, it drops to approximately 30 milliseconds.

## Resource defaults

| Resource       | Default       | Range                 |
| -------------- | ------------- | --------------------- |
| vCPUs          | 1             | 1–8                   |
| Memory         | 256 MB        | 128 MB–8 GB           |
| Disk (overlay) | 20 GB ext4    | Fixed                 |
| Boot time      | \~125 ms cold | \~30 ms from snapshot |

## Concurrency limits

The orchestrator's Firecracker manager enforces two semaphores to prevent resource exhaustion:

* **Create semaphore**: Maximum 1024 concurrent sandbox creations. Prevents OOM when many sandboxes start simultaneously.
* **Exec semaphore**: Set to `4 × CPU cores` (e.g., 128 slots on a 32-core machine). Limits concurrent `fork+exec` calls to reduce CPU contention during parallel sandbox creation.

## envd: the in-VM daemon

`envd` is a Go binary that starts as PID 1 inside every sandbox VM. It exposes an HTTP/REST server (with SSE for streaming) on port 49983 (accessible via the TAP/veth bridge from the host) with three service groups:

| Service    | Methods                                                                        |
| ---------- | ------------------------------------------------------------------------------ |
| Filesystem | Read, Write, WriteBatch, ListDir, Exists, Info, Remove, Rename, MakeDir, Watch |
| Process    | Start, Wait, Kill, SendStdin, List                                             |
| PTY        | Create, Kill, SendStdin, Resize                                                |

All communication between the SDK and the sandbox goes through the orchestrator, which relays HTTP calls to envd via the private veth pair. This traffic never crosses the public network.

## Snapshot mechanism

When `create_snapshot()` is called:

1. The VM is briefly paused (CRIU-style memory dump)
2. The memory image and overlay diff are written to GCS (GCP) or S3 (AWS)
3. The VM resumes

When restoring from a snapshot:

1. A fresh network slot is claimed
2. The memory image and overlay are downloaded and placed in the sandbox directory
3. sandbox restores the VM from the memory image (\~30 ms)
4. envd is already running — no boot phase