Skip to content

Improve DEV-LOCAL.md with prerequisites, verification steps, and troubleshooting #2273

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jakubno merged 2 commits into from
Apr 8, 2026
Merged

Improve DEV-LOCAL.md with prerequisites, verification steps, and troubleshooting #2273

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9c4d5a4
Improve local dev documentation with prerequisites, verification step…
renezander030 Mar 31, 2026
ddbe93e
Apply suggestions from code review
jakubno Apr 7, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
262 changes: 235 additions & 27 deletions DEV-LOCAL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,53 +2,261 @@

> Note: Linux is required for developing on bare metal. This is a work in progress. Not everything will function as expected.

## Prerequisites

| Requirement | Minimum | Notes |
|---|---|---|
| OS | Linux (bare metal or VM with nested virtualization) | KVM support required for Firecracker |
| CPU | 4+ cores recommended | Firecracker spawns microVMs |
| RAM | 8 GB minimum, 16 GB recommended | Huge pages alone reserve ~4 GB |
| Go | - | See `go.work` for the exact version |
| Docker / Docker Compose | Docker Engine 24+ with Compose v2 | `docker compose version` to verify |
| Node.js / npm | Required for building base template | `node --version` |
| gsutil | Google Cloud SDK CLI | Used to download prebuilt kernels and firecrackers |

**KVM check** — the orchestrator runs Firecracker microVMs, which require `/dev/kvm`:
```bash
ls -l /dev/kvm
# If missing: sudo modprobe kvm_intel (or kvm_amd)
# For cloud VMs, ensure nested virtualization is enabled.
```

## System prep
1. `sudo modprobe nbd nbds_max=64`
2. `sudo sysctl -w vm.nr_hugepages=2048` enable huge pages

1. Load the NBD (Network Block Device) kernel module:
```bash
sudo modprobe nbd nbds_max=64
```
Verify: `lsmod | grep nbd` should show the module loaded.

2. Enable huge pages:
```bash
sudo sysctl -w vm.nr_hugepages=2048
```
Verify: `grep HugePages_Total /proc/meminfo` should show `2048`.

> To persist these across reboots, add `nbd nbds_max=64` to `/etc/modules-load.d/nbd.conf` and `vm.nr_hugepages=2048` to `/etc/sysctl.d/99-hugepages.conf`.

## Download prebuilt artifacts (customized firecrackers and linux kernels)

1. `make download-public-kernels` download linux kernels
Requires `gsutil` from the Google Cloud SDK.

1. `make download-public-kernels` download linux kernels
2. `make download-public-firecrackers` download firecracker versions

Verify:
```bash
ls packages/fc-kernels/ # should contain kernel binaries
ls packages/fc-versions/builds/ # should contain firecracker binaries (executable)
```

## Run the local infrastructure
1. `make local-infra` runs clickhouse, grafana, loki, memcached, mimir, otel, postgres, redis, tempo

```bash
make local-infra
```

This runs the following Docker Compose services (defined in `packages/local-dev/docker-compose.yaml`):

### Required services (core functionality)

| Service | Purpose | Port(s) |
|---|---|---|
| **postgres** | Primary database (users, teams, templates) | 5432 |
| **clickhouse** | Analytics and sandbox metrics | 8123 (HTTP), 9000 (native) |
| **redis** | Caching, pub/sub, orchestrator coordination | 6379 |

### Observability stack (optional for basic operation)

These services provide logging, tracing, and metrics. The application runs without them, but you will see connection errors in logs. For a minimal setup you can comment them out in `packages/local-dev/docker-compose.yaml`.

| Service | Purpose | Port(s) |
|---|---|---|
| **grafana** | Dashboards and log/trace viewer | 53000 |
| **loki** | Log aggregation | 3100 |
| **tempo** | Distributed tracing | 3200 |
| **mimir** | Metrics storage (Prometheus-compatible) | — |
| **memcached** | Cache for Tempo | 11211 |
| **otel-collector** | OpenTelemetry collector | 4317 (gRPC), 4318 (HTTP) |
| **vector** | Log forwarder (ships to Loki) | 30006 |

Verify: `docker compose -f packages/local-dev/docker-compose.yaml ps` — all services should show `running` (except `tempo-init` which exits after setup).

## Prepare local environment

1. `make -C packages/db migrate-local` initialize the database
2. `make -C packages/clickhouse migrate-local` initialize the clickhouse database
3. `make -C packages/envd build` create the envd that will be embedded in templates
4. `make -C packages/local-dev seed-database` generate user, team, and token for local development
1. Initialize the Postgres database:
```bash
make -C packages/db migrate-local
```

2. Initialize the ClickHouse database:
```bash
make -C packages/clickhouse migrate-local
```

3. Build envd (embedded in sandbox templates):
```bash
make -C packages/envd build
```
Verify: `ls packages/envd/bin/envd` exists.

4. Seed the database with a local dev user, team, and API tokens:
```bash
make -C packages/local-dev seed-database
```
This creates a user (`user@e2b-dev.local`), a team (`local-dev team`), and the API tokens shown in [Client configuration](#client-configuration) below.

## Run the application locally

These commands will launch each service in the foreground, and will need multiple terminal windows.
These commands launch each service in the foreground. You need multiple terminal windows (or use `tmux`/`screen`).

1. **API server** (terminal 1):
```bash
make -C packages/api run-local
```
Verify: `curl -s http://localhost:3000/health` returns a response.

- `make -C packages/api run-local` run the api locally
- `make -C packages/orchestrator build-debug && sudo make -C packages/orchestrator run-local` run the orchestrator and template-manager locally.
- `make -C packages/client-proxy run-local` run the client-proxy locally.
2. **Orchestrator + template manager** (terminal 2):
```bash
make -C packages/orchestrator build-debug && sudo make -C packages/orchestrator run-local
```
> `sudo` is required because the orchestrator manages Firecracker microVMs, which need root access for KVM, networking (TAP devices), and cgroup management.

Verify: `curl -s http://localhost:5008/health` returns a response.

3. **Client proxy** (terminal 3):
```bash
make -C packages/client-proxy run-local
```
Verify: `curl -s http://localhost:3003/health` returns a response.

## Build the base template
- `make -C packages/shared/scripts local-build-base-template` instructs orchestrator to create the 'base' template

# Services
- grafana: http://localhost:53000
- postgres: postgres:postgres@127.0.0.1:5432
- clickhouse (http): http://localhost:8123
- clickhouse (native): clickhouse:clickhouse@localhost:9000
- redis: localhost:6379
- otel collector (grpc): localhost:4317
- otel collector (http): localhost:4318
- vector: localhost:30006
- e2b api: http://localhost:3000
- e2b client proxy: http://localhost:3002
- e2b orchestrator: http://localhost:5008

# Client configuration
Once all services are running:

```bash
make -C packages/shared/scripts local-build-base-template
```

This instructs the orchestrator to create the `base` sandbox template, which is required before you can create sandboxes.

## Verify it works (hello world)

After starting all services and building the base template, verify end-to-end operation:

```bash
# Create a sandbox using the local API
curl -s -X POST http://localhost:3000/sandboxes \
-H "X-API-Key: e2b_53ae1fed82754c17ad8077fbc8bcdd90" \
-H "Content-Type: application/json" \
-d '{"templateID": "base"}'
```

A successful response returns a JSON object with a `sandboxID`. You can then interact with the sandbox through the client proxy at `localhost:3002`.

Using the Python or JS SDK:
```python
import e2b

# Configure for local development
sandbox = e2b.Sandbox(
api_key="e2b_53ae1fed82754c17ad8077fbc8bcdd90",
api_url="http://localhost:3000",
sandbox_url="http://localhost:3002",
template="base",
)
print(sandbox.id) # sandbox is running locally
```

## Client configuration

```dotenv
E2B_API_KEY=e2b_53ae1fed82754c17ad8077fbc8bcdd90
E2B_ACCESS_TOKEN=sk_e2b_89215020937a4c989cde33d7bc647715
E2B_API_URL=http://localhost:3000
E2B_SANDBOX_URL=http://localhost:3002
```

These tokens are generated by the `seed-database` step and are hardcoded for local development.

# Services

| Service | URL / Connection string |
|---|---|
| Grafana | http://localhost:53000 |
| Postgres | `postgres:postgres@127.0.0.1:5432` |
| ClickHouse (HTTP) | http://localhost:8123 |
| ClickHouse (native) | `clickhouse:clickhouse@localhost:9000` |
| Redis | `localhost:6379` |
| OTel Collector (gRPC) | `localhost:4317` |
| OTel Collector (HTTP) | `localhost:4318` |
| Vector | `localhost:30006` |
| e2b API | http://localhost:3000 |
| e2b Client Proxy | http://localhost:3002 |
| e2b Orchestrator | http://localhost:5008 |

# Environment files

Each service reads its configuration from a `.env.local` file in its package directory. These are checked into the repo with working defaults for local development — you should not need to modify them.

| File | Purpose |
|---|---|
| `packages/api/.env.local` | API server config: database connections, Redis, OTEL, volume tokens |
| `packages/orchestrator/.env.local` | Orchestrator config: storage paths, firecracker/kernel dirs, Redis |
| `packages/client-proxy/.env.local` | Client proxy config: edge URL, service discovery, Redis |
| `packages/shared/scripts/.env.local` | Build scripts: API keys and URL for template builds |
| `tests/integration/.env.local` | Integration test config: API URL, test credentials |

# Troubleshooting

### `/dev/kvm` not found

The orchestrator requires KVM to run Firecracker microVMs.

- **Bare metal**: Load the module: `sudo modprobe kvm_intel` (Intel) or `sudo modprobe kvm_amd` (AMD).
- **Cloud VM**: Enable nested virtualization. On GCP: stop the VM, set `--enable-nested-virtualization`, restart. On AWS: use a `.metal` instance type.
- **WSL2**: KVM is not supported. Use a Linux VM with nested virtualization instead.

### Port conflicts

If `make local-infra` fails with "address already in use", check which service is conflicting:

```bash
sudo ss -tlnp | grep <port>
```

Common conflicts: PostgreSQL on 5432, Redis on 6379, or another Grafana on 3000 (the local Grafana uses 53000 to avoid this).

### Orchestrator fails without `sudo`

The orchestrator manages Firecracker microVMs, which require root privileges for:
- `/dev/kvm` access
- Creating TAP network interfaces
- Managing cgroups for microVM resource limits
- Mounting NBD devices

This is expected. Always run the orchestrator with `sudo`.

### `gsutil` not found when downloading artifacts

Install the Google Cloud SDK: https://cloud.google.com/sdk/docs/install

You do not need to authenticate — the public builds bucket allows anonymous access.

### ClickHouse migration fails

Ensure ClickHouse is fully started before running migrations. After `make local-infra`, wait a few seconds for ClickHouse to initialize, then retry:

```bash
make -C packages/clickhouse migrate-local
```

### Huge pages warning

If you see memory allocation errors from Firecracker, verify huge pages are allocated:

```bash
grep HugePages /proc/meminfo
```

If `HugePages_Free` is 0, the system may not have enough free memory. Stop other applications or increase RAM.
Loading