End-to-End Testing — The Full Journey

From bare metal to running production workloads, step by step.

---

What This Covers

We’ll walk through the entire lifecycle of a SwarmCracker deployment:

Bare Metal → Install → Set Up Cluster → Deploy Services → Scale → Update → Snapshot → Rollback → Monitor → Clean Up

Target Setup

┌──────────────────────────────────────────────────────────────┐
│                    A 3-Node Cluster                          │
│                                                               │
│  ┌─────────────┐  ┌──────────────┐  ┌──────────────┐        │
│  │  Manager-1   │  │   Worker-1    │  │   Worker-2    │       │
│  │  192.168.1.10│  │  192.168.1.11 │  │  192.168.1.12 │       │
│  │              │  │              │  │              │         │
│  │ SwarmKit API │  │ 3x nginx VMs │  │ 2x nginx VMs │        │
│  │ swarmctl     │  │ 1x redis VM  │  │ 2x redis VMs │        │
│  └──────────────┘  │ swarm-br0    │  │ swarm-br0    │        │
│         │ gRPC     │ VXLAN        │  │ VXLAN        │        │
│         └──────────┴──────┬───────┘────────────────┘        │
└──────────────────────────┴──────────────────────────────────┘

---

Phase 0: Get Your Environment Ready

What You Need (Per Node)

Role	vCPU	RAM	Disk
Manager	2+	2 GB	20 GB SSD
Worker	4+	8 GB	40 GB SSD

OS and Kernel

# Ubuntu 22.04 LTS (recommended) or Debian 12+
cat /etc/os-release

# You'll need kernel 5.15+ for Firecracker
uname -r

Check KVM Access

# See if KVM is there
ls -la /dev/kvm

# Verify your CPU supports virtualization
lscpu | grep Virtualization
# You should see: VT-x (Intel) or AMD-V (AMD)

# If KVM isn't loaded yet
sudo modprobe kvm_intel   # Intel
# or
sudo modprobe kvm_amd     # AMD

# Make it persistent across reboots
echo "kvm_intel" | sudo tee /etc/modules-load.d/kvm.conf

Network Setup

# Each node needs a static or reserved DHCP IP
# Make sure nodes can reach each other
ping -c 3 192.168.1.11   # from manager
ping -c 3 192.168.1.12   # from manager

# Open the ports we need
sudo ufw allow 4242/tcp   # SwarmKit gRPC
sudo ufw allow 7946/tcp   # SwarmKit control
sudo ufw allow 7946/udp   # SwarmKit gossip
sudo ufw allow 4789/udp   # VXLAN overlay

---

Phase 1: Install SwarmCracker

One-Line Install (On All Nodes)

curl -fsSL https://raw.githubusercontent.com/restuhaqza/SwarmCracker/main/install.sh | sudo bash
sudo swarmcracker setup install --download-kernel --download-rootfs
sudo swarmcracker setup network
sudo swarmcracker setup config --non-interactive

This sets up:

• swarmcracker binary → /usr/local/bin/swarmcracker
• swarmd-firecracker → /usr/local/bin/swarmd-firecracker
• swarmcracker-agent → /usr/local/bin/swarmcracker-agent
• Firecracker v1.15.1 + jailer → /usr/local/bin/ (via setup install)
• Kernel → /usr/share/firecracker/vmlinux (via setup install --download-kernel)
• Rootfs → /var/lib/firecracker/rootfs/bionic.rootfs.ext4 (via setup install --download-rootfs)
• Bridge + NAT → swarm-br0 (via setup network)
• Default config → /etc/swarmcracker/config.yaml (via setup config)
• Data directory → /var/lib/swarmkit/

Build From Source (Alternative)

git clone https://github.com/restuhaqza/SwarmCracker.git
cd SwarmCracker

# Install build tools
make install-tools

# Build everything
make all

# Install to system
sudo cp build/swarmcracker /usr/local/bin/
sudo cp build/swarmd-firecracker /usr/local/bin/
sudo cp build/swarmcracker-agent /usr/local/bin/

Make Sure It Works

swarmcracker version
# SwarmCracker v0.10.0
#   Build Time: <RFC3339 timestamp>
#   Git Commit: <short SHA>
#   Go Version: go1.26.x (linux/amd64)

swarmcracker --help

---

Phase 2: Get Your Cluster Running

Start the Manager

# On manager-1 (192.168.1.10)
sudo swarmcracker cluster init \
  --advertise-addr 192.168.1.10:4242 \
  --listen-addr 0.0.0.0:4242

You should see:

✓ SwarmKit manager initialized
✓ Control socket: /var/run/swarmkit/swarm.sock
✓ TLS certificates generated
✓ Join tokens saved to /var/lib/swarmkit/join-tokens.txt
✓ Node ID: abc123def456

Get Your Join Tokens

# Option A: Read the saved tokens
sudo cat /var/lib/swarmkit/join-tokens.txt

# Option B: Ask the CLI
sudo swarmcracker cluster token worker

Add Workers to the Cluster

# On worker-1 (192.168.1.11)
sudo swarmcracker cluster join 192.168.1.10:4242 \
  --hostname worker-1 \
  --token SWMTKN-1-<worker-token>

# On worker-2 (192.168.1.12)
sudo swarmcracker cluster join 192.168.1.10:4242 \
  --hostname worker-2 \
  --token SWMTKN-1-<worker-token>

Check That Everything’s Healthy

# List all nodes
swarmctl ls-nodes
# ID            STATUS   HOSTNAME     AVAILABILITY   ROLE
# abc123        READY    manager-1    ACTIVE         MANAGER
# def456        READY    worker-1     ACTIVE         WORKER
# ghi789        READY    worker-2     ACTIVE         WORKER

# Check cluster status
swarmcracker cluster health

✅ Phase 2 done: You’ve got a working 3-node cluster

---

Phase 3: Deploy Some Services

Launch an Nginx Web Service

swarmctl create-service nginx:latest
# Service created: svc-nginx-143022
# Image: nginx:latest

Scale It Up

# Scale to 5 replicas across workers
swarmctl scale svc-nginx-143022 5

See Your Tasks (MicroVMs)

swarmctl ls-tasks
# ID          SERVICE     STATUS    NODE        STATE
# task-001    nginx       RUNNING   worker-1    RUNNING
# task-002    nginx       RUNNING   worker-1    RUNNING
# task-003    nginx       RUNNING   worker-1    RUNNING
# task-004    nginx       RUNNING   worker-2    RUNNING
# task-005    nginx       RUNNING   worker-2    RUNNING

Each task is its own Firecracker microVM with its own kernel.

Add a Redis Backend

swarmctl create-service redis:7-alpine
swarmctl scale svc-redis-<id> 3

Inspect Your Running VMs

swarmcracker vm list
# VM ID         SERVICE     NODE        STATUS     MEMORY    VCPUS
# vm-nginx-001  nginx       worker-1    RUNNING    128MB     1
# vm-nginx-002  nginx       worker-1    RUNNING    128MB     1
# vm-nginx-003  nginx       worker-1    RUNNING    128MB     1
# vm-nginx-004  nginx       worker-2    RUNNING    128MB     1
# vm-redis-001  redis       worker-1    RUNNING    256MB     1
# vm-redis-002  redis       worker-2    RUNNING    256MB     1

Test Connectivity

# Get the VM IP from the bridge
# Each VM gets an IP on swarm-br0 (like 172.17.0.x)
# Try pinging between VMs to verify networking

✅ Phase 3 done: 8 microVMs running (5 nginx + 3 redis)

---

Phase 4: Update Services and Roll Back

Rolling Update

# Update to a new nginx version
swarmctl update svc-nginx-143022 --image nginx:1.25

# SwarmKit does a rolling update:
# 1. Start new VM with nginx:1.25
# 2. Run health checks
# 3. Shift traffic away from old VM
# 4. Stop old VM
# 5. Repeat for each replica

Update With Environment Variables

swarmctl update svc-nginx-143022 \
  --env NGINX_PORT=8080 \
  --env WORKER_PROCESSES=auto

Watch the Update

# Watch tasks during the update
swarmctl ls-tasks

# Check logs
swarmcracker vm logs <task-id>

Rollback (Using Snapshots — See Phase 5)

If something goes wrong, restore from a snapshot you made before the update.

✅ Phase 4 done: Zero-downtime rolling update works

---

Phase 5: Snapshots and Recovery

Make a Pre-Update Snapshot

# Before changing things, snapshot a VM
swarmctl snapshot create <task-id> pre-update-v1

List Your Snapshots

swarmctl snapshot list
# NAME               CREATED                   TASK_ID
# pre-update-v1      2026-04-11T13:00:00Z      <task-id>

Restore a Snapshot

# Something broke? Inspect the snapshot metadata to roll back:
swarmctl snapshot restore pre-update-v1

Delete Old Snapshots

swarmctl snapshot rm old-backup

✅ Phase 5 done: Snapshots let you undo mistakes

---

Phase 6: Manage Nodes

Drain a Worker (Maintenance Mode)

# Drain worker-1 for a kernel upgrade
swarmctl drain worker-1

# Tasks get rescheduled to worker-2
swarmctl ls-tasks
# All tasks now on worker-2

Bring the Worker Back

swarmctl activate worker-1
# Tasks rebalance back to worker-1

Promote a Worker to Manager

# Add a second manager for high availability
swarmctl promote worker-2

swarmctl ls-nodes
# worker-2 now shows ROLE: MANAGER

Demote a Manager

swarmctl demote worker-2

✅ Phase 6 done: Node lifecycle operations work

---

Phase 7: Monitor and Debug

Check Cluster Status

swarmcracker cluster health

View Service Logs

swarmcracker vm logs <task-id>

# Follow logs in real-time
swarmcracker vm logs --follow <task-id>

Inspect a Specific Task/VM

swarmctl inspect <task-id>
# Full JSON with VM config, network, resources

Check the Systemd Service

# If running as a systemd service (manager nodes use swarmcracker-manager)
sudo systemctl status swarmcracker-worker
sudo journalctl -u swarmcracker-worker -f

Debug Networking

# Check the bridge
ip link show swarm-br0

# Check VXLAN (interface is <bridge>-vxlan; default bridge is swarm-br0)
bridge fdb show dev swarm-br0-vxlan

# Check TAP devices for each VM
ip link show tap-*

✅ Phase 7 done: You can see what’s going on

---

Phase 8: Manage Storage

Create a Volume

# --size is an integer number of megabytes
swarmcracker volume create app-data --size 1024

Attach Volume to a Service

swarmctl update accepts only --image, --replicas, and --env; it has no --volume flag. Volume mounts are defined in the service/task spec.

List Volumes

swarmcracker volume ls

✅ Phase 8 done: Persistent storage works

---

Phase 9: Clean Up

Remove Services

swarmctl rm-service svc-nginx-143022
swarmctl rm-service svc-redis-<id>

Verify All VMs Stopped

swarmcracker vm list
# (empty)

Leave the Cluster (Workers)

# On worker-1 and worker-2
sudo swarmcracker cluster leave

Tear Down the Manager

# On manager-1
sudo swarmcracker cluster leave --force

Clean Up

sudo rm -rf /var/lib/swarmkit/
sudo rm -rf /var/run/swarmkit/
sudo rm -rf /etc/swarmcracker/

✅ Phase 9 done: Clean teardown verified

---

Validation Checklist

Use this to make sure everything works:

Infrastructure

• KVM available on all nodes
• Nodes can reach each other
• Required ports open (4242, 7946, 4789)

Cluster

• Manager initialized
• Workers joined
• All nodes show READY
• TLS certificates generated
• gRPC communication works

Workloads

• Service created and running
• Service scaled to multiple replicas
• Tasks spread across workers
• Each task = 1 running microVM
• Cross-VM networking works (VXLAN)

Lifecycle

• Rolling update (zero downtime)
• Environment variables injected
• Snapshot created and verified
• Snapshot restore works
• Service removal and VM cleanup

Node Operations

• Worker drain → tasks rescheduled
• Worker activate → tasks rebalanced
• Worker promoted to manager
• Manager demoted to worker

Monitoring

• Cluster status command works
• Service logs accessible
• Task inspection returns valid JSON
• Network debug commands work

Cleanup

• All services removed
• All VMs stopped
• Workers left cluster
• Manager torn down
• Data directories cleaned

---

Automate It

The production example has a deploy script that automates phases 1-3:

cd examples/production-cluster/
./deploy.sh --manager 192.168.1.10 --workers 192.168.1.11,192.168.1.12

See examples/production-cluster/README.md for details.

---

Local Development Alternative

If you want to test without multi-node hardware:

cd examples/local-dev/
./start.sh

This runs a single-node cluster (manager + worker on the same machine). See examples/local-dev/README.md.

---

Quick Troubleshooting

Problem	Check This
KVM not found	ls /dev/kvm → modprobe kvm_intel
Worker won’t join	Ping manager, check token, check port 4242
VMs not starting	journalctl -u swarmcracker-worker -f, check kernel images
No cross-node networking	Check VXLAN: bridge fdb show dev swarm-br0-vxlan
Service stuck updating	swarmctl ls-tasks, check health check config
Snapshot fails	Check disk space, VM must be RUNNING

---

This covers the complete SwarmCracker lifecycle. Each phase can be tested independently.