Building a Hybrid Docker Orchestrator in Go: The Journey from Single VM to Multi-Node Cluster

What if you could combine the native simplicity of Docker Compose with the decentralized targeting...

title: "Building a Hybrid Docker Orchestrator in Go: The Journey from Single VM to Multi-Node Cluster" published: true tags: devops, docker, ai, antigravity canonical_url: https://dev.to/marioezquerro/building-a-hybrid-docker-orchestrator-in-go-the-journey-from-single-vm-to-multi-node-cluster-abc cover_image: https://images.unsplash.com/photo-1607799279861-4dd421887fb3?q=80&w=1000&auto=format&fit=crop

What if you could combine the native simplicity of Docker Compose with the decentralized targeting and reliability of HashiCorp Nomad?

Meet Gubernator (or gbnt), a "Goldilocks" container orchestrator written in Go. In this post, I want to share how I took Gubernator from a single-node API to a fully decentralized, multi-node VM cluster with autonomous DNS resolution and local ingress routing—all co-authored alongside Antigravity, Google DeepMind's agentic AI pair programmer.

The Vision: Why Gubernator?

Kubernetes is the undisputed king of container orchestration, but for small-to-medium projects, homelabs, or edge deployments, it represents massive operational overhead. Docker Swarm is simple but lacks fine-grained task scheduling constraints.

Gubernator is designed to fill that sweet spot:

• Single Binary Portability: The same gbnt binary acts as the Central Manager (holding the centralized SQLite state) and the Worker Agents.
• Central SQLite with Local Cache: Workers run a local cache so that containers keep running and resolving internal routes even if connectivity to the Manager is lost.
• Decentralized Ingress & DNS: The cluster leverages a distributed network of CoreDNS and Caddy instances.

The Architecture: Multi-Node Setup

To test the orchestrator realistically, we provisioned three Multipass Ubuntu VMs:

• gbnt-manager (192.168.252.8)
• gbnt-worker1 (192.168.252.9)
• gbnt-worker2 (192.168.252.10)

graph TD
    Host[Mac/Laptop Host OS] -->|Resolves *.gbnt via local resolver| CoreDNS_Manager
    subgraph Manager VM [gbnt-manager: 192.168.252.8]
        CoreDNS_Manager[gbnt-coredns]
        Mgr[gbnt-manager API & DB]
        Caddy_Mgr[gbnt-caddy]
    end
    subgraph Worker 1 VM [gbnt-worker1: 192.168.252.9]
        Agent1[gbnt Agent]
        Caddy1[gbnt-caddy]
        CoreDNS1[gbnt-coredns]
        Cont1[App Containers]
    end
    subgraph Worker 2 VM [gbnt-worker2: 192.168.252.10]
        Agent2[gbnt Agent]
        Caddy2[gbnt-caddy]
        CoreDNS2[gbnt-coredns]
        Cont2[App Containers]
    end

    Mgr -->|Orchestrates| Agent1 & Agent2
    CoreDNS_Manager -->|Synchronizes Records| CoreDNS1 & CoreDNS2
    Caddy1 -->|Routes to local| Cont1
    Caddy2 -->|Routes to local| Cont2

Decentralized Ingress & Localized Caddy Routing

One of the biggest challenges in multi-host networking is how to route web traffic to containers without overloading the Manager.

Instead of routing all external traffic through a single ingress proxy on the Manager, we built a fully decentralized routing scheme:

1. Decentralized DNS (CoreDNS): When a stack is deployed, Gubernator registers the domain (e.g., hello-app.gbnt) pointing directly to the IP of the Worker VM hosting the container.
2. Localized Ingress (Caddy): Each VM runs its own independent Caddy Ingress container.
3. Decentralized Caddyfiles: Caddy on the Manager only manages reverse-proxy rules for containers running locally on the Manager. Worker agents periodically poll the Manager for their assigned tasks and generate a local Caddyfile targeting only their local containers.

This means if hello-app.gbnt is deployed on gbnt-worker2 (192.168.252.10):

• The client's browser queries DNS, which resolves to 192.168.252.10.
• The browser connects directly to Caddy on gbnt-worker2 on port 80.
• Caddy proxies the request to the local container IP (e.g. 172.17.0.2:80).
• Zero transit traffic touches the Manager VM.

Co-authoring with Antigravity (Google DeepMind)

What makes this project unique is that 100% of the Go code, GORM integrations, Flutter dashboard widgets, and cluster setups were co-authored with Antigravity, Google DeepMind's agentic AI coding assistant.

Unlike simple autocomplete or chat windows, Antigravity acts as a pair programmer with agentic capabilities:

• Debugging Complex Network Behaviors: We encountered an issue where containers connected to multiple Docker networks (like bridge and gbnt-monitor-net) had their IPs concatenated (e.g., 172.17.0.2172.19.0.7). Antigravity traced the container IP extraction logic, proposed a fix, and validated the parser.
• Infrastructure Bootstrapping: Antigravity ran Multipass commands to spin up the VMs, transfer Go binaries, configure authorization keys, and join the workers into the cluster via JWT tokens.
• Auto-Generating the Flutter Dashboard: Antigravity designed and iterated on the Flutter Web UI, implementing features like a real-time cluster topology map, live task statuses, and a settings dialog containing system metadata and version tags.

Cluster Observability

Observability is built-in. By running gbnt monitor init, the Manager spins up a complete telemetry stack connected via a dedicated network gbnt-monitor-net:

• cAdvisor: Exposes hardware and container metrics.
• Prometheus: Scrapes metrics from all nodes.
• Grafana: Visualizes metrics.
• Loki & Promtail: Aggregates logs across all nodes.

Conclusion

Gubernator proves that you don't need a heavy orchestrator like Kubernetes to manage multi-host Docker deployments. By combining Go, SQLite, CoreDNS, and Caddy, we created a lightning-fast, decentralized orchestrator.

Pair-programming with an agentic coder like Antigravity allowed me to focus on high-level architecture while the AI handled refactoring, cross-compilation, VM deployment, and frontend updates.

If you are interested in building lightweight orchestration systems, check out the Gubernator repository and start building!

Have you built or used lightweight orchestrators? Let me know in the comments below!

#devops #docker #golang #ai #pairprogramming #antigravity