Escaping the Cloud: How Five Repositories Stopped Sharing One Minute Limit
· 3 min read
Splitting the monorepo granted each of our five components an independent continuous integration pipeline. Architecturally, we achieved perfect component isolation. The problem hid in the strict limits of free GitHub Actions minutes. A unified build line used to efficiently compile everything in a single run. Five separate pipelines began burning through our starting two thousand minutes like wildfire by launching independent jobs for every tiny commit.
One Physical Machine for Five Clients
We flatly refused to buy extra cloud minutes and deployed our own self-hosted runner on a standard Windows office laptop. Inside ran an Ubuntu virtual machine powered by VirtualBox. This single virtual machine methodically took turns servicing tasks for all five repositories.
Official GitHub documentation heavily advises against attaching local runners to public projects due to remote code execution risks via malicious pull requests. Our repositories are completely private, keeping us safe from outside attacks. We nevertheless implemented paranoid hygiene to guarantee build reproducibility. A single runner processes code from five different projects, and leftover file debris from a previous build could easily ruin the next one.
We isolated the guest system through classic NAT (nic1=nat), blocking its access to the host’s local network. Before every new job, a script ruthlessly rolled the virtual machine back to a clean snapshot. The runner itself launched with the --ephemeral flag for one-time registration, while an additional Unregister-OrphanedRunner.ps1 script scrubbed hanging records via the API.
The Price of Genuine Isolation with a Custom Supervisor
Many wonder why we wrote a custom polling script when GitHub offers built-in Organization Runners. The standard mechanism certainly allows sharing runner pools across multiple repositories. It completely fails at managing the lifecycle of the virtual environments themselves.
The --ephemeral flag successfully removes a runner from the pool after task completion. It does nothing to restore the filesystem to its original pristine state. To wrap every single task in a strict cycle of snapshot rollbacks and VM launches, we still needed an external controller. The custom REST API polling script became a mandatory tax for ensuring a clean environment for every build.
The Watchdog Monitoring Itself
The single server became a critical point of failure for the entire project. Any freeze in the supervisor instantly paralyzed code checks and documentation publishing. For the first few weeks the script ran unsupervised and regularly demanded manual restarts whenever VirtualBox spawned zombie processes. We wrote a separate watchdog script tasked with automatically restarting the frozen controller.
The logic was primitive: if the supervisor.log file stopped updating and the VBoxHeadless process wasn’t running, the watchdog pulled the plug. We shot ourselves in the foot on day one. The watchdog diligently wrote its own diagnostic logs into the very same supervisor.log file. It constantly updated the file modification time, reliably blocking its own check condition.
We quickly separated the logs into different files, and the system appeared to run smoothly. The following morning the watchdog lost its mind and began systematically killing a perfectly healthy supervisor for no apparent reason. This sent us down a two-day detective investigation into a network ghost in the machine, which we will cover in the next part.