The Butterfly Effect: How Renaming One String Broke Our Distributed System
· 4 min read
When we designed the SaaS version of Flude, we were very proud of our architecture. We split the logic into independent components: the parser core (engine), the API gateway (control-plane), the cloud workers (job-runner), and the thin clients for CI integration (flude-action and similar).
All of this lived in different repositories and was deployed independently. We thought we had achieved perfect “loose coupling”. We were certain that changes in one module would never bring down the entire system.
Until one harmless refactoring proved us wrong.
A Harmless Refactoring (The First Flap of a Wing)
It all started with perfectionism in the core repository. I noticed a technical inaccuracy: our parser generated not just classic Markdown, but a highly specific dialect with metadata (frontmatter) and shortcodes, tailored strictly for the Hugo static site generator.
Leaving the output format under the generic name markdown was incorrect—it would mislead future users if they tried to feed this code into a different generator (like Docusaurus).
I did the most logical thing in the world: I renamed the CLI argument and the internal format identifier from markdown to hugo_markdown. The core tests predictably turned green, and the commit flew into main.
Chain Reaction
In that very minute, our “loosely coupled” microservice architecture began to fold like a house of cards. It turned out that the word "markdown" was not just a local variable, but an implicit hard contract wired into all layers of the system.
Strike #1: Control Plane
The API gateway, which receives jobs from users, fell first. It suddenly started throwing HTTP 422 Unprocessable Entity errors. Why? Because at the entrance stood a strict OpenAPI schema that validated the format field. The schema allowed only two values: ["markdown", "html"]. Receiving the new parameter hugo_markdown from the client, the API simply rejected the request as invalid. We had to urgently patch the validation schemas.
Strike #2: Cloud Workers
As soon as the API was fixed and requests passed through, the workers themselves (Data Plane) crashed. A worker would successfully run the updated parser core, but upon completion, it tried to find the out/markdown directory on the disk to pack it into a ZIP archive. Naturally, the folder wasn’t there—the updated engine was now carefully putting the result in out/hugo_markdown. The workers crashed with a FileNotFoundError.
Strike #3: Thin Clients (CI Scripts)
The workers were updated. Archives started forming. But the pipelines on the users’ side continued to burn red. Our thin clients (running in GitHub, GitLab, etc.) sent a request to download the artifact, expecting to receive the markdown.zip file. The server, however, was now returning hugo_markdown.zip. The scripts simply couldn’t find the needed file and failed the build.
An Expensive Lesson
To roll out the renaming of just one variable, we had to coordinate a synchronous release across four independent repositories.
That day, we learned an important lesson. Microservices do not save you from tight coupling if your contracts are based on implicit Magic Strings smeared across the code. If one service changes a data format and another hardcodes paths to files—that’s not a distributed system. It’s a distributed monolith, which is much more painful to break than a regular one.
What’s Next?
We stabilized the contracts, covered them with end-to-end tests, and finally brought the SaaS infrastructure to a working state. The system was ready for real loads.
But one fundamental architectural problem of the core, which started it all, wouldn’t let us go—Doxygen. We tolerated its flaws: the impossibility of proper AST caching, its slowness, and the huge size of its binary in containers. But one fine day, it simply capitulated to complex C++ code, and this became the beginning of the end.
How a harmless SFINAE template trick forced Doxygen to silently delete classes from the project, and why we had to save ourselves with unsafe INPUT_FILTER scripts—read about it in the next episode.