AI Ops Engineer
Project NSD
An AI-supervised deployment pipeline that ships a hardened, full-stack server — for about the price of a coffee.
- n8n · Ansible · 9 roles
- Deployed & attack-tested on a live host
- Case study v1.0 · 2026
Skills Demonstrated
Bounded Agency Design
The agent gets one allowlisted tool, one fix per pass, and a retry counter it does not own — the workflow decides when to stop.
Exit-Code-Driven Control Flow
Every role prints its exit code and the workflow reads it; “no visible error” is where verification starts, not where it ends.
Idempotent Infrastructure Automation
Nine Ansible roles built to survive a half-completed run and re-converge — the property the entire retry loop stands on.
Threat Modeling & Secret Hygiene
Five mapped attack surfaces; sanitized errors, no-log tasks, and two scoped SSH keypairs across three environments.
WAF Tuning (OWASP CRS)
Paranoia level and anomaly threshold treated as independent dials — the move from 10/13 to 12/13 on the live attack suite.
Built With
Self-hosted orchestration — the form, the control flow, the retry counters, and the human escalation gates.
Nine idempotent roles over SSH — the execution layer that makes re-running a failed role safe.
The Summary
Standing up a hardened, full-stack web server — firewall, reverse proxy, web application firewall, database, backend, frontend, TLS, and a real verification pass — is a day or more of careful senior work, and a contractor in Ethiopia can charge up to 300,000 birr for it. Project NSD compresses that into a single form submission. An n8n workflow hands each stage of the build to an Ansible control plane over SSH and reads the exit code between every role; when a role fails, a bounded Qwen agent is handed the sanitized error and one tightly scoped tool that can only edit that client's own configuration files. It proposes exactly one fix, the workflow re-runs the role and counts the attempt, and after five it stops and escalates to a human rather than build anything on a broken base. The run ends with a thirteen-attack suite fired at the live domain and an independent HTTP request made by n8n itself — not by the target — because a server asserting that it is up is not evidence. Compute cost of a full deployment: about six US dollars.
The Operational Friction
In Ethiopia, a professional contractor handling this kind of end-to-end server deployment can charge up to 300,000 Ethiopian Birr for the engagement. That is the number this project starts from, because it names a cost a small business or a solo founder absorbs quietly, one deployment at a time. In May 2026 the birr traded at roughly 161 to the dollar, which puts the engagement on the order of 1,860 US dollars — and that is not an outlier. Freelance infrastructure-automation projects are quoted at 2,000 to 5,000 dollars on Upwork, and DevOps engineers bill a median of 60 dollars an hour, rising past 150 for senior work. The Ethiopian figure isn't a local quirk. It's the worldwide price of an afternoon of scarce, senior expertise.
Now look at what that expertise is actually spending its hours on. A correct first deployment is not one task; it is a chain of interdependent ones, and each has a way of going wrong that only surfaces at the end. Locking yourself out with a bad firewall rule. A WAF that loads eight hundred rules and still blocks nothing. Postgres schema grants that silently break migrations. A React build that runs the box out of memory and dies. Certbot's edits getting overwritten on the next deploy. And, at the end of all of it, calling the thing done because there were no visible errors.
“The real cost of a manual deployment isn't the day it takes. It's the week you don't notice it went wrong.”
The friction isn't only cost and time. It's risk that lands after the invoice is paid. Human error is estimated to account for roughly 80% of IT outages, and a misconfigured or absent web application firewall is a direct path to the kind of incident whose average cost reached 4.44 million dollars globally in 2025 — and 10.22 million in the United States. The gap between “the deploy finished” and “the server is defensible” is exactly the gap where those numbers live.
It is not a small market to be standing in, either. Using AI to run and repair IT operations — what the industry calls AIOps — was valued at 12.4 billion dollars in 2024 and is projected to reach 123.1 billion by 2034. The thesis underneath Project NSD is that the capabilities driving that growth do not have to stay locked inside enterprises with enterprise budgets.
The System Map
Project NSD is a form. An operator fills in a client name, a target server, a domain, a repository or two, and a few options, and submits. What happens next is a supervised, end-to-end build of a production web server that ends with a verified, live link — or a precise, human-readable escalation explaining exactly where and why it stopped.
The workflow is deliberately parametric. Nothing about a specific client is hard-coded into the shared automation: the form submission is normalized into a single set of per-client variables — derived database credentials, a generated password, the deployment profile, file paths, domain handling, and base64-encoded environment blobs — and those variables drive everything downstream. Two different clients are two different sets of files in a per-client directory and nothing else. The shared roles are never edited to accommodate a deployment. That one design rule is what makes the system reusable rather than a one-off.
A run moves through four movements. Intake and preparation: the form is cleaned, the per-client files are written on the control plane, SSH trust is established to the target, and a pre-deployment snapshot is taken so there is something to roll back to. The build: nine Ansible roles run in dependency order, each wrapped in a self-healing supervisor. The security gate: a thirteen-attack suite runs against the live domain, and if the machine isn't blocking properly, a bounded agent tunes the firewall and the suite runs again. Verification and handoff: health checks confirm the services are active, n8n independently requests the live URL to prove it responds, and a Telegram message reports the result with the link.
The order of those movements is not arbitrary. You cannot route traffic to an application that doesn't exist yet, you cannot obtain a certificate before the domain resolves to a running server, and you cannot honestly verify security before the thing you are securing is up. The pipeline follows the real dependency chain of the work, which is why a failure early in the chain halts the rest instead of stacking more work onto a broken foundation.
What the diagram cannot show is the discipline inside the loop. Every role-running command ends by printing its exit code, and — unlike the original pipeline — the workflow now reads it. Success moves to the next role. Failure enters a bounded loop: the agent proposes one change to the client's config, a deterministic node re-runs the role and increments a counter the workflow owns, and after five attempts it escalates. The AI never decides when to stop. The workflow does. That is the structural reason the system is safe to let run unattended — the intelligence is bounded on every axis that matters: what it can touch, how many times it can try, and who it answers to when it can't.
The system takes input from a form, hands work to an AI, and holds real credentials for a live host. Each of those is a way in, so I mapped five places an attack or a leak can originate and treated each as a boundary to defend rather than a convenience to assume: form input concatenated into a shell string, prompt injection reaching the agent through the very failure logs it is asked to read, secrets printed into a debug line or into n8n's saved execution history, a private key readable by the wrong process, and an AI cheerfully reporting a success nobody verified.
The design rule underneath all five is simple to state and unusual to actually hold: nothing that arrives from outside is trusted, and nothing sensitive is ever allowed to leave. Form values are attacker-controlled until proven otherwise, so they are validated against an expected shape and written into structured per-client files — never interpolated into a command. The agent's output is a suggestion, never an instruction, so it lands in a file and a deterministic node deploys it. Secrets are referenced by name and never inlined, which had an unplanned dividend: the whole workflow and Ansible project could be sanitized for open source by swapping credential references for placeholders. There were no values to scrub, because none were ever written down.
The most important architectural decision in the system is a security decision disguised as a plumbing detail. n8n and Ansible run in different environments, and the only thing joining them is a narrow SSH hop. It would have been easier to install Ansible inside the n8n container and let the workflow shell out directly — and it would have collapsed two trust domains into one, handing the layer exposed to form input and AI output direct execution rights on the host. The boundary is inconvenient on purpose.
Decisions & Trade-offs
n8n over a bespoke orchestrator.
A custom Python or Go service could have run this pipeline. n8n won because the problem is control flow with human gates, retries, and a dozen integrations, and n8n makes that visible and modifiable without a redeploy. The trade-off is real, and it bit: a visual graph is harder to unit-test than a codebase, and expression bugs hide in places a type checker would catch. The What Failed section opens with exactly such a bug. The bet was that legibility and speed of change outweighed the loss of static guarantees for a system a small team needs to keep adapting. For this use case, that bet held.
Ansible over shell scripts.
The temptation in a deployment pipeline is to reach for SSH and a pile of shell. Ansible's idempotency is the reason not to. A shell script that fails halfway leaves the box in an unknown state; an Ansible role that fails halfway can be re-run to converge. Since the entire self-heal mechanism depends on safely re-running a failed role, shell was never actually an option. The cost is a steeper concept — roles, plays, inventory, variable precedence — but it buys the one property the system cannot live without.
Let the AI write config, but never let it execute.
The sharpest line in the design is the one between analysis and action. The agent writes a proposed configuration change into a file; a deterministic workflow node deploys it and restarts the service. This is slower to build than handing the model an open SSH tool, and it is far safer. An agent with an unrestricted shell is a liability whose blast radius is the whole server. An agent that can only write to one allowlisted directory, whose output is then applied by code that validates and can roll back, is a bounded, auditable component.
The agent is treated as the least-trusted component in the system, not the most — it is the one piece that can be steered by adversarial text, so it gets the smallest possible blast radius: one tool, one directory, no shell, a counter it can't touch, and a human at the end of the rope. Given a choice between a capable agent and a governable one, the system chooses governable every time.
Seed the WAF hardening per client, not in the shared role.
The durable fix for the firewall could have gone into the shared ModSecurity role. It went into the per-client configuration instead, seeded automatically at file-creation time. The reason is the project's central rule: the shared automation is never edited for a specific deployment. Keeping the hardening in the client layer preserves that rule, keeps the shared roles clean for an eventual open-source release, and still gives every deployment a WAF that blocks by default. The trade-off is that the same two lines live in every client folder rather than in one place. For a reusable system, that redundancy is the correct price.
What Failed
Every honest case study has this section. None of what follows is the headline kind of failure — no runaway agent, no spectacular hallucination. Each one is a gap between what a tool documents and what it does, and each cost real debugging time precisely because the error message rarely said what was actually wrong.
The WAF that inspected everything and blocked nothing. The security gate's whole purpose is to confirm the firewall blocks attacks. For a long stretch, it didn't. ModSecurity was installed, more than 800 OWASP rules were loaded, and the thirteen-attack suite sailed straight through — every attack returned 200 OK. The instinct was to assume the engine was off. It wasn't: the rule engine was on and the rules were loaded. The part most people skip is the layer below the on/off switch. The Core Rule Set runs an anomaly-scoring model governed by a paranoia level that decides which rules even execute, and the ruleset was sitting at level 1 — catching the blunt attacks, missing the evasions. The real challenge wasn't turning the WAF on. It was understanding that “on” and “blocking effectively” are two different dials.
A passing test that was lying. Two of the pipeline's branch conditions read a value from an upstream node using n8n's paired-item reference. After each role was rewired through a sub-workflow, that reference quietly stopped resolving, because a sub-workflow's output doesn't carry the item lineage it depends on. The conditions didn't error — they were rescued by a second, redundant condition that happened to be true — so the branches “passed” while silently ignoring the field they were supposed to evaluate. This is the exact hazard of visual orchestration: a type checker would have caught it; the canvas hid it behind a green checkmark. The fix was a reference that doesn't depend on lineage, and deleting the condition that was masking the bug.
Corruption caught because the process was checking. While preparing the sanitized open-source release, a base64-encoded configuration blob was transcribed with a single stray non-ASCII character mid-string. Decoded, it would have produced a broken firewall config that failed Nginx's own config test and stopped a fresh deployment cold. It surfaced because the process verifies checksums rather than trusting a copy, and because a later scan re-read the file it had written. The lesson isn't “be careful.” It's that a pipeline handling config at scale has to assume transcription and transfer will occasionally corrupt data, and build the verification in so corruption surfaces immediately instead of three deployments later.
The SSH key that was too private to work. The bridge from the n8n container to the host refused to authenticate, and every instinct said the private key's permissions were the problem — so the first move was to tighten them to owner-only, the setting SSH nags you to use. It kept failing. The counterintuitive fix was to loosen them: the key isn't owned by the user n8n's process runs as, and at owner-only that process simply can't read it. SSH reports the failure as an auth rejection rather than a permission error, which sends you debugging the wrong layer entirely. “More locked down” and “working” are not the same axis, and the right permission is the one that matches the process identity that actually has to read the file — verified, not assumed from habit.
What surprised me is duller and more useful. The slowest single step in the build is compiling ModSecurity v3 from source: fifteen to twenty-five minutes on a fresh target. It dominates the wall-clock time of a first deployment, and it is the one part a prebuilt module would meaningfully accelerate. I left it as-is because it is idempotent and correct, and correctness came first. It's the obvious next optimization, and naming it here is more useful than pretending the run is instant.
Measurable Outcomes
12/13
WAF attacks blocked, live
~$6
Compute cost per deployment
~300×
Local cost reduction
Measured on the target server against a live split-domain deployment. Human time per deployment went from one to three person-days of senior work to roughly zero — one form submission, unattended. Wall-clock went from hours-to-days-with-debugging to about twenty to forty minutes, dominated by the ModSecurity compile. Behavior on a failed step went from “the whole pipeline stops, or worse, continues on a broken base” to a bounded self-heal capped at five attempts and then a human. And proof of “live” went from the absence of visible errors to an independent HTTP request made by n8n itself.
Two of those deserve emphasis. The move from 10/13 to 12/13 on the WAF is not a cosmetic bump: 12/13 is the exact threshold at which the suite reports a production-ready firewall, and it was reached by tuning the anomaly-scoring model rather than by weakening any check — with no false positives observed on legitimate traffic across the home page, the API, static assets, and multi-parameter queries. The shift in failure behavior is the other one: it is the difference between a script you have to babysit and a system you can trust to run while you sleep.
The cost claim deserves the same scrutiny. The ~$6 is dominated by virtual-server runtime for the twenty to forty minutes a build takes; the AI portion is close to a rounding error. With the agent firing only on a failed role and capped at five attempts, a clean deployment spends essentially zero on the model, and even a run with several self-heal cycles spends cents. The AI is the cheapest part of the system. Its value isn't in tokens consumed — it's in the senior judgment call it removes from the critical path.
And the honest boundary: this system does initial deployment, and it does it well. It is not — yet — drift detection, ongoing patching, or disaster recovery. The ModSecurity compile is slow. The reusable model is validated on one platform, Ubuntu, and extends to others only through an explicit compatibility matrix, not a universal promise. The value is real and it is bounded, and saying so is the difference between a case study and a sales page.
Reflections
Supervised, self-healing, security-gated deployment is the sort of capability that has lived inside platform teams at companies with platform-team budgets. There is no longer a good reason for that. Everything in this system is either open-source, self-hostable, or priced in cents: n8n on a server you own, Ansible from Red Hat, ModSecurity and the OWASP Core Rule Set from the community, an inexpensive model doing a small, bounded job. The enterprise version of this is a six-figure platform investment and a team to run it. This version is a workflow, a control plane, and a well-drawn cage around an AI, and it fits on two modest virtual machines.
The through-line is a business operator's instinct applied to infrastructure: not “what is the most sophisticated system I can build,” but “what is the smallest, safest system that removes the most expensive human bottleneck.” The AI is here because it closes the last gap — the judgment call on a failed step — not because a system needs an AI to be interesting.
The real challenge with this workflow isn't teaching an AI to fix a broken deployment. It's building the structure around the AI that makes its help trustworthy.
The intelligence was the easy part. The governance was the engineering.
Case Study · PDF
Project NSD: An AI-Supervised Deployment Pipeline