Skill Files, Not Code: How I Built a Human-in-the-Loop Workflow for My AI Coding Agent

This was my first time vibe coding locally, and my instinct going in was: write good prompts, pick a smart model, ship. Turns out that’s the smaller half of the problem.

I like working with AI, but I want to be the one deciding where the project goes, and I still want to understand what the code is actually doing. That ruled out handing the whole thing off and letting an agent run unsupervised. What I ended up building instead is closer to a human-in-the-loop workflow defined through skill files: markdown documents that tell my AI coding agent how to plan, execute, and hand off work, with me approving at each checkpoint. Not full agentic coding, just AI doing the typing while I stay in the driver’s seat.

The tool itself is simple. A PNG to WebP converter, vanilla JS, HTML, CSS, no framework, runs entirely client-side. I built it because some of the converter I found online routes your image through a server just to compress it, which is a strange amount of trust to ask for something this basic. Mine doesn’t send anything anywhere. I use it myself.

What took longer than the tool was figuring out how to get an AI coding agent to work on it reliably without me babysitting every line. I didn’t want to sit there approving each keystroke, but I also didn’t want to let it run loose without clear direction. That’s how you end up burning tokens on work that goes nowhere, or work that goes somewhere you didn’t ask for.

The part most people skip

Most advice about working with coding agents is about the prompt: word it better, add examples, pick the smartest model. Fair enough, but it puts all the weight on one message and none on what happens after. Workflow, task decomposition, who owns what, what happens when something fails, how a change gets reviewed. That part gets treated as an afterthought, if it’s mentioned at all.

I ended up leaning on ideas that already exist in regular software engineering:

• Separation of concerns
• Single responsibility
• Small iterations
• Explicit scope
• A rollback strategy
• Human review
• A clear definition of done

None of this is new. It’s just not usually applied to “AI writes my code” the way it’s applied to a team of humans writing code.

What actually worked

• One skill doesn’t fit every task. Debugging and building a new feature are different jobs. I split mine into three: atomic-iteration for planning and orchestration, efficient-code for the actual coding rules, and semantic-git for commits and PRs. Each one is narrow on purpose.
• Small instructions beat long ones. Every extra paragraph in a system prompt is something the model has to hold onto, and the important constraints get diluted the longer the document gets. My skills and system instructions for this project are deliberately short. When I added a rule, I tried to add it in the place it actually belonged instead of restating it somewhere else.
• Treat it like project management, not code generation. This was the real shift for me. The agent is the worker. The deliverable is source code. The scope is whatever files, tools, and constraints I hand it. Once I started thinking about it that way instead of “AI, write me a feature,” the whole thing got more predictable.
• Narrow the scope aggressively. Which files can be touched, which tools are available, what “done” actually means. Every bit of ambiguity I removed was one less thing the agent had to guess at, and guessing is where things go sideways.
• Atomic iterations over full features. Instead of asking for an entire feature in one go, I broke work into small goals: one objective, one expected outcome, one review, each time. This means more oversight from me, not less, but the output got noticeably more reliable. My job shifted from writing code to checking incremental progress, which feels closer to how you’d actually manage a junior engineer than to “automating software engineering.”

Where I actually got it wrong

I thought the three-skill setup was solid until I sat down and walked through it turn by turn with my AI coding agent, checking for gaps.

Two of my skills both defined retry logic for errors during execution, with no rule for which one takes priority. On paper that reads fine. In practice it means an agent hitting an error mid-task could interpret it as one retry counter or two nested ones, and get a different number of attempts depending on which reading it landed on. The fix wasn’t adding more explanation, it was deleting the duplicate: retry logic now lives only in efficient-code, and atomic-iteration just defers to it.

There was also no rollback rule. If a task failed after the max retries, what happened to the half-finished changes sitting in the working directory? Nothing, is the honest answer. Now a failed task reverts its own file changes to the last commit before halting, and anything depending on it gets marked blocked instead of attempted.

No context budget either. Long enough iteration, and full diffs and logs from tasks that already finished just sit in context, getting reasoned over the same way as the task currently in progress. Completed tasks now compress down to one line. Only the current task and anything failed or blocked keep full detail.

And no verification step before I looked at it myself. The agent would finish a task and immediately hand it to me for manual review, syntax errors and all. Now there’s a lint/syntax check before it ever reaches me, gated through the same retry counter instead of becoming its own separate failure mode. I’m not writing unit tests for a tool this size, so this is the right amount of automated checking for what I’m building. If I add one later, the plan is for test cases to come from the same success criteria I already write during planning, not a separate spec.

None of these were things I would’ve caught reading my own skill files. They only showed up when I actually traced through what happens on a real error, step by step.

Closing thoughts

Guardrails reduce risk, they don’t remove it. A misread instruction, a conflicting rule between two skills, context the agent picked up wrong, any of that can still happen even with all of the above in place. Human review isn’t a fallback for when the system fails. It’s part of the system.

I went into this expecting the hard part to be getting the AI to write good code. It mostly does, when the code itself is the whole job. The hard part was everything around it: deciding what “one task” even means, what happens when a task fails halfway through, how much of the conversation history is worth keeping. That’s project management with extra steps, and pretending otherwise is how you end up debugging an agent instead of your app.

There’s one thing I’m still deliberately not handing over: I don’t let the agent write and grade its own tests. Its job is to satisfy whatever I asked for, so a test suite it writes for its own code tends to confirm the code did what it thinks it should do, not what it should actually do. If I add tests later, the test cases come from success criteria I write during planning, not from the agent guessing at what “correct” means after the fact.

None of this made the tool itself more interesting. It’s still a PNG to WebP tool, nothing flashy. But the workflow is the part I’d actually reuse on the next thing I build, and that’s a strange place for the real learning to end up.