This video demystifies 'self-improving AI' by building a concrete skill-refinement pipeline: capturing real-world feedback (rejected drafts, call transcripts) as evidence, routing each finding to the right destination (skill.md, context files, memories), and gating judged change proposals behind a blast-radius-aware human review.
Mansel ScheffelWatchTranscript found
Quick learning frame
Read this before watching.
Agent ops treats agents like services: observable state, queues, permissions, logs, recovery, and post-run review.
New playlist item from Mansel Scheffel; queued for transcript-backed review, topic mapping, and a practical learning artifact.
Skill you build: The ability to design a skill-refinement loop that turns rejected outputs and call transcripts into evidence-backed, judged skill updates — deciding by blast radius which changes can auto-apply and which need a human gate.
Watch for the shift from claim to mechanism. The learning value is the point where the transcript reveals a repeatable action, tool boundary, context move, review habit, or artifact.
Concept diagram
Where this video fits.
01Project state
02Session
03Queue/Kanban
04Tools
05Logs
06Recovery
07Post-run review
Deep lesson
Turn this video into working knowledge.
4,520 cleaned transcript words reviewed across 1,252 timed caption segments.
Thesis
I Made My Claude Skills Learn Without Going Rogue teaches a practical hermes operations move: This video demystifies 'self-improving AI' by building a concrete skill-refinement pipeline: capturing real-world feedback (rejected drafts, call transcripts) as evidence, routing each finding to the right destination (skill.md, context files, memories), and gating judged change proposals behind a blast-radius-aware human review.
The goal is not to remember the video. The goal is to extract the operating principle, tie it to timestamped evidence, test how far the claim transfers, and make something reusable.
1:03
Refinement is not evals
“but I did it in the system that it was ending up in instead of just getting clawed to go and fix it. So our goal here is to stop that stupid manual pattern that a lot of...”
A skill is just a procedure the AI follows every time; evals grade it against known static examples while you build it, whereas refinement teaches it over time from real usage — usage informs the notes, notes create the rules, and rules improve the skill — instead of you fixing the same mistakes downstream by hand. For one skill you use, write down the last three times you fixed its output in the destination system instead of fixing the skill itself.
6:57
Three-layer pipeline
“anything related to that client, it takes into account that new context instead of living off of the old one. You get the point by now. We are rooting that raw information, sorting through all of it, and...”
Signal capture pulls rejected drafts, Fathom transcripts, and customer comments into an evidence inbox as evidence cards; an evidence router then decides each item's destination — behavioral fixes go to skill.md, new client facts to context files like clients/acme.md — because not everything belongs in the skill itself. Sketch your own three boxes — capture, route, refine — and list which real systems (Slack, call transcripts, rejections) would feed the intake folder.
13:53
Human gate by blast radius
“assigned DPA before any vendor touches production data. That's a hard gate. Now, again, direct evidence from that transcript. You get the point here. We are building that evidence box before we push it further through the system.”
Proposed changes land in a proposals folder with a router verdict, reasoning, an AI-judge confidence score, and a diff (reviewable like a pull request); the three M's — megaphone (audience), money, and meaning — decide when a human must approve, since one wrong auto-updated fact like your ICP cascades into every skill that depends on it. Classify each of your skills by worst-case blast radius using the three M's and mark which could safely auto-refine versus which require human review.
01
Project state
Start with this video's job: This video demystifies 'self-improving AI' by building a concrete skill-refinement pipeline: capturing real-world feedback (rejected drafts, call transcripts) as evidence, routing each finding to the right destination (skill.md, context files, memories), and gating judged change proposals behind a blast-radius-aware human review. Treat "Project state" as the outcome you are trying to make visible, not a topic label. Anchor it to 1:03, where the video says: “but I did it in the system that it was ending up in instead of just getting clawed to go and fix it. So our goal here is to stop that stupid manual pattern that a lot of...”
02
Session
Use "Session" to locate the part of the hermes operations mechanism the video is demonstrating. Ask what changes in your real setup if this claim is true. Anchor it to 6:57, where the video says: “anything related to that client, it takes into account that new context instead of living off of the old one. You get the point by now. We are rooting that raw information, sorting through all of it, and...”
03
Queue/Kanban
Turn "Queue/Kanban" into the reusable artifact for this lesson: A Hermes-style agent-ops runbook with health checks, state transitions, logs, recovery steps, and review criteria. This is where watching becomes something you can inspect and reuse.
04
Tools
Use "Tools" as the application surface. Decide whether the idea touches a browser flow, a local file, a model choice, a source document, a UI, or a review step.
05
Logs
Use "Logs" to prove the lesson. The evidence should connect back to the video title, transcript anchors, and a concrete output, not a generic best-practice claim.
06
Recovery
Use "Recovery" to carry the idea forward: save the prompt, checklist, diagram, or operating rule that would make the next agent run better.
07
Post-run review
Connect "Post-run review" to I Made My Claude Skills Learn Without Going Rogue by naming the claim, the evidence, and the artifact it should produce.
Example
Source-backed artifact packet
Convert the video into a scoped artifact request that includes the transcript claim, mechanism, acceptance criteria, and proof. The output should be a hermes-style agent-ops runbook with health checks, state transitions, logs, recovery steps, and review criteria..
Example
Hermes operations proof brief
Separate what the speaker claims, what the demo actually proves, and what still needs outside verification before you adopt the hermes operations pattern.
Example
Teach-back module
Transform the lesson into a definition, a Project state -> Session -> Queue/Kanban -> Tools -> Logs -> Recovery -> Post-run review diagram, one misconception, one practice exercise, and a check-for-understanding question.
Do not learn it wrong
Treating the title as the lesson without checking what the transcript actually says.
treating UI features as reliability
missing logs
no stop/recover path
Letting the lesson drift into feature cheerleading.
Letting the lesson drift into ops advice without logs/state.
Letting the lesson drift into assuming reliability from a demo alone.
Do not count this as learned until these are true.
01
State the transcript-backed claim in your own words: This video demystifies 'self-improving AI' by building a concrete skill-refinement pipeline: capturing real-world feedback (rejected drafts, call transcripts) as evidence, routing each finding to the right destination (skill.md, context files, memories), and gating judged change proposals behind a blast-radius-aware human review.
02
Explain the practical stakes without hype: New playlist item from Mansel Scheffel; queued for transcript-backed review, topic mapping, and a practical learning artifact.
03
Map the idea onto the Use case -> Workflow pain -> Agent role -> Adoption path -> Risk -> Metric -> Pilot sequence and name the weakest link.
04
Produce the artifact and include the evidence that proves it: A one-page AI workflow decision memo with use case, leverage claim, risks, metric, and pilot plan.
Put it into practice
Give this grounded prompt to Codex or Claude after watching.
You are helping me turn one specific YouTube video into real, durable learning.
Source video:
- Title: I Made My Claude Skills Learn Without Going Rogue
- URL: https://www.youtube.com/watch?v=QsA4ITJ0kIQ
- Topic: AI Strategy
- My current learning frame: Build a minimal refinement loop for one skill: an intake folder where rejected outputs land with explicit rejection reasons, a routing pass that proposes a diff to skill.md or a context file, and a weekly ten-minute human review of proposals before anything merges.
- Why this matters: New playlist item from Mansel Scheffel; queued for transcript-backed review, topic mapping, and a practical learning artifact.
Transcript anchors from this exact video:
- 1:03 / Evidence 1: "but I did it in the system that it was ending up in instead of just getting clawed to go and fix it. So our goal here is to stop that stupid manual pattern that a lot of..."
- 2:53 / Evidence 2: "building our skill with a very very clear definition of done. You need to come in there at least having an understanding of what you're trying to achieve. For instance, if we're trying to write LinkedIn content out..."
- 5:01 / Evidence 3: "your VS Code environment or claude code or are you just going to have a weekly review that runs as a separate skill? There are many ways to skin this catch. You can even use hooks if you..."
- 6:57 / Evidence 4: "anything related to that client, it takes into account that new context instead of living off of the old one. You get the point by now. We are rooting that raw information, sorting through all of it, and..."
- 9:19 / Evidence 5: "to screw up every single one of the workflows that rely on that. So you need to look at this from the perspective of who is affected, what value is at stake, does this change our promise or..."
- 13:53 / Evidence 6: "assigned DPA before any vendor touches production data. That's a hard gate. Now, again, direct evidence from that transcript. You get the point here. We are building that evidence box before we push it further through the system."
- 16:44 / Evidence 7: "problem with Hermes in that it creates skills for nearly everything when that's absolutely not necessary for me. I prefer a constraint-based approach, meaning I only build a skill when I have a business need or a problem..."
Video-aware target:
- Prompt lane: Hermes operations
- Mechanism to extract: Identify the operations control that makes long-running agent work visible, recoverable, or safer.
- Artifact to produce: A Hermes-style agent-ops runbook with health checks, state transitions, logs, recovery steps, and review criteria.
- Artifact must include: health check; state model; permission boundary; log source; recovery action
Your task:
1. Use the transcript anchors above as the primary source packet. If you add outside context, label it clearly as outside context and keep it secondary.
2. Create a source-check table with columns: timestamp, claim, transcript support, what the demo proves, confidence, and what still needs verification.
3. Extract the actual teachable mechanism from the video: Identify the operations control that makes long-running agent work visible, recoverable, or safer. Do not invent claims that are not supported by the title, lesson frame, or transcript anchors.
4. Build a reusable learning artifact: A Hermes-style agent-ops runbook with health checks, state transitions, logs, recovery steps, and review criteria.
5. Include:
- a plain-English definition of the core idea
- a diagram or structured model using this sequence: Project state -> Session -> Queue/Kanban -> Tools -> Logs -> Recovery -> Post-run review
- answers to these source questions: What operational failure is prevented? | What state is visible? | What can be recovered or redirected?
- 3 concrete examples that apply the video idea to real agentic work, such as Hermes Kanban triage; local model endpoint check; agent swarm recovery review
- 2 failure modes the video helps prevent, chosen from the transcript evidence and these likely risks: treating UI features as reliability; missing logs; no stop/recover path
- a checklist for the next real workflow, focused on: status, model/backend, tools, logs, recovery
- one practical exercise with a clear done signal: Write a runbook for restarting one stuck Hermes-style agent session.
6. Add a "learning transfer" section: what changes in my workflow tomorrow if I actually learned this?
7. Add a "source check" section that cites which transcript anchor supports each major takeaway.
Quality bar:
- Make this specific to "I Made My Claude Skills Learn Without Going Rogue", not a generic AI Strategy essay.
- Ground each ops recommendation in transcript evidence about state, queues, models, tools, security, logs, or recovery.
- Prefer operational examples, failure modes, and reusable artifacts over broad definitions.
- Call out uncertainty instead of smoothing over weak evidence.
- Avoid these generic drifts: feature cheerleading; ops advice without logs/state; assuming reliability from a demo alone.
- If evidence is weak or missing, stop and say what transcript segment or timestamp needs review instead of guessing.
- Finish with a concise artifact I could paste into my learning app.
Misconceptions
What to stop believing.
Every new AI tool deserves a trial.
Every tool has integration cost. Start from workflow pain, not novelty.
If an agent can do it once, it is automated.
Automation means repeatable, monitored, recoverable, and reviewable.
Practice studio
Learning only counts when you make something.
01
Transcript evidence map
Separate what the video actually says from what you already believe about the topic.
3 source-backed takeaways with timestamps, confidence, and a transfer note.02
One useful artifact
Apply the video to a real workflow and produce a hermes-style agent-ops runbook with health checks, state transitions, logs, recovery steps, and review criteria..
A reusable artifact with a done signal and one verification step.03
Hermes operations teach-back card
Explain the hermes operations mechanism to someone who has not watched the video yet.
A 90-second explanation, one diagram, one example, and one misconception to avoid.
Recall check
Answer first, then reveal — without rewatching.
How does the video distinguish evals from skill refinement?
Why doesn't all captured evidence get routed into skill.md?
What are the three M's and what decision do they support?
Source shelf
Use the video as a doorway, then verify with primary sources.
