Automation & Agents

Have You Actually Read the Whole AI Skill Before Installing It? A 10-Minute Pre-Install Review and Post-Install Smoke Test

A high star count and a polished README don't mean what's inside is clean. This note gives you a ten-minute routine to run before and after every install, plus a real case: the review passed, and the post-install test still caught a path mismatch the README never mentioned.

09/29/2026 ｜ 3 min read

One Core Idea

An AI skill or plugin is not a static tool. It’s an instruction set you authorize to run. With traditional software you can at least see what it does (UI, logs). An AI skill is often a block of text that the AI interprets and acts on once loaded: it can read and write files, call outside services, and even hide behavior you won’t notice. Judge it by “how much access would I give a stranger on my system”, not by “how good is the pitch”.

Step by Step (about 10 minutes)

Step 1 | Before installing, list every file that will be loaded

Most skills are several files: a main instruction file (usually SKILL.md), reference docs, sometimes yaml/json config. List the full set. Don’t stop at the main file; the sneaky content tends to sit in “reference docs” that look secondary.

Ask your AI:

I'm about to install [skill name / path]. List every file under its folder, including subfolders.

Step 2 | Read each file in full, not just the headings

Ask three questions while reading:

1. Does it connect to outside URLs, download executables, or send data somewhere?
2. Does it ask for passwords, API keys or tokens and pass them along?
3. Is there text that looks like an instruction but has nothing to do with what the skill is for (say, a sudden system command, or an edit to an unrelated config file)?

If any answer is “yes, and I’m not sure why”, don’t install it yet. Find out, or walk away.

Ask your AI:

Review this [paste SKILL.md contents] for suspicious external links, credential-harvesting behavior, or instructions unrelated to its stated function. List the evidence for each judgment.

Step 3 | After installing, run one test you can verify

• Confirm the system actually loaded and recognized it (most tools show some “available” signal).
• If it has executable behavior (not just text instructions), run the smallest real case and check the output is what you expected.
• If it writes files, try it first in a throwaway environment, not against real data.

Step 4 | Write down what you installed, where, and the verdict

One line is enough: which skill, when, whether the review passed, any known limits. Six months from now you won’t remember what you installed or why, and the more tools you add the truer that gets.

A Real Case: Review Passed, Test Still Found a Gap

I took apart a 16k-star GitHub repo that ships a few Claude Code skills. Before installing I read every file and confirmed they were pure prompt and reference files, with no outside links and no hidden instructions. Only then did I install.

The post-install test turned up a structural mismatch: the skill’s default read/write paths didn’t match the conventions of my existing setup. Nothing in the README mentioned it. Only running it caught it.

Traps You Will Hit

1. Reading only the README. The README is marketing for humans; what actually gets loaded and executed is other files. Reading only the README is reading the box, not the contents.
2. Treating “installed” as “tested”. “Installed” only proves the format loads. It doesn’t prove the behavior is right or that there are no side effects you didn’t expect. The first real use is itself the last test, so don’t assume it works the way you think.
3. Batch-installing and rubber-stamping. Once you install five or more at a time, it’s easy to skim each one and wave it through. Make the list first, review one by one, and decide afterward; more items is not a reason to lower the bar for each.

One-Line Summary

An AI skill is execution permission you hand over, not a decorative feature list. Read it in full before, test it after, and ten minutes buys you the floor of knowing what you installed.