Can an agent pick better skills using the atlas?
Short answer
Yes, for avoiding bad picks. With the atlas, an agent's top 5 picks almost never repeat the same skill, it sees more real choices, and it no longer picks skills from known harmful collections. The picks are about as relevant as without the atlas. The atlas filters out bad options; it doesn't yet find better ones.
What we found
1. Far fewer duplicates
out of 100 picks were copies of a skill already picked, with plain file search vs with the atlas.
2. More real choices
different skills in the top 5 (plain search that also skips exact copies: ).
3. No picks from harmful collections
out of picks came from known harmful collections or were high risk, plain search vs atlas. Before this week's fixes the atlas still picked .
4. Same relevance
of picks fit the task in a blind check. Picks that fit well and are reusable: → , too small a gap to be sure.
The atlas's own search (the API agents call) used to return nothing for
Duplicate picks, out of 100
Picks from harmful collections or high risk, out of 300
Both counts use today's labels, so the change shows what the atlas now avoids, not a change of yardstick. Plain search has no atlas, so it doesn't change.
What the atlas doesn't do yet
- It can't tell if a skill only works in one project. That was the most common problem in the blind check. Skills marked use were judged reusable
of the time, about the same as skills marked ignore ( , mostly good skills that are copies). - It doesn't rank by how well a skill fits the task. It removes copies, slop and danger from whatever the search found.
- Its risk rules are rough outside known harmful collections. In a hand check of 30 newly high-risk skills outside those collections, only 3 really sent secrets or keys somewhere unvetted. Most were security-testing notes or skills with a hard-coded server address.
- Security-tool collections are not flagged as harmful. For example AeonDave/malskill is an offensive-security toolkit, not a set of booby-trapped skills, so it keeps normal verdicts.
Honest caveats
- One judge, not people. The blind checks were done by the AI assistant that ran the eval, without seeing which method picked what.
judgments over tasks. The judgments added after the rerun were blind to method, but the judge knew they came from the rerun. - Small test.
tasks, 5 picks each. The ranges in "How we measured this" show how sure we are. - Some numbers are circular. The atlas scores 0 "bad" picks by its own labels because it filters on those labels. We don't count that as a win; findings 1, 2 and 4 don't use atlas labels.
- Finding 3 is a safety check we added after spotting the problem. It's a fair before/after, but it wasn't planned at the start.