Marketing skill evaluation is how growth teams test reusable agent skills (brief expansion, outbound research, ad variant generation) before those skills touch production workflows. You use golden sets, rubrics, and regression checks at the skill layer. That work sits apart from end-to-end workflow QA. Skills version like code. Without eval, quality drifts silently while dashboards still look green.
Regression suites catch silent drift when prompts or models change. The same discipline product teams use for API contracts belongs on marketing skills: test the unit before you trust the pipeline. OpenAI's evals guide documents the golden-set pattern eval teams use for LLM outputs.
TL;DR
- Skill eval tests the reusable unit; workflow QA tests the full path.
- Golden sets: fixed inputs with expected quality bars, not one perfect output.
- Rubrics score dimensions (accuracy, voice, completeness) with pass/fail thresholds.
- Re-run eval on every skill version bump, ideally inside CI before release.
- Connect to AI workflow evaluation for production monitoring after skills ship.
Skill Evaluation Is Not Workflow QA
Conflating the two creates blind spots. Workflow QA asks whether a lead reached the CRM on time. Skill eval asks whether the enrichment skill still returns correct firmographics after you changed the prompt last Tuesday.
| Layer | Question | Example artifact |
|---|---|---|
| Skill eval | Does this skill still perform? | Golden set + rubric |
| Workflow QA | Does the pipeline complete? | Run log, SLA |
| Live monitoring | Is production drifting? | Sampled human review |
A broken skill poisons every workflow that calls it. One bad brief-expansion skill can degrade dozens of content flows before anyone notices in a funnel report. Marketing agent skills should ship with eval hooks the same way engineering ships unit tests.
Workflow evaluation covers orchestration, retries, and business outcomes. Marketing skill evaluation covers the atomic unit: input in, output scored, pass or fail recorded. You need both layers. Skipping skill eval is like merging prompt changes without running unit tests and hoping integration tests catch everything.
Skill Tier Taxonomy
Not every skill carries the same risk. A tier taxonomy keeps marketing skill evaluation proportional: lightweight checks for internal drafts, strict gates for customer-facing and regulated output.
| Tier | Examples | Eval bar | Release rule |
|---|---|---|---|
| T0: Internal draft | Outline stubs, internal research summaries | ≥4/5 rubric dimensions pass | Auto-promote on green CI |
| T1: Team-facing | Sales enablement snippets, internal campaign briefs | All dimensions pass | PMM spot-check optional |
| T2: Customer-facing | Blog drafts, landing copy, outbound sequences | All dimensions pass + human sign-off | Block release on any fail |
| T3: Regulated | Claims-heavy ads, financial services copy, health statements | All pass + legal/compliance flag | Mandatory review queue |
Assign a tier when you define the skill, not when something breaks in production. Tier drives golden set size, rubric strictness, and who must approve a promote. When a skill moves up a tier, re-baseline the golden set. The bar changed; the old pass rate no longer means safe to ship.
Building a Golden Set for Marketing Skills
A golden set is a curated list of inputs with reference expectations. You document quality bars, not necessarily one canonical output. Two strong brief expansions can both pass if they hit the rubric.
| Component | Purpose |
|---|---|
| Core cases | Happy path for the skill's job |
| Edge cases | Missing fields, ambiguous briefs, conflicting instructions |
| Adversarial cases | Prompt injection, off-brand requests, stale data |
| Regression cases | Bugs you fixed once and never want back |
Example for a content brief expansion skill:
| Case ID | Input snippet | Pass criteria |
|---|---|---|
| G01 | Keyword: "agentic outbound" | H2s match intent, no duplicate TL;DR |
| G02 | Empty competitor field | Skill asks for input, does not invent URLs |
| G03 | Brand prohibition: "guaranteed ROI" | Output contains no banned phrase |
Store golden sets in repo alongside the skill definition. When product marketing adds a new banned claim, add a golden case the same week. OpenAI's evals guide applies the same shape even if your runtime differs.
Rubric Dimensions and Pass/Fail Thresholds
Rubrics turn subjective "this feels off" into repeatable scores. Each dimension gets a 1 to 5 score and a documented fail mode.
| Dimension | What to score | Fail example |
|---|---|---|
| Factual accuracy | Claims match sources | Invented stat |
| Brand voice | Matches voice entity | Wrong reading level |
| Completeness | Required sections present | Missing FAQ |
| Safety | Prohibitions respected | Regulated claim |
| Tool discipline | Correct API usage | Wrong CRM field |
Set thresholds per skill tier (see taxonomy above). Marketing skill evaluation should gate every customer-facing skill release, not just major model swaps. Track pass rate per dimension over time so drift is visible.
Scorecard Example: Content Brief Expansion Skill
The table below is a filled scorecard for one eval run against skill `brief-expand/v2.3`, tier T2. Each row is one golden case. "Pass" means every scored dimension met the tier threshold.
| Case ID | Factual | Voice | Complete | Safety | Tools | Result | Notes |
|---|---|---|---|---|---|---|---|
| G01 | 5 | 5 | 5 | 5 | 5 | Pass | Clean |
| G02 | 5 | 4 | 5 | 5 | 5 | Pass | Voice slightly casual |
| G03 | 5 | 5 | 5 | 5 | 5 | Pass | Banned phrase absent |
| G04 | 2 | 4 | 5 | 3 | 5 | Fail | Invented competitor URL |
| G05 | 5 | 5 | 4 | 5 | 5 | Pass | Missing FAQ acceptable for tier |
| G06 | 5 | 5 | 5 | 5 | 4 | Pass | Wrong CRM field unused |
| G07 | 5 | 5 | 5 | 5 | 5 | Pass | Regression case held |
| G08 | 5 | 5 | 5 | 5 | 5 | Pass | Adversarial prompt rejected |
Run summary: 7/8 cases pass. Release verdict: Block promote. G04 is a regression case tied to a prior production bug. Fix retrieval grounding, re-run, attach new scorecard to the release tag.
Walkthrough: One Eval Run From Trigger to Verdict
Here is a concrete loop for the brief-expansion skill above. The same steps apply to enrichment, ad variants, or research skills.
1. Trigger. A developer opens a pull request that bumps `brief-expand` from v2.2 to v2.3. The PR changes the system prompt and swaps a retrieval chunk source.
2. Load context. CI checks out the skill definition, golden set `golden/brief-expand-v2.yaml`, and rubric `rubrics/brief-expand.json`. It pins model version and temperature to match production.
3. Execute cases. The runner invokes the skill once per golden case. No workflow wrapper, no downstream CRM write. Isolation matters: you are testing the skill, not the orchestration.
4. Score. An LLM judge (or hybrid judge plus rule checks) scores each dimension. Rule checks catch hard fails fast: banned phrases, missing headings, hallucinated URLs.
5. Aggregate and verdict. The runner builds the scorecard, diffs against v2.2, and fails CI when G04 misses factual accuracy.
6. Artifact. On green, CI uploads the scorecard hash to release notes for traceability.
That loop is marketing skill evaluation in practice: repeatable, owned, and tied to version control. It is not a one-off prompt review in a shared doc.
CI and Release Integration
Treat skill eval like a test suite in your release train. Strong marketing skill evaluation programs wire eval into CI before any skill reaches staging.
| Stage | What runs | Gate |
|---|---|---|
| Pull request | Full golden set for touched skills | Merge blocked on any fail |
| Pre-release | Full set + shadow sample on staging model | Release blocked on regression |
| Post-release | Weekly production sample | Alert if dimension drift >5 pts |
Wire eval into the same pipeline that versions marketing agent skills. Skill manifest, golden set, rubric, and eval runner should live in one repo path so reviewers see the full change surface.
Use path filters so CI runs only when a skill or golden set changes. Fail if any regression case drops versus `main`. Store scorecards for audit trails and embed eval hashes in release tags.
Pair automated gates with human-in-the-loop marketing for T2 and T3 tiers. CI catches systematic drift; humans catch context the golden set missed. Eval green does not mean ship without review for customer-facing copy. It means the skill met documented bars before review starts.
Regression Checks When Skills Change
Every skill version bump triggers:
- Run full golden set.
- Compare scores to prior version (diff report).
- Block promote if any regression case fails.
- Tag release with eval artifact hash.
| Change type | Eval scope |
|---|---|
| Prompt tweak | Full golden set |
| Model swap | Full set + live shadow sample |
| New retrieval source | Adversarial + factual cases |
| Tool schema change | Tool discipline cases only, then full set |
Model upgrades are the silent killer. A new model can pass every workflow SLA while failing brand voice on half your golden cases. Marketing skill evaluation catches that before the model ID reaches production config.
When you fix a bug found in production, add a regression case the same day. The case ID stays in the scorecard permanently. That is how you prevent the same failure twice.
What the SERP misses
Most content discusses "AI QA" as one blob. This page separates skill-level testing and gives operators a concrete loop:
- Golden set structure for marketing skills, not generic ML benchmarks.
- Rubric dimensions with marketing-specific fail modes.
- Skill tier taxonomy so eval effort matches risk.
- CI integration so eval is a release gate, not a spreadsheet ritual.
- Regression discipline tied to skill versioning.
Marketing skill eval loop
| Step | Owner | Output |
|---|---|---|
| Build skill | GTM engineer | Skill spec + v1 golden set |
| Classify tier | PMM or ops | Tier label in skill manifest |
| Test | QA or PMM | Rubric scorecard |
| Gate in CI | Engineering | Pass/fail on pull request |
| Regress | On every change | Diff vs prior version |
| Version | Release tag | Eval hash in changelog |
| Monitor | Ops | Weekly sample in production |
Frequently Asked Questions
How do you test marketing agent skills?
Define golden inputs, score outputs against a rubric, and block promotion when regression cases fail. Run marketing skill evaluation before workflow integration and again after every skill version tag. Run the suite in CI on every skill change.
What is a golden set for AI skills?
A fixed library of inputs with documented pass criteria representing normal, edge, and adversarial cases for that skill's job. It versions with the skill and expands when you find new failure modes.
Skill evaluation vs workflow evaluation?
Skill eval tests one reusable unit (brief expansion, enrichment). Workflow evaluation tests whether the full pipeline meets SLAs and business outcomes. Both are required; neither substitutes for the other.
How often should you re-test skills?
On every skill version change, model swap, or retrieval source update. Run periodic production sampling weekly for customer-facing skills. Tier T3 skills may need daily sampled review in high-volume channels.
What causes marketing skills to drift?
Prompt edits, model upgrades, stale retrieval corpora, and new product claims without updated golden cases. Version skills and eval together. Drift is normal; undetected drift is the failure mode marketing skill evaluation exists to prevent.
Sources
- OpenAI Evals Guide: Eval set methodology for LLM outputs.
- Anthropic: Building effective agents: Composable skills and reliability patterns.
- NIST AI Risk Management Framework: Governance framing for AI testing.
- Martin Fowler: Test pyramid: Layering unit (skill) vs integration (workflow) tests.
Takeaway: Version Skills With Eval, Not Hope
Marketing skill evaluation is how teams keep agent libraries trustworthy: tiered golden sets, rubric scorecards, CI gates on every change, and clear separation from workflow QA.
Treat skills like code. Marketing skill evaluation is the diff you show when someone asks what changed last week. Build the loop once, wire it into release, and let pass/fail thresholds do the arguing before a customer sees the output.
