*Define what "good" and "bad" looks like for your AI application.*


![Maud Nalpas](https://web.dev/images/authors/maudnalpas.jpg) Maud Nalpas [GitHub](https://github.com/maudnals) [LinkedIn](https://www.linkedin.com/in/maudnalpas) [Bluesky](https://bsky.app/profile/maudnals.bsky.social)

<br />

Before designing your tests, look at a typical perfect output from
[ThemeBuilder](https://developer.chrome.com/docs/ai/evals/mental-model.md.txt#themebuilder).
Each evaluation processes a version of this object:

```json
{
  "id": "example-002",
  "userInput": {
    "companyName": "Nova news",
    "description": "Space exploration news and educational content.",
    "audience": "science enthusiasts",
    "tone": [
      "informative",
      "scientific",
      "inspiring"
    ]
  },
  "appOutput": {
    "motto": "Unveiling the universe.",
    "colorPalette": {
      "textColor": "#E2E8F0",
      "backgroundColor": "#0B0D17",
      "primary": "#7000FF",
      "secondary": "#00C2FF"
    }
  }
}
```

## Define success and failure

The first step to creating an evaluation is to define success and failure.
To do so, you must be familiar with your data; understand what faulty outputs
are likely to appear in production. If available, review production data.

Examples of faulty outputs for ThemeBuilder include:

- Incorrect data structure:
  - Invalid JSON, missing keys
  - Color palette values are not hexadecimal
  - The motto or some colors are empty strings
  - The motto is longer than our set limit of 6 words.
- Bad motto:
  - The motto doesn't match the brand, audience or tone.
  - The motto is toxic.
- Bad color palette:
  - The color palette doesn't match the brand, audience or tone.
  - The text-to-background color contrast ratio is less than 4.5.

> [!CAUTION]
> **Caution:** Even if the model provider supports structured outputs, we recommend that you implement an eval for data format. Valid JSON syntax is not enough, because data inside a valid JSON object can still be broken due to token limits, or violate our domain logic (such as, invalid hex colors).

### Example user input

```markdown
User input: {
 "companyName": "Moon Cafe",
 "description": "A cozy nocturnal coffee shop serving late-night espresso and pastries.",
 "audience": "night owls and students"
}
```

### Output: Incorrect data

```markdown
// Wrong key `tagline` instead of `motto`.
// Array of colors instead of the required `colorPalette` object.
Output: {"tagline": "Freshly brewed", "colors": \["\#f0f0f0"\]}

// The motto is over our 6-word limit
Output: {
  "motto": "The best place for late night espresso and cozy pastries",
  "colorPalette": ...
}

// Colors are invalid hexadecimal strings
Output: {
  "motto": "Brewed for the moon.",
  "colorPalette": {"textColor": "grey", "backgroundColor": "white", "primary": "neon-purple", "secondary": "\#\#00C2FF"}
}
```

### Output: Bad motto

```markdown
// Brand and tone mismatch (too cold for a cozy vibe)
Output motto: "Beans for maximum productivity."

// Toxic (rude and unwelcoming)
Output motto: "Go away loser, we're busy."
```

### Output: Bad color palette

```markdown
// Brand and tone mismatch (clashing neon colors for a cozy cafe)
Output color palette: {
  "textColor": "\#00FF00", "backgroundColor": "\#FF00FF",
  "primary": "\#FFFF00", "secondary": "\#0000FF"
}

// Contrast ratio below the 4.5:1 requirement
Output color palette: {
  "textColor": "\#CCCCCC", "backgroundColor": "\#FFFFFF",
  "primary": "\#EEEEEE", "secondary": "\#DDDDDD"
}
```

## Define evaluation criteria and methods

You can define evaluation criteria and methods, based on how an output fails
to meet your expectations:

- To test the objective criteria, create rule-based evals (use regular code).
- To test the subjective criteria, use a judge model.

| Evaluation criteria | Evaluation method |
|---|---|
| The data format is correct: Valid JSON, all keys present, hexadecimal colors, no empty values, motto is under six words | Rule-based (objective) |
| The text-to-background color contrast ratio is accessible | Rule-based |
| The motto matches the brand, audience and tone | LLM judge (subjective) |
| The color palette matches the brand, audience and tone | LLM judge |
| The motto isn't toxic | LLM judge |

### Rubric

There's no such thing as a perfect creative motto or color palette. So
instead of comparing ThemeBuilder's output to an ideal result, provide the judge clear guidelines.

> [!IMPORTANT]
> **Key Term:** A *rubric* is a structured set of criteria or scoring rules that a judge (human or AI) uses to evaluate an output. Rubrics provide a consistent framework to assess subjective qualities across every evaluation.

```text
// Example rubric for color palette brand fit 
Criteria:
1. **Psychological and literal association**: Do the colors logically map
   to the literal product and evoke the right vibe?
2. **Constraint verification**: Does the palette violate any fundamental
   keywords (such as "sustainable", "discreet", or "organic")?
3. **Appropriate and harmonious**: Is the palette suitable for the company's
   industry baseline, regardless of secondary trendy adjectives?
```

## Use task-specific criteria

Beside your use-case specific metrics, use standard criteria and metrics
relevant to the task. For example, for
[summarization](https://web.dev/articles/test-llm-capabilities),
common metrics include:

- **Alignment**: The summary follows specific user instructions, tone, or style.
- **Concision**: The summary is saying just what is needed and nothing more.
- **Richness**: The summary includes all key points.
- **Correctness**: The summary is factual and true.
- **Groundedness**: Every claim is traced back to the source to prevent hallucinations.

## Prebuilt evals

[Evals solutions and tools](https://developer.chrome.com/docs/ai/evals/sources-tools.md.txt#eval_tools) offer managed evals or prebuilt metrics that may fit your use case.
Explore what's available.