# Evals

Concept · [Building with AI](https://ecosystem.samcarlton.com/ai)

Also called: eval, evaluation, evaluations, model evaluation, AI evaluation, test set

Tests that check whether the outputs of a model or an AI product meet the criteria its builders set. [Teams](https://ecosystem.samcarlton.com/microsoft-teams) run them to measure quality and to compare one version with the next.

**For example:** A team writes 50 sample customer questions with correct answers. After each change to the [prompt](https://ecosystem.samcarlton.com/prompt), they run all 50 and count how many answers pass.

## Connects to (19)

- [Benchmark](https://ecosystem.samcarlton.com/benchmark): A shared, public test used to score different models on the same tasks so they can be compared.
- [Testing](https://ecosystem.samcarlton.com/testing): Checking that software does what it should, before your users find the problems for you.
- [Hallucination](https://ecosystem.samcarlton.com/hallucination): When an AI model states something false or invented as though it were true.
- [Prompt](https://ecosystem.samcarlton.com/prompt): The instructions or question you give an AI tool in everyday words.
- [Ground truth](https://ecosystem.samcarlton.com/ground-truth): The verified, correct answers or facts that an AI system's output is checked against.
- [MLOps](https://ecosystem.samcarlton.com/mlops): The practices and tools for running machine learning models in production: versioning data and models, deploying them, monitoring quality…
- [AI safety](https://ecosystem.samcarlton.com/ai-safety): The work of making AI systems behave safely and of measuring how safe they are.
- [Sycophancy](https://ecosystem.samcarlton.com/sycophancy): The habit AI models have of telling you what you want to hear: agreeing too readily, praising weak ideas or giving up a correct answer…
- [Probabilistic vs deterministic](https://ecosystem.samcarlton.com/probabilistic-vs-deterministic): Deterministic software gives the same output for the same input every time.
- [Prompt optimization](https://ecosystem.samcarlton.com/prompt-optimization): Improving a prompt through repeated testing: change it, run it against a set of examples, measure results and keep the better version.
- [Overfitting](https://ecosystem.samcarlton.com/overfitting): When a model learns its training examples too closely, including their quirks, and then performs poorly on new data.
- [Vertex AI](https://ecosystem.samcarlton.com/vertex-ai): Google Cloud's platform for building, running and evaluating AI models and agents in a business setting, with access to Gemini and other…
- [Gen AI Evaluation Service](https://ecosystem.samcarlton.com/gen-ai-evaluation-service): A part of Google Cloud's Gemini Enterprise Agent Platform, formerly Vertex AI, that scores the outputs of AI models against criteria a…
- [Gemini Prompt Optimizer](https://ecosystem.samcarlton.com/gemini-prompt-optimizer): A Google Cloud tool that rewrites a prompt's instructions and tests the versions against examples to find one that scores better.
- [Artificial Analysis](https://ecosystem.samcarlton.com/artificial-analysis): An independent company that tests and compares AI models and providers on quality, speed and price, and publishes the results.
- [LMArena](https://ecosystem.samcarlton.com/lmarena): A public website, now called Arena, where people compare answers from two anonymous AI models and vote for the better one.
- [promptfoo](https://ecosystem.samcarlton.com/promptfoo): An open-source tool for testing prompts and AI apps.
- [Braintrust](https://ecosystem.samcarlton.com/braintrust): A platform for evaluating and monitoring AI products.
- [NIST AI Risk Management Framework](https://ecosystem.samcarlton.com/nist-ai-rmf): A voluntary framework from NIST, the US standards agency, that helps organizations manage the risks AI can pose to people, organizations…

Map view: https://ecosystem.samcarlton.com/evals