Skip to content
  1. Home
  2. AI tools
  3. DeepEval

DeepEval

Pytest-style evaluation framework for LLM apps and agents

What it is

A Python framework for unit-testing LLM systems in a Pytest-like style. It provides metrics such as G-Eval, task completion, answer relevancy and hallucination, mostly using an LLM as judge, and integrates with common agent frameworks. For developers who want eval tests alongside their normal test suite.

Good for

  • Write unit tests for an LLM feature
  • Score a RAG pipeline on relevancy
  • Gate a release on agent task completion

Install or run

pip install -U deepeval

Command as given in the project README. Read the repository before you run anything on your machine.

What you need

Python 3.9+; an LLM API key for judge-based metrics

Worth knowingJudge-based metrics cost tokens and vary between runs; the login step that sends results to the vendor's cloud is optional.

Sources

The summary is ours, written from the README. Spotted an error? See how we work.

More for security & quality

See the topic →
  • Security Guidance plugin

    NousResearch/hermes-agent

    Warns when written code contains known-dangerous patterns

    ToolGitHub stars: 263k
  • security-guidance

    anthropics/claude-plugins-official

    Security warnings on edits plus LLM diff review

    ToolGitHub stars: 47k
  • Langfuse

    langfuse/langfuse

    Open-source tracing, evals and cost tracking for LLM applications

    ToolGitHub stars: 37k
  • Promptfoo

    promptfoo/promptfoo

    CLI for evaluating and red-teaming prompts, agents and RAG apps

    ToolGitHub stars: 27k