Skip to content
  1. Home
  2. AI tools
  3. llama.cpp

llama.cpp

C/C++ LLM inference engine with CLI and OpenAI-compatible server

What it is

A C/C++ inference engine for running language models on CPUs and GPUs. It includes a command-line client and a server with an OpenAI-compatible REST API, and can pull GGUF models directly from Hugging Face. For users who want direct control over local inference or a lightweight server.

Good for

  • Run a GGUF model from the command line
  • Host an OpenAI-compatible local endpoint
  • Embed LLM inference in a C/C++ application

Install or run

curl -LsSf https://llama.app/install.sh | sh

Command as given in the project README. Read the repository before you run anything on your machine.

What you need

Models in GGUF format; hardware suited to the model size

Worth knowingLower-level than app-style runners: expect to choose quantisations and tune flags yourself.

Sources

The summary is ours, written from the README. Spotted an error? See how we work.

More for productivity & knowledge

See the topic →
  • Ollama

    ollama/ollama

    Download and run open LLMs locally behind a simple CLI and API

    ToolGitHub stars: 187k
  • Open WebUI

    open-webui/open-webui

    Self-hosted chat interface for Ollama and OpenAI-compatible APIs

    ToolGitHub stars: 159k
  • Claude-Mem

    thedotmack/claude-mem

    Records what your agent did and loads a summary into later sessions

    ToolGitHub stars: 117k
  • AnythingLLM

    Mintplex-Labs/anything-llm

    Desktop and Docker app for chatting with your documents via any LLM

    ToolGitHub stars: 68k