LM-Polygraph: Uncertainty Estimation for LLMs

Installation | Basic usage | Overview | Benchmark | Demo application | Documentation

LM-Polygraph provides a battery of state-of-the-art uncertainty estimation (UE) methods for LLMs in text generation tasks. High uncertainty can indicate the presence of hallucinations and knowing a score that estimates uncertainty can help to make applications of LLMs safer.

LM-Polygraph is also one of the most widely used benchmarks for the consistent evaluation of uncertainty estimation and hallucination detection methods. It is adopted by hundreds of researchers and technology companies.

Installation

From GitHub

The latest stable version is available in the main branch, it is recommended to use a virtual environment:

python -m venv env # Substitute this with your virtual environment creation command
source env/bin/activate
pip install git+https://github.com/IINemo/lm-polygraph.git

You can also use tags:

pip install git+https://github.com/IINemo/lm-polygraph.git@v0.5.0

From PyPI

The latest tagged version is also available via PyPI:

pip install lm-polygraph

Optional dependencies

Some features require additional packages that are not installed by default:

• COMET metric (translation evaluation): unbabel-comet pins numpy<2.0 which may conflict with packages like vLLM. Install via extras: shell pip install lm-polygraph[comet] If you need numpy 2.x (e.g., for vLLM), install without the extra and add comet manually: shell pip install lm-polygraph pip install unbabel-comet --no-deps

Basic usage

1. Initialize the base model (encoder-decoder or decoder-only) and tokenizer from HuggingFace or a local file, and use them to initialize the WhiteboxModel for evaluation:

from transformers import AutoModelForCausalLM, AutoTokenizer
from lm_polygraph.utils.model import WhiteboxModel

model_path = "Qwen/Qwen2.5-0.5B-Instruct"
base_model = AutoModelForCausalLM.from_pretrained(model_path, device_map="cuda:0")
tokenizer = AutoTokenizer.from_pretrained(model_path)

model = WhiteboxModel(base_model, tokenizer, model_path=model_path)

1. Specify the UE method:

from lm_polygraph.estimators import *

ue_method = MeanTokenEntropy()

1. Get predictions and their uncertainty scores:

from lm_polygraph.utils import estimate_uncertainty

input_text = "Who is George Bush?"
ue = estimate_uncertainty(model, ue_method, input_text=input_text)
print(ue)
# UncertaintyOutput(uncertainty=-6.504108926902215, input_text='Who is George Bush?', generation_text=' President of the United States', model_path='Qwen/Qwen2.5-0.5B-Instruct')

1. More examples: basic_example.ipynb
2. See also a low-level example for efficient integration into your code: low_level_example.ipynb

Using with LLMs deployed as a service

LM-Polygraph can work with any OpenAI-compatible API services:

from lm_polygraph import BlackboxModel
from lm_polygraph.estimators import Perplexity, MaximumSequenceProbability

model = BlackboxModel.from_openai(
    openai_api_key='YOUR_API_KEY',
    model_path='gpt-4o',
    supports_logprobs=True  # Enable for deployments
)

ue_method = Perplexity()  # or MeanTokenEntropy(), EigValLaplacian(), etc.
estimate_uncertainty(model, ue_method, input_text='What has a head and a tail but no body?')

UE methods such as EigValLaplacian() support fully blackbox LLMs that do not provide logits.

More examples:

• basic_example.ipynb: simple examples of scoring individual queries
• low_level_example.ipynb: low-level integration into inference and claim-level UE
• low_level_vllm_example.ipynb: low-level example using vLLM for faster inference
• basic_visual_llm_example.ipynb: examples for visual LLMs

Overview of methods