Fact Checker
Freefact-checking LLM outputs with langchain 
About Fact Checker
Fact Checker is an open-source Python tool that demonstrates fact-checking large language model (LLM) outputs using prompt chaining and a self-ask technique. The process works by first asking an LLM a question, then having the LLM generate an initial answer. Next, the LLM self-interrogates to identify the assumptions underlying that answer. Each assumption is sequentially verified for truthfulness using additional LLM queries. Finally, a new answer is generated that incorporates the verified information or acknowledges inconsistencies. The project includes a command-line interface (fact_checker.py) and a Jupyter notebook (fact_checker.ipynb) for interactive use. A provided example shows the method uncovering that while elephants are mammals, they do not lay eggs, leading to a corrected answer. The repository is a proof-of-concept by Jasper Gilley and is intended to showcase the potential of structured verification to improve LLM reliability.
Key Features
Pros & Cons
- Open-source and free to use
- Simple, clear implementation that is easy to understand and modify
- Demonstrates a novel approach to improving LLM accuracy through self-verification
- Step-by-step output provides transparency into the reasoning process
- Works with any LLM accessible via langchain (not model-specific)
- Proof-of-concept; not production-ready for complex fact-checking tasks
- Sequential verification of assumptions can be slow and API-costly
- Relies on the underlying LLM's ability to generate accurate assumptions and verifications
- No built-in handling for contradictory or ambiguous verification results
- Limited to text-based questions; no support for multimodal inputs
- Requires manual installation and configuration of dependencies (Python, langchain, LLM API keys)