Agent Monitoring

Arize Phoenix

Found in: Yigtwxx/Awesome-RAG-Production

LangSmith

LangSmith is an all-in-one platform to develop, debug, test, and monitor LLM applications, ensuring high performance, accuracy, and reliability.

Langfuse

An open-source LLM engineering platform for tracing, evaluation, prompt management, and metrics. #opensource Found in: steven2358/awesome-generative-ai

Helicone

Self-host / Cloud Found in: Yigtwxx/Awesome-RAG-Production

Weights & Biases Weave

Weave by Weights & Biases is an LLM observability toolkit that provides tracing, evaluation, and dataset management for AI applications. It integrates with the broader W&B experiment tracking ecosystem.

Openllmetry

Open-source observability for your LLM application, based on OpenTelemetry !GitHub Repo stars Found in: kyrolabs/awesome-langchain

Datadog LLM Observability

Datadog LLM Observability provides end-to-end monitoring for LLM applications within the Datadog platform. It offers trace visualization, prompt/response inspection, cost tracking, and quality evaluations alongside your existing APM data.

LangWatch

LangWatch is an LLMops platform designed to monitor, evaluate, and optimize large language model (LLM) applications throughout their lifecycle 1256. It caters to domain experts, developers, and business stakeholders, offering tailored features 6. LangWatch helps organizations build and deploy high-quality, reliable LLM applications by providing tools for real-time performance monitoring, quality assessment using pre-built and custom evaluations (offering over 30 off-the-shelf evaluations), performance optimization through prompt engineering and model selection (integrating with DSPy for automated prompt optimization), debugging with tracing and observability, and risk mitigation against jailbreaking, data leakage, and hallucinations 1251112. Key features include dataset management, customizable dashboards, version control, integration with various LLMs (OpenAI, Claude, Azure, Gemini, Hugging Face, Groq, LangChain, DSPy, Vercel AI SDK, LiteLLM, OpenTelemetry, and LangFlow), API access via a REST API, and a focus on security and compliance (GDPR compliant, working towards ISO27001, with self-hosted or hybrid deployment options) 12345. LangWatch has use cases in AI chatbots (monitoring performance, detecting off-topic conversations, and preventing data leaks), RAG applications (evaluating quality), AI-powered tools (improving accuracy and reliability), and generative AI (ensuring quality and safety) 8. Unique selling points include its comprehensive LLMops platform, ease of use, flexibility in supporting various LLMs and frameworks, collaborative workflows, and potential cost-effectiveness through prompt optimization 1259. The platform supports Python and TypeScript, integrates with OpenTelemetry, and offers SDKs for both languages 46. Integration is facilitated through REST APIs 3. While specific awards are not mentioned, positive user testimonials and company growth suggest market acceptance 5. Recent updates include the addition of helm charts, UI improvements, ongoing development of the REST API and SDKs, and recent funding secured by the company 3711.

Honeycomb AI

Honeycomb extends its observability platform to LLM applications with OpenTelemetry-based tracing, natural language querying, and AI-powered root cause analysis for understanding complex AI system behavior.

Humanloop

Humanloop is a cutting-edge platform designed to optimize the development and deployment of AI models, with a particular focus on Large Language Models (LLMs). By allowing seamless collaboration between product managers, engineers, and domain experts, Humanloop ensures that AI projects are executed efficiently and effectively. The platform provides best-in-class tools for prompt management, evaluation, and fine-tuning, making it easier for teams to develop differentiated AI products. Whether you're a startup looking to prototype AI applications rapidly or an enterprise aiming for large-scale deployment, Humanloop offers robust solutions tailored to your needs. One of the standout features of Humanloop is its commitment to data privacy and security. With Humanloop, you can safely activate LLMs using your private data, all while retaining full ownership of your data and models. This makes it an ideal choice for industries with strict data governance requirements, such as healthcare, finance, and legal services. Additionally, the platform's ability to manage test data, define custom metrics, and integrate evaluations into CI/CD workflows ensures that your AI models are not only high-performing but also reliable and secure. Humanloop also boasts a range of success stories from leading AI teams. Companies like Duolingo, Filevine, and Athena have leveraged Humanloop to accelerate their AI development timelines, achieve significant cost reductions, and enhance product performance. Whether it's tripling AI product velocity for customer service platforms or doubling annual revenue for legal case management systems, Humanloop has proven its efficacy in real-world applications. Join the ranks of satisfied customers who have transformed their AI capabilities with Humanloop's unparalleled platform.

Galileo AI

Galileo provides AI evaluation tools that detect hallucinations, measure response quality, and monitor LLM application performance. It offers real-time guardrails and automated quality metrics for RAG and agent systems.

AgentOps

Discover AgentOps the ultimate tool for testing, debugging, and optimizing AI agents. Track, analyze, and enhance agent performance seamlessly.