Agenta

Agenta is an open-source platform for building, evaluating, and deploying LLM applications with a focus on collaborative prompt engineering and systematic evaluation. The platform provides a web-based interface where teams can experiment with prompts, compare model outputs, run evaluations, and deploy optimized configurations — bringing structured development practices to the often ad-hoc process of building LLM applications.

The platform's playground feature provides a side-by-side comparison interface for testing prompts across different models, parameters, and configurations. Teams can iterate on prompts visually, compare outputs in real-time, and save successful configurations as versioned variants. This collaborative approach replaces the typical workflow of testing prompts in notebooks or chat interfaces without systematic tracking.

Agenta's evaluation framework supports both automated and human evaluation workflows. Pre-built evaluators cover common quality dimensions like similarity, accuracy, and relevance. Custom evaluators can be defined using Python functions or LLM-as-judge patterns. Evaluation results are tracked over time and across variants, providing data-driven insights for prompt optimization.

The deployment system lets teams deploy LLM application variants as API endpoints with built-in versioning, rollback, and traffic splitting. This enables A/B testing of different prompt configurations in production and gradual rollout of improvements. Each deployment includes monitoring for latency, cost, and quality metrics.

Agenta supports multiple application types beyond simple prompt-response patterns, including RAG pipelines, multi-step chains, and agent workflows. The platform is framework-agnostic — it works with LangChain, LlamaIndex, custom code, or direct API calls. This flexibility makes it useful for teams with diverse AI application architectures.

As an open-source project (MIT license), Agenta can be self-hosted for full data control. The managed cloud offering provides hosting, scaling, and team collaboration features for organizations that prefer not to manage infrastructure.