1

What are the main alternatives for prompt ops nowadays? By prompt ops, I mean a comprehensive solution for tracking prompt engineering experiments and also registering prompts in different stages, similar to how I would with an ML model in a model registry.

I went through MLFLow and it seems to have somehow what I look for, instead Sagemaker has tracking prompt experiments features, but it seems to allow prompt registration just using a "fake Estimator" object, that's not a very clean solution. Please correct me if there is another option for SageMaker I did not go through.

Raffaele
  • 73
  • 1
  • 5

3 Answers3

2

We use Langfuse to manage 100% of our prompts + us them for observability, evals, too. Excellent product & the team is very responsive to requests and it's open source if you care about that: https://github.com/langfuse/langfuse/

Jordie
  • 36
  • 1
0

Check out https://github.com/agenta-ai/agenta it's open-source and provides all the tools for prompt management (versioning + environments), llm evaluation and llm observability.

Mahmoud
  • 101
  • 1
0

We use Maxim AI (https://www.getmaxim.ai/) to manage prompts, run agent simulation, evaluations and also for logging and tracing of our LLM applications. You could check them out if you are building GenAI applications.

user183663
  • 1