PacktPublishing/LLM-Engineers-Handbook
The LLM's practical guide: From the fundamentals to deploying advanced LLM and RAG apps to AWS using LLMOps best practices
Language:Python
Total stars: 165
Stars trend:
20 Oct 2024
9am ██▉ +23
10am ██▍ +19
11am ██▎ +18
12pm █▋ +13
1pm ███▎ +26
#python
#aws, #finetuningllm, #genai, #llm, #llmevaluation, #llmops, #mlsystemdesign, #mlops, #rag

NVIDIA/garak
the LLM vulnerability scanner
Language:Python
Total stars: 1604
Stars trend:
17 Nov 2024
5am ▏ +1
6am ▏ +1
7am +0
8am ▎ +2
9am +0
10am ▏ +1
11am ▏ +1
12pm ▏ +1
1pm ▋ +5
2pm ██▏ +17
3pm ███▏ +25
4pm ███▎ +26
#python
#ai, #llmevaluation, #llmsecurity, #securityscanners, #vulnerabilityassessment

Helicone/helicone
🧊 Open source LLM-Observability Platform for Developers. One-line integration for monitoring, metrics, evals, agent tracing, prompt management, playground, etc. Supports OpenAI SDK, Vercel AI SDK, Anthropic SDK, LiteLLM, LLamaIndex, LangChain, and more. 🍓 YC W23
Language:TypeScript
Total stars: 2410
Stars trend:
20 Dec 2024
9am ▏ +1
10am +0
11am █▍ +11
12pm █▍ +11
1pm █▏ +9
2pm ▉ +7
3pm ▌ +4
4pm █▏ +9
5pm █▏ +9
6pm ▍ +3
7pm ▍ +3
8pm █▏ +9
#typescript
#agentmonitoring, #analytics, #evaluation, #gpt, #langchain, #largelanguagemodels, #llamaindex, #llm, #llmcost, #llmevaluation, #llmobservability, #llmops, #monitoring, #opensource, #openai, #playground, #promptengineering, #promptmanagement, #ycombinator

Agenta-AI/agenta
The open-source LLMOps platform: prompt playground, prompt management, LLM evaluation, and LLM observability all in one place.
Language:Python
Total stars: 2418
Stars trend:
12 Apr 2025
3pm ▎ +2
4pm +0
5pm +0
6pm █▊ +14
7pm █▋ +13
8pm █ +8
9pm ▋ +5
10pm ▉ +7
11pm ▊ +6
13 Apr 2025
12am █ +8
1am █▏ +9
2am ▍ +3
#python
#llmasajudge, #llmevaluation, #llmframework, #llmmonitoring, #llmobservability, #llmplatform, #llmplayground, #llmtools, #llmopsplatform, #promptengineering, #promptmanagement, #ragevaluation

comet-ml/opik
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
Language:Python
Total stars: 6904
Stars trend:
26 Apr 2025
1am ▎ +2
2am ▍ +3
3am ▍ +3
4am ▏ +1
5am ▏ +1
6am ▍ +3
7am ████▊ +38
8am █▎ +10
9am ▌ +4
10am █▏ +9
11am ▎ +2
12pm ▌ +4
#python
#langchain, #llamaindex, #llm, #llmevaluation, #llmobservability, #llmops, #opensource, #openai, #playground, #promptengineering

confident-ai/deepeval
The LLM Evaluation Framework
Language:Python
Total stars: 6493
Stars trend:
22 May 2025
9am ▏ +1
10am +0
11am +0
12pm ▎ +2
1pm ██▏ +17
2pm █▏ +9
3pm █▏ +9
4pm █▎ +10
5pm ▊ +6
6pm █▏ +9
7pm ▊ +6
8pm █▏ +9
#python
#evaluationframework, #evaluationmetrics, #llmevaluation, #llmevaluationframework, #llmevaluationmetrics

cvs-health/uqlm
UQLM: Uncertainty Quantification for Language Models, is a Python package for UQ-based LLM hallucination detection
Language:Python
Total stars: 217
Stars trend:
25 May 2025
4pm ███ +24
5pm ██▍ +19
6pm ██▏ +17
7pm █▋ +13
8pm █▊ +14
#python
#aievaluation, #aisafety, #confidenceestimation, #confidencescore, #hallucination, #hallucinationdetection, #hallucinationevaluation, #hallucinationmitigation, #llm, #llmevaluation, #llmhallucination, #llmsafety, #uncertaintyestimation, #uncertaintyquantification

promptfoo/promptfoo
Test your prompts, agents, and RAGs. Red teaming, pentesting, and vulnerability scanning for LLMs. Compare performance of GPT, Claude, Gemini, Llama, and more. Simple declarative configs with command line and CI/CD integration.
Language:TypeScript
Total stars: 6971
Stars trend:
31 May 2025
9pm ▌ +4
10pm ▋ +5
11pm ▏ +1
1 Jun 2025
12am █ +8
1am ▋ +5
2am ▊ +6
3am █▏ +9
4am ▊ +6
5am ▉ +7
6am █▏ +9
7am ▍ +3
8am █▋ +13
#typescript
#ci, #cicd, #cicd, #evaluation, #evaluationframework, #llm, #llmeval, #llmevaluation, #llmevaluationframework, #llmops, #pentesting, #promptengineering, #prompttesting, #prompts, #rag, #redteaming, #testing, #vulnerabilityscanners