dipampaul17/KVSplit
Run larger LLMs with longer contexts on Apple Silicon by using differentiated precision for KV cache quantization. KVSplit enables 8-bit keys & 4-bit values, reducing memory by 59% with <1% quality loss. Includes benchmarking, visualization, and one-command setup. Optimized for M1/M2/M3 Macs with Metal support.
Language: Python
#apple_silicon #generative_ai #kv_cache #llama_cpp #llm #m1 #m2 #m3 #memory_optimization #metal #optimization #quantization
Stars: 222 Issues: 1 Forks: 5
https://github.com/dipampaul17/KVSplit

RunanywhereAI/RCLI
Talk to your Mac, query your docs, no cloud required. On-device voice AI + RAG
Language: C++
#ai_assistant #apple_silicon #kitten_tts #kokoro_tts #lfm2 #llama_cpp #llm #local_ai #metal #on_device_ai #parakeet #qwen3 #rag #speech_to_text #text_to_speech #tool_calling #voice_assistant
Stars: 627 Issues: 10 Forks: 19
https://github.com/RunanywhereAI/RCLI

StayLameBro/backburner
Your iPhone helps your Mac run a 27B model: faster prompt reading and more context over a USB-C cable
Language: Python
#apple_silicon #ios #iphone #llama_cpp #llm_inference #local_llm #macos #metal #qwen #sme2 #speculative_decoding
Stars: 693 Issues: 4 Forks: 66
https://github.com/StayLameBro/backburner

ARahim3/DigUp
Search your Mac by what's in your files: photos, PDFs, recordings, videos and code. On-device, built on Google DeepMind's EmbeddingGemma 2.
Language: Swift
#embeddinggemma #embeddings #llama_cpp #local_first #macos #menubar_app #multimodal #on_device #semantic_search #swift
Stars: 1348 Issues: 25 Forks: 93
https://github.com/ARahim3/DigUp