Open-source engine enables 4-bit Gemma 4 26B model on M-series Macs with 2 GB RAM

TurboFieldfare Breaks Ground in On-Device AI Performance

In a groundbreaking achievement, a specialized inference engine called TurboFieldfare has been developed to run 4-bit Gemma 4 26B-A4B-IT models on M-series Macs using as little as 2 GB of RAM. The engine, written in Swift and Metal, pushes the limits of on-device AI by utilizing a novel approach to memory management. By keeping the shared part of the model and the key-value (KV) cache in RAM, while streaming only the required experts from the SSD, TurboFieldfare enables the processing of large models that would otherwise exceed the available memory.

The engine's performance is impressive, with TurboFieldfare generating 5-6 tokens per second on an 8 GB M2 MacBook Air and 31-35 tokens per second on an M5 MacBook Pro. The developer, who has run over 100 experiments to perfect the engine, has also added an experimental OpenAI-compatible local server that supports streaming and tool calls. This server reuses one prompt prefix from the KV cache, further enhancing the engine's capabilities. The Mac app is easily installable, and the model can be downloaded from Hugging Face.

TurboFieldfare is a testament to the ingenuity of its developer and the rapidly evolving field of on-device AI. The engine's capabilities and performance make it an exciting tool for developers and researchers, and the developer is eager to receive feedback and continue improving the engine. With its potential to unlock new possibilities in on-device AI, TurboFieldfare is an achievement worth noting.