Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone leonickson1 / Swiftlet Public Notifications You must be signed in to change notification settings Fork 0 Star 18 mainBranche
By Coderz Club · 2026-08-04 · Tags: git
Show HN: Run an 80B Qwen in 4.3 GB of RAM on a Mac, and a 35B on an iPhone
leonickson1 / Swiftlet Public Notifications You must be signed in to change notification settings Fork 0 Star 18 mainBranchesTagsGo to fileCodeOpen more actions menuFolders and filesNameNameLast commit messageLast commit dateLatest commit History6 Commits6 CommitsSourcesSources Tests/SwiftletCoreTestsTests/SwiftletCoreTests assets/model-configsassets/model-configs fixturesfixtures referencesreferences scriptsscripts .gitignore.gitignore IPHONE.mdIPHONE.md LICENSELICENSE PLAN.mdPLAN.md Package.resolvedPackage.resolved Package.swiftPackage.swift README.mdREADME.md THIRD_PARTY_NOTICES.mdTHIRD_PARTY_NOTICES.md View all filesRepository files navigationSwiftlet Run 35B and 80B Qwen models on ordinary Apple devices, including iPhones. Swiftlet is a Swift + Metal runtime for the Qwen3-Next and Qwen3.5/3.6 MoE hybrid model family. It keeps only the small dense core of a model resident in memory and streams the routed Mixture-of-Experts weights from storage on demand. The result: Model Disk Peak RAM Decode speed (M5 Mac) Qwen3.6-35B-A3B, 4-bit 18 GB 2.6 GB 7 to 11 tok/s Qwen3-Next-80B-A3B, 4-bit 42 GB 4.3 GB 4.5 to 5 tok/s The 35B also runs on an iPhone 17 in about 2.5 GB of RAM, at about 1 tok/s today. As far as we know, that is the first time a model of this class has run natively on a phone. Status: working end to end. Both models generate correct, validated output. The current focus is kernel speed (the decode loop is dispatch bound, not IO bound, so there is clear headroom). One expectation to set honestly: only about 3B parameters are active per token, so these models chat and write like large models but recall facts like small ones. Quick start: try it on a Mac git clone https://github.com/leonickson1/Swiftlet.git && cd Swiftlet swift build -c release # Download the 35B container from Hugging Face (resumable): .build/release/swiftlet-repack \ --from-hf Leonickson/Qwen3.6-35B-A3B-qpack \ --output ~/models/qwen3.6-35b.qpack # Or the 80B (42 GB on disk, still only ~4.3 GB of RAM): .build/release/swiftlet-repack \ --from-hf Leonickson/Qwen3-Next-80B-A3B-qpack \ --output ~/models/qwen3-next-80b.qpack # Chat (applies the model chat template, disables the reasoning block, # keeps conversation state so follow-ups prefill only the new turn): .build/release/swiftlet chat ~/models/qwen3.6-35b.qpack \ "Who wrote One Hundred Years of Solitude?" "What language did he write it in?" # One-shot generation with stats: .build/release/swiftlet generate ~/models/qwen3.6-35b.qpack \ --gpu --chat --prompt "Explain expert streaming in one paragraph." # OpenAI-compatible server (loopback only): .build/release/swiftlet-server --model ~/models/qwen3.6-35b.qpack --port 8080 The same command also repacks raw MLX checkpoints (--from-hf mlx-community/... or --source /path/to/checkpoint). Requirements: Apple Silicon, macOS 14+ or iOS 17+, free SSD space for the container (18 GB for the 35B, 42 GB for the 80B). Try it on your phone The 35B runs on iPhone inside Priv AI on the App Store: open Settings, then Experimental Models, and download the model. It streams from storage and chats on-device with no server involved. The Experimental Models feature ships in the newest app version, which is still in App Store review, so it may not appear for a couple of days. If you want the phone experience today, build the app from source: the app is open source at leonickson1/localLLM. Clone this repo next to it as swiftlet, open the Xcode project, and run it on your iPhone. How it works These models activate only about 3B of their parameters per token. Each layer routes every token to 10 of 512 experts (80B) or 8 of 256 (35B). Swiftlet: keeps the dense weights resident: attention, DeltaNet projections, routers, shared experts, embeddings. About 1.3 GB (35B) or 2.5 GB (80B) at 4-bit; repacks the tens of thousands of routed experts into fixed-stride blobs in a .qpack container, so fetching one expert is exactly one pread from SSD, no mmap and no page-cache thrash; caches hot experts in a bounded pool with LFU plus recency eviction. Cache size barely affects speed (measured 43 to 70 percent hit rates at the same throughput), because Apple SSDs absorb the misses; runs the whole forward pass on Metal with runtime-compiled shaders, so no Metal toolchain is needed at build time and the same code ships on iOS. 75 percent of the layers use Gated DeltaNet linear attention with a fixed-size recurrent state, so there is no growing KV cache for those layers at any context length. Four ways to use it Swiftlet is a library first: The Swift package. Add SwiftletCore to any macOS or iOS app and use SwiftletSession for chat with streaming deltas, conversation caching, sampling with repetition control, and memory-pressure handling built in. The CLI. swiftlet chat and swiftlet generate for local use and benchmarking, swiftlet-repack to build containers from MLX checkpoints (including streaming straig
leonickson1 / Swiftlet Public Notifications You must be signed in to change notification settings Fork 0 Star 18 mainBranchesTagsGo to fileCodeOpen more actions menuFolders and filesNameNameLast commit messageLast commit dateLatest commit History6 Commits6 CommitsSourcesSources Tests/SwiftletCoreTestsTests/SwiftletCoreTests assets/model-configsassets/model-configs fixturesfixtures referencesreferences scriptsscripts .gitignore.gitignore IPHONE.mdIPHONE.md LICENSELICENSE PLAN.mdPLAN.md Package.resolvedPackage.resolved Package.swiftPackage.swift README.mdREADME.md THIRD_PARTY_NOTICES.mdTHIRD_PARTY_NOTICES.md View all filesRepository files navigationSwiftlet Run 35B and 80B Qwen models on ordinary Apple devices, including iPhones. Swiftlet is a Swift + Metal runtime for the Qwen3-Next and Qwen3.5/3.6 MoE hybrid model family. It keeps only the small dense core of a model resident in memory and streams the routed Mixture-of-Experts weights from storage on demand. The result: Model Disk Peak RAM Decode speed (M5 Mac) Qwen3.6-35B-A3B, 4-bit 18 GB 2.6 GB 7 to 11 tok/s Qwen3-Next-80B-A3B, 4-bit 42 GB 4.3 GB 4.5 to 5 tok/s The 35B also runs on an iPhone 17 in about 2.5 GB of RAM, at about 1 tok/s today. As far as we know, that is the first time a model of this class has run natively on a phone. Status: working end to end. Both models generate correct, validated output. The current focus is kernel speed (the decode loop is dispatch bound, not IO bound, so there is clear headroom). One expectation to set honestly: only about 3B parameters are active per token, so these models chat and write like large models but recall facts like small ones. Quick start: try it on a Mac git clone https://github.com/leonickson1/Swiftlet.git && cd Swiftlet swift build -c release # Download the 35B container from Hugging Face (resumable): .build/release/swiftlet-repack \ --from-hf Leonickson/Qwen3.6-35B-A3B-qpack \ --output ~/models/qwen3.6-35b.qpack # Or the 80B (42 GB on disk, still only ~4.3 GB of RAM): .build/release/swiftlet-repack \ --from-hf Leonickson/Qwen3-Next-80B-A3B-qpack \ --output ~/models/qwen3-next-80b.qpack # Chat (applies the model chat template, disables the reasoning block, # keeps conversation state so follow-ups prefill only the new turn): .build/release/swiftlet chat ~/models/qwen3.6-35b.qpack \ "Who wrote One Hundred Years of Solitude?" "What language did he write it in?" # One-shot generation with stats: .build/release/swiftlet generate ~/models/qwen3.6-35b.qpack \ --gpu --chat --prompt "Explain expert streaming in one paragraph." # OpenAI-compatible server (loopback only): .build/release/swiftlet-server --model ~/models/qwen3.6-35b.qpack --port 8080 The same command also repacks raw MLX checkpoints (--from-hf mlx-community/... or --source /path/to/checkpoint). Requirements: Apple Silicon, macOS 14+ or iOS 17+, free SSD space for the container (18 GB for the 35B, 42 GB for the 80B). Try it on your phone The 35B runs on iPhone inside Priv AI on the App Store: open Settings, then Experimental Models, and download the model. It streams from storage and chats on-device with no server involved. The Experimental Models feature ships in the newest app version, which is still in App Store review, so it may not appear for a couple of days. If you want the phone experience today, build the app from source: the app is open source at leonickson1/localLLM. Clone this repo next to it as swiftlet, open the Xcode project, and run it on your iPhone. How it works These models activate only about 3B of their parameters per token. Each layer routes every token to 10 of 512 experts (80B) or 8 of 256 (35B). Swiftlet: keeps the dense weights resident: attention, DeltaNet projections, routers, shared experts, embeddings. About 1.3 GB (35B) or 2.5 GB (80B) at 4-bit; repacks the tens of thousands of routed experts into fixed-stride blobs in a .qpack container, so fetching one expert is exactly one pread from SSD, no mmap and no page-cache thrash; caches hot experts in a bounded pool with LFU plus recency eviction. Cache size barely affects speed (measured 43 to 70 percent hit rates at the same throughput), because Apple SSDs absorb the misses; runs the whole forward pass on Metal with runtime-compiled shaders, so no Metal toolchain is needed at build time and the same code ships on iOS. 75 percent of the layers use Gated DeltaNet linear attention with a fixed-size recurrent state, so there is no growing KV cache for those layers at any context length. Four ways to use it Swiftlet is a library first: The Swift package. Add SwiftletCore to any macOS or iOS app and use SwiftletSession for chat with streaming deltas, conversation caching, sampling with repetition control, and memory-pressure handling built in. The CLI. swiftlet chat and swiftlet generate for local use and benchmarking, swiftlet-repack to build containers from MLX checkpoints (including streaming straig