vitalik.eth: Qwen 3.8 flash and llama.cpp make local AI models viable

Submit
vitalik.eth@vitalik.eth

Qwen 3.8 flash is truly impressive, and llama.cpp has been rapidly getting better and better at processing it columns are: pre-existing prompt, new prompt, generated, input tok/s, output tok/s This is on my laptop (strix halo). I think we're very close to the point where you can just use local models for a large share of tasks, and for anything more advanced, workflows like "use your local model to orchestrate queries to powerful models so your queries don't leak your personal information" actually become viable.

Cast image
farcaster.xyz

vitalik.eth: Qwen 3.8 flash and llama.cpp make local AI models viable

AI summary of the linked article

Vitalik Buterin writes that Qwen 3.8 flash is impressive and that llama.cpp has been rapidly improving at running it, and he says local models are close to handling a large share of tasks. He reports benchmark results from his laptop, which uses a Strix Halo chip, with columns for pre-existing prompt, new prompt, generated tokens, input tokens per second, and output tokens per second. He also suggests that using a local model to orchestrate queries to powerful remote models could become viable, so that personal information does not leak to those models.

avatar
I actually have been running qwen 3.8 in ollama local on my laptop since it’s out. it is not fast, but fast enough. I like it, it’s very nice, using it all the time. what Vitalik does not mention it’s also free ;)