Skip to content

ai 1 min read

TIL - How to run Hugging Face models with Ollama

You can now deploy any GGUF model with Ollama, in just a few clicks!

You can use any GGUF quants created by the community on Hugging Face directly with Ollama, without the need to create a new Modelfile . It works like a charm with all llama.cpp compatible models, with all sizes, from 0.1B up to 405B parameters.

Simply filter GGUF models, select the quant type as per your requirement, and it is done!

ollama run hf.co/bartowski/Qwen2.5.1-Coder-7B-Instruct-GGUF:Q5_K_L
shell

Recommended small models - a bartowski CollectionThis is everything recent smaller than ~25B parameters that are high quality/reputable a bartowski Collection

More Information

Use Ollama with any GGUF Model on Hugging Face HubWe’re on a journey to advance and democratize artificial intelligence through open source and open science. huggingface.co