05/24/2025
Very small: can run on the edge to allow something like a Raspberry Pi to make basic decisions for your appliance even if disconnected from the internet. Example: those are some time series parameters and instructions, decide if watering the plants or not; vision models that can watch a camera and transcribe what it is seeing in a basic way, ...
Small: runs in an average laptop not optimized for inference of LLMs, like Gemma 3 4B.
Medium: runs in a very high spec computer that people can buy for less than 5k. 30B, 70B dense models or larger MoEs.
Large: Models that big LLM providers sell as "mini", "flash", ...
Extra Large / SOTA: Gemini 2.5 PRO, Claude 4 Opus, ChatGPT O3, ...
05/24/2025
I'm not sure if you're implying that very small language models would be run in your raspberry pi example, but for use cases like the time series one, wouldn't something like an LSTM or TiDE architecture make more sense than a language model?
These are typically small and performant both in compute and accuracy/utility from what I've seen.
I think with all the hype at the moment sometimes AI/ML has become too synonymous with LLM
05/24/2025
Sure if you have a specific need you can specialize some NN with the right architecture, collecting the data, doing the training several times, testing the performances, ... Or: you can download an already built LLM and write a prompt.
05/24/2025
So one of the use cases we're serving in production is predicting energy consumption for a home. Whilst I've not tried, I'm very confident that providing an LLM the historical consumption and asking it to predict future consumption will under perform compared to our forecasting model. The compute required is also several orders of magnitude lower compared to an LLM
05/24/2025
What zero shot would you suggest for that task on an rpi? A temporal fusion thing?
05/24/2025
The small gemma 3 and Qwen 3 models can do wonders for simple tasks as bag of algorithms.
05/24/2025
Those would use more ram than most rpi have wouldn't they? Gemma uses 4GB right?
05/24/2025
Gemma 3 4B QAT int4 quantized from bartowsky should barely fit in a 4GB Raspberry Pi, but without the vision encoder.
However the brand-new Gemma 3n E2B and E4B models might fit with vision.
05/24/2025
Yep, the Gemma 3 1B would be 815MB, with enough margin for a longer prompt. Probably more realistic.
05/24/2025
Nope, gemma3 and qwen3 exist of many sizes, including very small ones, that 4-bit quantized can run on very small systems. Qwen3-0.6B, 1.7B, ... imagine if you quantize those to 4 bit. But there is the space for the KV cache, if we don't want to limit the runs to very small prompts.
05/24/2025
He's talking about general purpose zero shot models.
05/24/2025
Why in the world do you need such sophistication to know whether to water the plants or not?
05/24/2025
When you have a golden hammer everything starts to look like a nail
05/24/2025
this
05/24/2025
There are places where: a) weather predictions are unreliable, b) there is scarcity of water. Just making the right decision on at what hour to water is a huge monthly saving of water.
05/24/2025
None of which need AI hype crap. Some humidity sensors, photosensors, etc. will do the job.
05/24/2025
Need is a very strong word. We don't need a lot of we have today.
But as a hobbyist I would prefer to program in an LLM than learn a bunch of algorithms, and sensor readings. It's also very similar to how I would think about it, making it easier to debug.
05/24/2025
I think there’s two schools of thought. The models will get so big everyone everywhere will use them for everything and they will make lots of money on api calls. The models will get cheaper and cheaper computationally on inference that implementing them on the edge will cost nothing and so an LLM will be in everything. Then every computational device will have one as long as you pay a license fee to the people who trained them.
05/24/2025
Or a farmer
05/25/2025
In a greenhouse operation with high-valued crops. Automated control technologies in those applications have been around for decades, and AI is competing with today’s sophisticated control technology designed, operated and continually improved by agriculturists with detailed site-specific knowledge of water (quality, availability, etc.), cultivars, markets, disease pressures, etc.. The marginal improvements AI can make in a process of poor data quality and availability, an existing, finely tuned, functioning control system, and facing the vagaries of managing dynamic living systems are…tiny.
The solution for water-constrained operations in the Americas is move to a location with more water, not AI.
For field crops…in the Americas, land and water is too cheap and crop prices are too low to be optimized with AI at the present era. The Americas (10% of world pop) could meet 70% of world food demand if pressed with today’s technologies…40% without breaking a sweat. The Americas are blessed.
Talk to the Saudis, Israel, etc. but, even there, you will lose more production by interfering in the motivations, engagement levels and cultures of working farmers than can be gained by optimizing by any complex opaque technological scheme, AI or no. New cultivars, new chemicals, new machinery even…few problems (but see India for counter examples). Changing millennia of farming practice with expensive, not-locally-maintainable, opaque technology…just no. Great truth learned over the last 70 years of development.
05/24/2025
Does it have to be computed at the edge by every person?
05/24/2025
Just as the other comment "have to" is a very strong word. But there are benefits to it: a) adaptability to local weather patterns, b) no access to WiFi in large properties.
05/26/2025
I see. I guess it all boils down to how low power you can make this.
Keep in mind that there are other wireless communication systems that are long range and low power that are specifically designed to handle this scenario
05/24/2025
In this case, "sophistication" meaning throwing insane amounts of compute power and data at the problem? In older times we'd probably call that "brute forcing".
05/25/2025
Today, I asked a colleague to pass me a pen. Was that a egregiously simple task for such a powerful intelligence?
05/24/2025
For “very small”, I would add “can be passively cooled” as a criterion.
05/24/2025
> Example: those are some time series parameters and instructions, decide if watering the plants or not
How is that a "language model"?
05/24/2025
Is language model used to mean neural net, with transformers, attention that takes in a series of tokens and out outs a prediction as a value?
Working with time series data would work in that case.
05/24/2025
> Example: those are some time series parameters and instructions, decide if watering the plants or not; vision models that can watch a camera and transcribe what it is seeing in a basic way, ...
This is the problem I have with the general discourse of "AI" even on Hacker News, of all places. Everything you listed is not an example of a *language model*.
All of those can either be implemented as a simple "if", decision tree, decision table, and finally actual ML in the example of cameras and time series predication.
Using an LLM is not just ridiculous here but totally the wrong fit and a waste of resources.
05/25/2025
> Using an LLM is not just ridiculous here but totally the wrong fit and a waste of resources.
Time and labor are resources too. There's a whole host of problems where "good enough" is tremendously valuable.
05/24/2025
How do we call the models beyond extra large which are so big they can't be served publicly because their inference cost is too high? Do such exist?
05/24/2025
The term is too overloaded.
I'll add one more: a LLM small enough that it can be trained from scratch on one A100 in 24 hours. Is it really small if it takes $10,000 to train? Or leave that term for $200 models?
Back to your definitions, there are sub-1B models people are using. I think I saw one in the 400-600M range for audio. Another person posted here a 100M-200M model for extracting data from web pages. We told them to just use a rules-based approach where possible but they believed the SLM worked better.
Then, there's projects like BabyLM that can be useful at 10M:
05/24/2025
But you only have to train the foundational model once - so with open weights it's not really a problem.
Maybe resources needed for fine-tuning would be nice to see.
05/24/2025
Most have been trained on illegally-distributed, copyrighted works. They might output them, too. People might want untainted models. Additionally, some have weaknesses due to tokenizers, pre-training data, or moral alignment (political bias).
For those reasons, users might want to train a new model from scratch.
Researchers of training methods have a different problem. They need to see whether a new technique, like an optimization algorithm, gets better results. They try them more quickly with less money if they have small, training runs representative of what larger models do. If BabyLM-10M was representative, they could test each technique at the FLOPS/$ of a 10M model instead of a 1B model.
So, both researchers and users might want new models trained from scratch. The cheaper to train, the better.
05/24/2025
05/24/2025
> Another person posted here a 100M-200M model for extracting data from web pages
Could you post a link to this comment or thread. I can't seem to find this model by searching but world love to try it out.
05/24/2025
I think I found it. I could be getting the numbers mixed up with another SLM. That example's smaller model was 500M:
https://news.ycombinator.com/item?id=41515730
05/24/2025
I think of “fits on the overpowered M1/2/3/4 64GB MacBook Pro my employer gave me” as the dividing line. We’re getting to within spitting distance of models that can code well at that size.
05/24/2025
https://mistral.ai/news/devstral and https://huggingface.co/nvidia/AceReason-Nemotron-14B were released in just the last couple of days and work in 24GB 4090 GPUs/32GB Macbook Pros just fine
05/25/2025
+1 that is my experience. devstral 24B on my Mac does very well designing code. I am writing a book on AI first software development and I have been exploring using small models in a specific process of separate steps for analysis, design, implementation, etc.
05/24/2025
I want my next laptop to be the 128gb M series monster. That will run not quite frontier models but ones that are close in performance, and run them fast.
05/24/2025
And, also quite important, leave your system enough RAM to do anything else.
05/25/2025
I don’t know about that. Could a cheaper 2nd MacBook in all cases be used as an outboard unMath/unTPU coprocessor to handle those other tasks? (I have no clue.)
05/25/2025
are you sure? lpddr5 is somewhere in the range of ~0.25 W/GB (for some reason this stat is hard to find good values for), so 128gb of RAM will mean your laptop idles at >25 watts.
05/25/2025
True. But I am usually near an outlet, and with the M series even the overpowered ones just bring you back to the battery life of a good Intel laptop (3-4 hours) instead of 8+ for smaller ones.
That and running local LLMs pretty much requires an outlet. The GPU goes up to 50 watts. Battery life drops from many hours to less than one.
05/25/2025
that 25 watts was just for ram. You still need another ~5 watts for cpu and display at idle, so you're talking about a laptop that lasts ~3 hours with the screen on literally doing nothing. For comparison, my 2021 XPS-13 (with a pretty small 55wH battery) lasts at least 4 hours under actual use (video playback, browsing github, compiling code etc).
Also, when your laptop is using 25W just for the RAM, that's ~20 watts less that the CPU/GPU get when you want to power up the CPU/GPU.
05/24/2025
05/24/2025
There is a "small language model", and then there is a "small LARGE language model". In late 2018, BERT (110 million params) would've been considered a "large" language model. A "small" LM would be some markov chain or a topic model (e.g. latent dirichlet allocation) - technically they would be considered generative language models since they learn joint distributions of params and data (words), and can then sample from that distribution. But today, we usually map "small" LMs to "small" LLMs, so in that sense a small LLM would be anything from BERT to around 3-4B params.
05/24/2025
Maybe we should appropriate the old DOS/x86 memory model names and give them “class-relative” sizes.
“tiny” can run on a microcontroller, “compact” on a Rpi, “small” on a phone, “medium” on a single GPU machine, “large” on AI class workstation hardware, and “huge” on a data center cluster.
05/24/2025
I appreciate how it redefines “small” not by parameter count but by practical impact and deployability.
05/24/2025
I do not — parameter count is objective, practical impact depends on such a multitude of factors that any comparison becomes virtually meaningless.
05/24/2025
The standard for parameters count is rapidly evolving. Something large now will be small tomorrow, there is no point in using such a moving target as a criterion.
05/24/2025
Sure, but nonetheless whether the model is called "small" at some time t should depend on the parameter count and t, not some arbitrarily specified metric of deployability.
05/24/2025
How can a Large Language Model be a small language model?
05/24/2025
Because words are arbitrary. See Saussure.
05/24/2025
Why wouldn’t there be any? Right now there are large large language models, medium large language models and small large language models. You can say there are also tiny large language models and extra large large language models. Nothing confusing about it.
05/24/2025
See also the Little Giant Girl who is part of The Sultan's Elephant and several other Royal de Luxe performances. She's clearly a little girl, but, she's also clearly a giant.
05/24/2025
I want to see more models that can be streamed to a browser and run locally via wasm. That would be my hope for small models. In the <100mb range.
05/24/2025
After experimenting with 1B models, I am starting to think that any model with 1B parameters or less will probably lack a lot of the general intelligence that we observe in the frontier models, because it seems physically impossible to encode that much information into so few parameters. I believe that in the range of very small models, the winner will be models that are fine tuned to a small range of tasks or domains, such as a model that can translate between English and any other language, or a legal summarization model, etc.
05/24/2025
Have you heard of Transformers.js? They are running onnx inside browser:
https://huggingface.co/docs/transformers.js/en/index
05/24/2025
05/24/2025
Why? Just so user data stays local?
05/24/2025
Yes. And also, cost to run it.
05/25/2025
Sounds like a case where egress could easily cost more than compute.
05/24/2025
> Small models used to mean tiny. Now they mean "runs without drama."
Does this mean without a dedicated electric power plant?
I wanted to say "Right, big-sized. Do you want fries with that?", but I couldn't figure out how to work that in, so I won't say it.
05/24/2025
On this topic, I've been wondering if models are capable of recommending other models for a given machine spec, for example: which model, if any, would be recommended for a laptop with a Ryzen 9 6000S and RTX 3060m (random spec).
05/24/2025
These terms are all relative, but there's also "BabyLlama", which measures its parameter count in millions rather than billions.
05/24/2025
A traditional Markov model trained (rather, just "fitted") on tokens or words is a small language model.
05/24/2025
(To share a recent personal experience about Markov models: I bootstrapped recently a HMM with hand-assigned weights. It was around 15x15 class transitions, 225 weights. That's small. Or rather, microscopic. Then I ran it against real data, and picked up examples of wrong classifications, and made them auxillary training data. Of course, it was not a language model, language model is impossible to fit in such a small space. It was a model of transitions of chapter "types" in novels, where types are something like "Epilogue" , "Prologue", "Chapter 23", "Table of Contents", "Afterword" etc.)
05/25/2025
It’s always been a little arbitrary. Can it fit on 3090 seems like a reasonable cutoff to me for now
05/24/2025
This post is 100% rewritten or fully generated by gpt-4o. It has the gpt smell all over it.
05/24/2025
> In a world chasing ever-bigger models, small ones are quietly doing more with less—and that's exactly what makes them powerful.
100%. It has enough technical details that maybe a human did something. But who knows.
05/24/2025
Is there a problem with that? If so, what is it? I don't mind as long as it's not the boilerplate AI spits out by default.
05/24/2025
Nah not really, the information content is what counts of course. It’s just a bit cringe to see it happen.
05/24/2025
This is also where MoE shines with a mixture of small and large language models.
05/24/2025
whatever fits into gaming GPU such as GeForce 3080
05/24/2025
[dead]
05/24/2025
Just ask my ex-wife!