\

The Smallest Brain You Can Build: A Perceptron in Python

288 points - today at 12:28 AM

Source
  • romaniv

    today at 2:13 PM

    I think it should be quite obvious that perceptrons are far from the smallest units that are capable of learning. They store many bytes of information, require a non-local update process, need numeric (i.e. symbolic) inputs and involve relatively complex computations. You can go much simpler. For example:

    https://medium.com/@VictorBanev/the-simplest-learning-machin...

    This is a description of a 5-line algorithm that learns and stores approximate probability of an event using just 1 byte of persistent memory.

      • a1o

        today at 2:44 PM

        That is a cool algorithm, indeed very interesting 5 lines. Also fun to see things in C#. :)

        • DevarshRanpara

          today at 4:37 PM

          True, there can be simpler versions compared to perceptron, just like you made. I have learned something new from that, Thanks for sharing.

      • kzrdude

        today at 10:51 AM

        I think Karpathy's microgpt blogpost is the best in this genre in a long time, and it also includes a multi layer perceptron. It's a step up in the hierarchy, so reading both is helpful, of course.

        https://karpathy.github.io/2026/02/12/microgpt/

          • Lerc

            today at 11:31 AM

            I'm not sure if I'd like to declare a best. There are so many different approaches and I think their ability to inform is cumulative,

            I like the ability of this article to do the tiny training runs in browser. It makes the point of a bias clear. Too many tutorials get sucked into the proof of zero times anything is zero. Everyone knows that. What you should show is where that mstters in the problem at hand.

            3blue1brown does one of the best depictions of why we need an activation function.

            Karpathy's videos are a little tougher for a beginner to grasp, but excel at solving a complete problem. I knew all of the theory behind what it takes to make micrograd before I made my own by following the video, but what you get from doing it can't be understated.

            It's hard to describe but it what you learn is more of a feel than pure knowledge. It gives you a better sense of knowing when the principles apply in other circumstances.

            Perhaps it's the distinction of understanding how springs and gears work, then looking at a clock and understanding how the gears and springs move the hands. There's still more needed if you want to make a clock. And that stuff is what let's you also make a wind up toy.

              • DevarshRanpara

                today at 4:54 PM

                I can't agree more with you, It took me many days to understand the "By we need bias?" I know maths, I know programming, but why was not clear. I love 3blue1brown.

        • utopiah

          today at 3:31 PM

          One day I'll write about my 1-liner physics engine...

              let gravity = setInterval( _ => { if (projectile.object3D.position.y > 0) projectile.object3D.position.y \*= .99 }, 100)
          
          Jokes aside I find that providing ridiculously short toy examples that provide the very limited foundation of a concept are extremely empowering in pedagogy. You "get" it right away because it "fits" in your mind, then you dare tinker with it and quickly see how limited it is, then get excited again. It's a powerful trick to learn more IMHO.

            • DevarshRanpara

              today at 4:55 PM

              Yeah, I will try to make more of these, I like to lean things from core, and I like to keep everything as simple as possible.

          • rahen

            today at 6:31 AM

            In the early days of machine learning (before the first AI winter), networks like this were often implemented and trained in hardware: https://en.wikipedia.org/wiki/ADALINE

            That was the first thing that came to mind when I read "the smallest brain you can build". Nowadays, that "small brain" would likely be built on a breadboard using op-amps instead.

              • WarmWash

                today at 3:42 PM

                The quasi-mythical memristor would be choice for bread boarding a brain. However I suppose you could train a model and then manually place fixed resistors to build the network

                • Schlagbohrer

                  today at 1:41 PM

                  Amazing and anachronistic to see something like that from 1960. And then it makes me wonder why there wasn't more progress on neural nets being used for many things prior to the 21st century. (I haven't read the history of the AI winters but I have heard of them)

                    • rahen

                      today at 2:27 PM

                      The first AI winter was largely triggered by Minsky in a book he published in 1969, which mathematically proved that single-layer perceptrons couldn't solve non-linear problems. Favorite quote: "Our intuitive judgment is that the extension [to multilayer systems] is sterile."

                      Yet we had the computational power to run backpropagation in the 1960s and small Transformers in the 1970s (I'm the author of both):

                      https://github.com/dbrll/Xortran (backprop on IBM 1130, 60s)

                      https://github.com/dbrll/ATTN-11 (Transformer on PDP-11, 70s)

                      What was missing wasn't the raw processing power, but the ideas and algorithms themselves. Because funding and research were completely discouraged during the AI winter, neural networks research was left dormant and we lost two decades.

                        • WarmWash

                          today at 3:45 PM

                          I wonder had we invented transformer architecture back in the 70's or 80's, if the pace of hardware innovation would have naturally slowed AI progression, and given humans decades to slowly adapt, rather than the current tidal wave (that seems to grow in size daily) bearing down on us.

                      • mr_toad

                        today at 1:53 PM

                        > why there wasn't more progress on neural nets being used for many things prior to the 21st century

                        They were simply too computationally expensive to train for the limited things they could do. It wasn’t until we had the ability to train large neural networks on commodity hardware that things really took off.

                        • j_bum

                          today at 1:52 PM

                          This doc on Ilya Sutskever & Geoffrey Hinton gives a great background on the progression of deep learning over the past decades [0].

                          Tl;dr - compute was the bottleneck.

                          I am not associated with this channel/video, just love it. I’ve shared it here before.

                          [0] https://youtu.be/glWvwvhZkQ8?si=XjcwWWy43305tl6O

                  • zkmon

                    today at 5:20 AM

                    The IF statement is the root creator of software programming. It has the ability to compare two values against each other and branch out to blocks of instructions. So it is perceiving (reading), decision making and routing - all that which differentiate life from inanimate objects. The AI agents perform the exact same loop, by delegating the first two steps to a model.

                    Going further backwards, the transistor (or a PNP junction) is the hardware level enabler of the IF statement. The action (switching) driven by the current which in turn controls other switches, is the first manifestation of "observe and act" by inanimate things at the speed of electricity.

                    Mechanical equivalents existed ofcourse - speed of a governer which controls the flow of fuel which in turn controls the speed of the governer.

                      • RetroTechie

                        today at 2:00 PM

                        > Going further backwards, the transistor (or a PNP junction) is the hardware level enabler of the IF statement. The action (switching) (..)

                        Back up a bit please! Analog computing is a thing. And it isn't even new - not by a long shot.

                        There are good reasons why practically all computing today is the digital kind. But electronic 'equivalents' of neural nets is one area where analog might make sense. Adding inputs can be as simple as a bunch of resistors + a transistor. Even on modern silicon nodes, that might be a more efficient setup than digital inputs, N-bit adders/multipliers etc. Not saying that's the case, and AI hardware should be based on analog circuitry. But it could be, and perhaps found to be practical.

                          • lambertsimnel

                            today at 2:50 PM

                            > But electronic 'equivalents' of neural nets is one area where analog might make sense.

                            That's an interesting idea, but could the weights be transferred to different hardware and still work? If not, that would be a significant limitation, even if it were preferable in some cases.

                        • mr_toad

                          today at 3:34 PM

                          An artificial neuron needs values to compare (the sum of weighted inputs). You can add values with transistors of course, but you need more than a dozen just to do simple addition. The activation function could be a simple binary comparison (e.g. between a weight and a threshold), but it’s usually more complicated.

                          Artificial neurons are significantly more complex that single transistors, and even a minimal hardwired circuit to implement just one neuron requires quite a number of transistors.

                          • Lerc

                            today at 11:42 AM

                            Fundimentally, when you talk about a if statement, you are talking about the ability to do something different dependent upon some state.

                            It's the same thing as stimulus, response.

                            Unchanging in response to circumstances is static.

                            Changing in the absence of circumstances is randomness.

                            The conditional is all that remains. Changing in response to circumstances

                            (Arguably, unchanging in the absence of circumstances completes the truth table, but it's a whole lot of nothing)

                            • BatteryMountain

                              today at 9:15 AM

                              So, what if, we build a stack/set of transistors in same shape as a trained model? It would eliminate most of the software stack too and should run very fast. No memory/gpu required, the chip acts as both storage and processing device, purpose built to be physical model of a trained model.

                                • mr_toad

                                  today at 3:14 PM

                                  But it can only run that model, so it will be outdated in a few years at best.

                                  • tomtom1337

                                    today at 10:41 AM

                                    This is literally what talaas has done with chatjimmy.ai.

                                    Try it, it's llama 3.1 8B at 16000 tokens per second.

                                    chatjimmy.ai https://taalas.com/the-path-to-ubiquitous-ai/

                                      • jupr

                                        today at 2:28 PM

                                        Wow that incredibly fast. I like this outcome more than centralized datacenters.

                                    • rusk

                                      today at 11:52 AM

                                      There’s lots of things you can do in hardware that could be done in software but cost. FPGA should have solved this long ago, but apparently the guys who own the IP want to make it as hard as possible to use it …

                                  • adrian_b

                                    today at 9:15 AM

                                    The alternative IF expression or statement, in the form introduced by John McCarthy in 1958 (which he used in early 1959 to define his version of AND, OR and NOT), is one of the possible primitives for computation.

                                    There exist several equivalent sets of primitive operations. While the sets containing only NAND or only NOR, or both AND and NOT or both OR and NOT are more notorious, these logical operations are more abstract and they do not indicate precisely a hardware implementation, i.e. there are many distinct hardware methods to make such logical gates.

                                    Other sets of primitive operations map directly to hardware devices, e.g. the sets of primitive operations composed of maximum and complement or of minimum and complement map directly to a hardware implementation using rectifier diodes and inverting amplifiers (which can be made with either semiconductor devices or with vacuum tubes, or also with pneumatic or hydraulic devices).

                                    Other sets of primitive operations are obtained by replacing the maximum or minimum circuits with series or parallel connections of switches, like in the CMOS logic that is nowadays dominant.

                                    The alternative IF expression corresponds in hardware to a 2-way multiplexer, which, together with the 2 constant functions "0" and "1" (a.k.a. "false" and "true" or "low voltage" and "high voltage"), is sufficient for a complete set of computational primitives.

                                    Besides those mentioned above, the main remaining variant for a complete set of computational primitives consists of an analog (possibly weighted) adder and an analog comparator, which had been used in the so-called RTL circuits (resistor-transistor logic) and which also corresponds to perceptrons. RTL had been used in some early integrated circuits, before being replaced by DTL and TTL circuits (which are based on minimum and complement functions).

                                    In hardware, e.g. in RTL circuits, a combined analog adder+comparator can be made with a single high-gain amplifying device, together with a set of weighting resistors and a bias resistor. RTL circuits can implement complex logic with fewer devices (e.g. they can implement a neural network in the analog domain), but they were replaced during the sixties of the past century with DTL, then TTL, because those were faster (in RTL, the resistors limit the charging currents for input capacitors and parasitic capacitors, which slows down the logical transitions) and the fact that they needed more devices was not important, due to the quick increase in circuit density.

                                      • lioeters

                                        today at 10:29 AM

                                        This reminds me of a book, whose title I forget, about creating a compact set of knowledge that will enable us to "rebuild civilization from scratch" in case of a future/fictional post-collapse scenario. I sometimes wonder, given the immense complexity and global supply-chain dependencies of our computing stack, whether we could bootstrap from first principles something of equivalent power and expressivity, but orders of magnitude simpler.

                                        There's a wide variety of computational primitives, including lambda calculus, combinators, cellular automata, rewriting systems. Perhaps some are more practical to implement in hardware, particularly the kind of DIY electronics or analog machines that can also be put together from scratch. It might look like a whole building of mechanical switches, powered by a water wheel ("watermill"), for example.

                                          • coldtea

                                            today at 2:08 PM

                                            I think if civilization collapses they'll have other priorities, and very little benefit to get from the effort required for such simpler computing, for a good while...

                                              • lioeters

                                                today at 2:30 PM

                                                I suppose people will have more important things to work on, like growing food and fighting off roving bands of bandits, than building primitive analog calculators/computers.

                                    • gpderetta

                                      today at 9:12 AM

                                      it not really an if statement here in a perceptron though. It is more akin a logic gate.

                                      A transistor (driven to saturation) is a much better model.

                                      • utopiah

                                        today at 8:10 AM

                                        You might enjoy playing with Turing Tumble.

                                    • ankit84

                                      today at 3:42 AM

                                      I learnt a lot today from the interactive demo. You have the best clarity and right skill to educate

                                        • DevarshRanpara

                                          today at 3:47 AM

                                          Thank you, I will try to make more demo on other concepts.

                                      • ninalanyon

                                        today at 2:28 PM

                                        Is this something that could be scaled up and used, for instance, to recognize features in images?

                                        Or to put it another way are there any local only tools that can be trained on my own set of images to automatically tag new images? Tools that do not already have built in classes of image.

                                        I take a lot of photographs and it would be handy to reduce the drudgery of tagging them so to say broadly what the subject was so that they are easier to find later.

                                        • trekhleb

                                          today at 3:32 AM

                                          Nice and minimalistic

                                          I played with similar approach in JavaScript and built a NanoNeuron https://github.com/trekhleb/nano-neuron (it is more verbose than Python though)

                                          • vain

                                            today at 5:37 PM

                                            Shameless plug of my own interactive version of this (ai assisted, but probably not slop) https://sourceobscure.com/perceptron/

                                            • virajk_31

                                              today at 11:07 AM

                                              Not a ML expert, but ML tutorials shall start with something like this... Good read. Thanks.

                                              • techteach00

                                                today at 3:18 PM

                                                Is this too high level for my 8th grade comp sci class?

                                                • Bimos

                                                  today at 4:41 AM

                                                  > A perceptron *is* the smallest brain you can build.

                                                  > In 1958, a researcher named Frank Rosenblatt built a machine *he called* the perceptron.

                                                  > It was *inspired* by a single brain cell, a neuron.

                                                    • Lerc

                                                      today at 11:58 AM

                                                      I did a lecture once which included a 5 minute whirlwind tour of neural net history.

                                                      I included a remark about how time travellers would find Rosenblatt a better target than Miles Dyson.

                                                      I was never quite sure on how close, or over, the line that was on appropriateness. It was definitely thought provoking.

                                                      • lmf4lol

                                                        today at 6:44 AM

                                                        Yes . But at least the post seems to be written by OP himself!

                                                        and its an a great learning resource - which is arguably more important :-)

                                                    • esafak

                                                      today at 12:53 AM

                                                      If you want to learn the fundamentals of ML I recommend a book, such as Deep Learning: Foundations and Concepts by Chris Bishop. If you insist on staying online, one option is https://course.fast.ai/

                                                      If you don't know ML I don't think you're going to learn much through ad hoc demos.

                                                        • mysterydip

                                                          today at 1:51 AM

                                                          Checked out the book on your recommendation, and they even have a free online option on their site! Very generous: https://www.bishopbook.com/

                                                          • llimllib

                                                            today at 1:44 PM

                                                            I remember sitting in the senior study lounge reading the previous Bishop book and implementing the perceptron from it, 22 years ago: https://github.com/llimllib/personal_code/blob/945b017b2915c...

                                                            (before numarray and numpy merged!)

                                                            • rishabhaiover

                                                              today at 1:13 AM

                                                              This book equipped me with the right intuition and tools to visualize machine learning. I wish I was smart enough to hold it all together.

                                                                • andai

                                                                  today at 1:29 AM

                                                                  >I wish I was smart enough to hold it all together.

                                                                  I used to have a wife, but they took her in the divorce!

                                                                  The human mind isn't very good at correlating its contents[0]. You can "know" something for years without realizing its implications.

                                                                  The human mind traverses its knowledge like a man with a small flashlight in total darkness. Our beam of attention is small and narrow, so you need to put the right things in it, or the magic doesn't happen.

                                                                  This has important implications for learning. I don't know what they are though.

                                                                  Probably something like, "you can know something without knowing what it means." You haven't connected it to the things it's supposed to be connected to yet. I don't know how to fix that though. (Something involving the Feynman technique, maybe?)

                                                                  [0] H.P. Lovecraft quote - https://www.goodreads.com/quotes/193944-the-most-merciful-th...

                                                              • stuxnet79

                                                                today at 1:52 AM

                                                                I didn't know Bishop had released a new textbook. I will have to take a look at it. I wasn't the biggest fan of his Pattern Recognition book as I found it overly dense. I much preferred the Murphy and Alpaydin books.

                                                                EDIT: His son is co-author?

                                                                  • zxexz

                                                                    today at 5:15 AM

                                                                    I still find his pattern recognition book useful and informative. It may be dense, but some of us consider that a positive for 'reference' literature. That book was one of very few that still holds up well fr when it was published - truly in on of the last "dark ages" of ML.

                                                                    I think those down voting you are perhaps overly eager. I upvoted. Grab "Deep Learning" - you'll find it useful, imteresting, and likely less 'dense' in the negative sense!

                                                                      • stuxnet79

                                                                        today at 5:33 AM

                                                                        Appreciate your comment. I skimmed the online version and it covers all the 2010s era developments all the way to Transformers which is enough to earn it a spot on my bookshelf.

                                                                        > Grab "Deep Learning" - you'll find it useful, imteresting, and likely less 'dense' in the negative sense!

                                                                        Absolutely! I just ordered it and it's enroute :)

                                                                • DevarshRanpara

                                                                  today at 3:38 AM

                                                                  This fast AI course looks soo good man! Definitely I will start learning soon. Thank you!

                                                              • opem

                                                                today at 10:32 AM

                                                                I have still so many questions left, but regardless of that it was a great read. Thanks for sharing!

                                                                • warengonzaga

                                                                  today at 9:03 AM

                                                                  This is amazing insight, thanks for sharing!

                                                                    • CyberDildonics

                                                                      today at 1:05 PM

                                                                      This is your first comment in six years, what is amazing about it?

                                                                  • today at 12:28 AM

                                                                    • charcircuit

                                                                      today at 3:59 AM

                                                                      I can build a smaller brain.

                                                                      f(x) = 0.

                                                                        • rippeltippel

                                                                          today at 5:30 AM

                                                                          That's great, now make it learn something :)

                                                                            • jeffwass

                                                                              today at 7:40 AM

                                                                              It's the simplest AI nihilist!

                                                                          • lioeters

                                                                            today at 10:11 AM

                                                                            I take your brain, add a couple more rules, and presto! It can perform any computation.

                                                                                Ix = x
                                                                                Kxy = x
                                                                                Sxyz = xz(yz)

                                                                            • opem

                                                                              today at 10:35 AM

                                                                              This brain is interesting. Basically you get a no for everything you ask, right?

                                                                              • moffkalast

                                                                                today at 9:40 AM

                                                                                Yeah, this is small brain time.

                                                                            • sspoisk

                                                                              today at 3:05 PM

                                                                              [flagged]

                                                                              • today at 9:43 AM

                                                                                • infoinlet

                                                                                  today at 10:13 AM

                                                                                  [flagged]

                                                                                  • knightops_dev

                                                                                    today at 2:19 AM

                                                                                    [flagged]

                                                                                    • jkwang

                                                                                      today at 3:36 AM

                                                                                      [flagged]

                                                                                      • haeseong

                                                                                        today at 4:12 AM

                                                                                        [dead]

                                                                                        • EvanXue

                                                                                          today at 3:05 AM

                                                                                          [flagged]

                                                                                          • karinatran

                                                                                            today at 2:56 AM

                                                                                            [dead]

                                                                                            • hbwang2076

                                                                                              today at 1:29 AM

                                                                                              [dead]

                                                                                              • b33j0r

                                                                                                today at 1:19 AM

                                                                                                Okay, it’s conscious. But can it run doom? I rest my case.

                                                                                                  • andblac

                                                                                                    today at 2:10 PM

                                                                                                    I know you're joking, but if you really wanted to, you can if you have a network of these, since you can build NAND gates from perceptrons. If you have NAND gates, then you can build any other gates from these and then you can build a computer [1].

                                                                                                    [1] https://www.nand2tetris.org/

                                                                                                    • Waterluvian

                                                                                                      today at 1:22 AM

                                                                                                      Can you run Doom? Let’s find out!