Super Fact 148 : The number of parameters in AI systems has increased by tens of billions of times since the 1950’s. Transistors per Microchip have increased by more than 50 million Times in 50 Years. The size of AI training data sets increased trillions of times. The number of computations for training has increased hundreds of sextillion times. The number of parameters, transistors per microchips, and computations for training are growing exponentially. This is one reason many AI experts believe there is a real chance that human-level artificial intelligence will be developed within the next decades.

The Number of Parameters above are plotted on a logarithmic axis. This means that exponential growth becomes a straight line and each tick-step is a 100 doubling. The regression lines show a rise in parameters since 2010, driven by the success of deep learning methods that leverage neural networks and massive datasets.
Theseus an AI system from 1950 had 40 parameters. The perceptron an early neural network (Artificial Intelligence) built in 1960 had 1,000 parameters. Deep Blue, the AI supercomputer that beat chess world champion Garry Kasparov in 1997 had 8,000 parameters. It could evaluate roughly 200 million chess positions per second, which is hard to beat. In 2012 AlexNet a neural network with multiple hidden layers, so called deep learning, had 60 million parameters. Switch from built in 2021 has 1.57 trillion parameters and GPT-4 (recent ChatGPT) has 1.8 trillion parameters. The number of parameters from Theseus to GPT-4 increased 45 billion times.

The exponential growth in the number of computations used for training is even more impressive. The exponential growth took a sharp upturn in 2010. As can be seen in the graph below the computation used to train notable AI systems has doubled every 6 months since 2010. This means a doubling by million times in 10 years, a doubling by a trillion times in 20 years, a doubling by a quintillion times in 30 years, a doubling by a septillion times in 40 years, and a doubling of nonillion times in 50 years. Naturally, we don’t know if the rapid exponential growth will continue but also don’t forget Moore’s Law. You can read more about the rapid exponential growth in training computation here.

As mentioned, the rapid exponential growth in parameters, transistors per chip, computations, etc., is one reason many AI experts believe there is a real chance that human-level artificial intelligence will be developed within the next decades, and some believe that it will exist much sooner. You can explore more on this topic here.
I consider this a super fact because it is important information and the exponential growth of parameters in AI, computations and the size of training sets, is shocking and should surprise many of us.
What Are AI Parameters ?
Parameters are variables in an AI system whose values are adjusted to establish how input data gets transformed into the desired output; for example, the connection weights in an artificial network and the biases associated with each neuron. Biases are offset values added to the weighted sum at each neuron to help the model fit the data better. Below is a very simple illustration of a small neural network (drawn by me) with three hidden layers (multiple hidden layers is called deep learning), an input and output layer. There are 19 neurons and 64 connections with weights yielding a total of 83 parameters.

I should add that in other types of AI systems parameters might be something different. For example, at work I used an AI algorithm called C5, a type of decision tree algorithm, and in that algorithm the tree parameters controlled how the model split the data. You listed most of the parameters you wanted in a text file.
The human brain has an estimated 86 billion neurons (plus/minus around 15 billion) The human brain also has an estimated 100 trillion to 1,000 trillion synaptic connections. That is a lot more than the estimated 1.8 trillion parameters of GPT 4. However, keep in mind 1.8 trillion will pretty soon be 100’s of trillions, or quadrillions of parameters and then quintillions of parameters, then sextillions of parameters, etc.
The Rummelhart Backpropagation Algorithm for Training
There are many kinds of neural networks, but modern Large Language Models are feedforward neural networks using backpropagation for training, specifically modernized versions of the 1986 Rumelhart, Hinton, and Williams algorithm. In 1985/1986, Geoffrey Hinton, David Rumelhart and Ronald J. Williams presented an algorithm applied to a neural network featuring a hidden layer that was very successful. It was effective and guaranteed to learn patterns that were possible to learn. It set off a revolution in Neural Networks. Geoffrey Hinton received the Nobel Prize in physics in 2024 for his inventions enabling machine learning with artificial neural networks. David Rumelhart and Ronald J. Williams were dead and could therefore not receive the Nobel prize. You can read more about backpropagation here and read the original Rumelhart backpropagation paper here..
I can add that the next year, in 1987, when I was a college student, I used this algorithm on a neural network featuring a hidden layer to do simple OCR (optical character recognition). Note that a computer reading an image with a letter is very different from someone typing it on a keyboard. In the case of the image, you must use OCR, a complicated and smart algorithm for the computer to know which letter it is.
Other Super Facts Related to Artificial Intelligence

- The Nobel Prize in Physics and Neural Networks
- Artificial Intelligence is Not New
- Large Language Models is just One Branch of Artificial Intelligence
- The Dark Side of Artificial Intelligence
- Transistors per Microchip Increased by More than 50 million Times in 50 Years
- AI is Already More Capable Than Humans in Many Areas
- A Typical AI Query Use Less Energy Than Using a Laptop One Minute
- The Future of AI is Superintelligence
- The Unfolding Driverless Car Revolution
To see the other Super Facts click here
I would also like to mention that today November 15 is “I Love to Write Day”