AI applications such as large language models (LLMs) have become an integral part of our everyday lives. The required computing, storage and transmission capacities are provided by data centers that consume vast amounts of energy. In Germany alone this amounted to around 16 billion kWh in 2020, or around 1% of the country’s total energy consumption. For 2025, this figure is expected to increase to 22 billion kWh.
New method is 100 times faster with comparable accuracy
The arrival of more complex AI applications in the coming years will substantially increase the demands on data center capacity. These applications will use up huge amounts of energy for the training of neural networks. To counteract this trend, researchers have developed a training method that is 100 times faster while achieving accuracy comparable to existing procedures. This will significantly reduce the energy consumption for training.
The functioning of neural networks, which are used in AI for such tasks as image recognition or language processing, is inspired by way the human brain works. These networks consist of interconnected nodes called artificial neurons. The input signals are weighted with certain parameters and then summed up. If a defined threshold is exceeded, the signal is passed on to the next node. To train the network, the initial selection of parameter values is usually randomized, for example using a normal distribution. The values are then incrementally adjusted to gradually improve the network predictions. Because of the many iterations required, this training is extremely demanding and consumes a lot of electricity.
Parameters selected according to probabilities
Felix Dietrich, a professor of Physics-enhanced Machine Learning, and his team have developed a new method. Instead of iteratively determining the parameters between the nodes, their approach uses probabilities. Their probabilistic method is based on the targeted use of values at critical locations in the training data where large and rapid changes in values are taking place. The objective of the current study is to use this approach to acquire energy-conserving dynamic systems from the data. Such systems change over the course of time in accordance with certain rules and are found in climate models and in financial markets, for example.
“Our method makes it possible to determine the required parameters with minimal computing power. This can make the training of neural networks much faster and, as a result, more energy efficient,” says Felix Dietrich. “In addition, we have seen that the accuracy of the new method is comparable to that of iteratively trained networks.”
Rahma, Atamert, Chinmay Datar, and Felix Dietrich, “Training Hamiltonian Neural Networks without Backpropagation”, 2024. Machine Learning and the Physical Sciences Workshop at the 38th conference on Neural Information Processing Systems (NeurIPS) https://neurips.cc/virtual/2024/99994
Bolager, Erik L, Iryna Burak, Chinmay Datar, Qing Sun, and Felix Dietrich. 2023. “Sampling Weights of Deep Neural Networks.” In Advances in Neural Information Processing Systems, 36:63075–116. Curran Associates, Inc. https://proceedings.neurips.cc/paper_files/paper/2023/hash/c7201deff8d507a8fe2e86d34094e154-Abstract-Conference.html
If you found this piece useful, please consider supporting our work with a small, one-time or monthly donation. Your contribution enables us to continue bringing you accurate, thought-provoking science and medical news that you can trust. Independent reporting takes time, effort, and resources, and your support makes it possible for us to keep exploring the stories that matter to you. Together, we can ensure that important discoveries and developments reach the people who need them most.
