The computers that run AI — and how much power they consume

The hardware behind AI

When you ask a question to ChatGPT or Claude, your message travels to a massive data center somewhere in the world. There, racks of specialized computers perform thousands of calculations within seconds to generate your answer. Those computers are fundamentally different from the laptop or smartphone you’re reading this on.

The core of modern AI hardware is the GPU (Graphics Processing Unit). Originally designed for video games, GPUs turned out to be ideally suited for the kind of matrix calculations that machine learning requires. Where a CPU (the regular processor in your computer) performs a small number of tasks quickly, a GPU performs thousands of tasks simultaneously but slightly slower. For AI training, that parallel power is decisive.

The GPU: heart of the AI revolution

Illustration created with Canva AI

NVIDIA dominates the AI GPU market with more than 80% market share. The most in-demand chips are:

• NVIDIA H100 — the current standard for AI training. Costs approximately $30,000 per unit. Consumes 700 watts. A single H100 delivers more computing power than a supercomputer from ten years ago.
• NVIDIA H200 — successor to the H100 with 141 GB HBM3e memory. Better performance with large models.
• NVIDIA Blackwell B200 — the latest generation (2025). Consumes up to 1,000 watts per chip but delivers 2.5x more performance than the H100. Wait times of more than a year at major cloud providers.
• Google TPU v5 — Google’s own AI chip, optimized for TensorFlow and its own models. Not for sale, only available via Google Cloud.
• AMD Instinct MI300X — the most serious alternative to NVIDIA. More memory (192 GB) but less ecosystem support.

How many GPUs does an AI model need?

The scale is staggering:

• GPT-3 (2020) was trained on approximately 10,000 NVIDIA V100 GPUs over several weeks.
• GPT-4 (2023) — exact details are secret, but estimates put it at tens of thousands of A100s.
• Llama 3 405B (Meta, 2024) was trained on more than 16,000 H100s over 30 days.
• Inference (daily use) for ChatGPT requires an estimated 30,000–50,000 GPUs permanently active at current user numbers.

Data centers: the factories of the AI economy

All those GPUs live in specialized data centers — buildings sometimes covering several hectares, filled with server racks. The world’s largest AI data centers are being built in the US (Virginia, Texas, Arizona), Europe (Ireland, Netherlands) and Asia.

A modern AI data center requires:

• Megawatts of power connection — a single large data center can consume 500 MW to 1 GW. For comparison: an average coal power plant produces about 500 MW.
• Cooling — half the energy consumption goes toward cooling. GPUs produce extreme heat; without cooling they would melt within minutes.
• Redundant power supply — emergency generators, UPS systems and dual grid connections.
• Network — hundreds of kilometers of fiber optic cable internally, and connections to the internet backbone.

The power consumption of AI

How large is the energy consumption of AI exactly? The numbers are shocking:

• Training GPT-4: estimated at 50–100 GWh — comparable to the annual consumption of 5,000–10,000 average US households.
• One ChatGPT conversation: estimated at 0.001–0.01 kWh per message — 10x more than a Google search.
• Microsoft Azure AI services: Microsoft’s total data center usage increased by 30% in 2023 following the ChatGPT investment.
• Google: reported in 2024 that AI caused its CO₂ emissions to rise by 48% compared to 2019, despite large investments in renewable energy.

The IEA (International Energy Agency) predicted in 2024 that data centers would consume as much power as all of Japan by 2026 — and AI is the fastest-growing component.

Water consumption: the forgotten problem

In addition to electricity, data centers consume enormous quantities of water for cooling. Servers are cooled via water circuits that remove heat through cooling towers. Microsoft’s data centers already consumed 6.4 million cubic meters of water in 2022 — comparable to the annual consumption of 50,000 households.

This is a growing problem in water-scarce regions. Arizona, where many large AI data centers are being built, is suffering from severe drought due to climate change.

Efficiency improvements

The industry is not passive. Serious efforts are being made to reduce energy consumption:

• Smaller models — Models like Mistral 7B, Phi-4 and Llama 3.2 1B offer surprisingly good performance at a fraction of the computing power of the large models.
• Quantization — compressing models from 32-bit to 4-bit or 8-bit precision, allowing them to run on less powerful hardware.
• More efficient architectures — Mixture of Experts (MoE) activates only part of the network per query, saving energy.
• On-device AI — small models on smartphones and laptops eliminate the server round-trip for simple tasks.
• Green energy — Google, Microsoft and Amazon commit to 100% renewable energy for their data centers, though promises and reality often diverge.

The geopolitics of AI hardware

The dependence on NVIDIA GPUs has geopolitical consequences. The US has imposed export restrictions preventing advanced chips from being delivered to China. This has forced China to develop its own chips (Huawei Ascend, Cambricon) and has fundamentally disrupted the global AI hardware supply chain.

Taiwan plays a crucial role: TSMC (Taiwan Semiconductor Manufacturing Company) produces virtually all advanced AI chips, including those of NVIDIA. The geopolitical tension around Taiwan is therefore directly connected to the future of AI.

Conclusion

AI runs on a physical infrastructure of unprecedented scale and complexity — and the energy hunger is growing faster than the efficiency improvements. Whether AI’s energy consumption is sustainable is not a technical but a political and social choice: which benefits justify which costs?