If you had to pick one idea, the most important idea in all of computer science, what would it be? John Ousterhout asked Donald Knuth this question. Knuth answered “layers of abstraction”. Ousterhout says that is almost right. His own answer is problem decomposition: how to take a complicated problem and chop it into pieces that you can build independently (watch A Philosophy of Software Design by John Ousterhout). Layers of abstraction are one of the best tools for that decomposition. A layer hides the complexity of the implementation below it and gives the layer above a clear, easy-to-use interface.
One excellent example is the layers of computer architecture (watch Computer Architecture - Levels of Transformation by Onur Mutlu). The instruction set architecture (ISA) is the interface between software and hardware. Another example is the layers of computer networks (watch Computer Networks - Layering by Jim Kurose). The TCP/IP layer names below come from RFC 1122.
The Map
Jess Martin wrote a note about software layers, Pace Layers for Software. He sees a software system as a layered structure, where each layer has a different capability. Building on his framework, I made some adjustments and additions and redrew the map below. It may look trivial to experienced software engineers, but it is a useful way to place any piece of software you meet.
The sections below walk the map from top to bottom. For each layer, ask two questions: what does it hide, and what interface does it give to the layer above?
Extending an Application: Plugins and Scripts
The top layer extends an application that already exists. It hides nothing. It adds behaviour on top of a host.
A plugin or extension is an add-on installed into a host program. It runs inside the host, through the API that the host exposes. There is no accepted difference between the two words. Wikipedia treats plug-in, add-on and extension as synonyms. What matters is how much the host’s API allows. For example, VS Code extensions cannot access the DOM of the VS Code UI, while Chrome extensions can change web pages and customize the browser’s user interface.
Examples include Microsoft Office Add-ins, Anki Add-ons, Obsidian Plugins, Zoom Apps, PostgreSQL Extensions, Chrome Extensions, VS Code Extensions and Gradle Plugins.
A script extends a host through a language instead of a binary API. Some hosts embed a scripting language, such as Python in Blender or Lua in Neovim. Others accept a DSL (domain-specific language), a small configuration language such as a Dockerfile or a GitHub Actions workflow. Scripts sit beside the application in the map because they can drive it from outside as well as from inside.
Application
The application is the layer that users see. It hides everything below and gives users a product. Most software engineers work here, on desktop, web and mobile applications, backend services, and machine learning and data analysis applications.
Application Infrastructure
Application infrastructure hides the problems that every application has. It gives application code ready-made parts. It has three kinds: libraries, frameworks and platforms.
Library
A library is a collection of reusable code that you call, such as a language’s standard library or a third-party package. Each language has a registry where libraries are published:
| Language | Registry |
|---|---|
| JavaScript | npm |
| Python | PyPI |
| Java | Maven Central |
| .NET | NuGet |
| Ruby | RubyGems |
| Rust | crates.io |
| Go | Go Modules |
| PHP | Packagist |
Framework
A framework also gives ready-made parts, but it is defined by inversion of control (IoC). Martin Fowler puts it this way: “Inversion of Control is a key part of what makes a framework different to a library.” You call a library. A framework calls your code.
Examples are frontend frameworks such as React, Angular and Vue; backend frameworks such as Spring, Django, Express and Flask; microservice frameworks such as Spring Cloud and Dapr; and machine learning frameworks such as TensorFlow and PyTorch.
Platform
A platform is software that your application runs on or talks to. It hides a whole subsystem, such as storage, search or payments, behind a network API. You either install and operate it yourself, or call a provider’s API and let them operate it.
Programming Language
A programming language hides the machine and gives you a way to write down a program. Higher-level languages hide more details of memory management, concurrency and system calls, while lower-level languages expose more control over them. The choice of language decides how much of the system’s complexity you see. Most languages also ship with a standard library for data structures, file I/O, networking and concurrency, so that you don’t have to call the operating system directly.
Runtime
A runtime hides the operating system and the CPU, and gives a language one machine to target everywhere. Many languages do not run directly on the operating system. They run on a runtime or virtual machine. The JVM specification says the Java Virtual Machine “knows nothing of the Java programming language, only of a particular binary format, the class file format”. That is why Kotlin, Scala and Clojure can share it. WebAssembly is “a binary instruction format for a stack-based virtual machine”, which lets C, C++ and Rust run in the browser.
Operating System
The operating system hides the hardware and gives programs processes, files and sockets. It handles process management, memory management, file systems, device drivers and networking. System calls are its interface to languages and applications.
On desktops and servers, the main families are Linux (Ubuntu, Debian, Arch Linux), Windows, macOS and the BSDs (FreeBSD, OpenBSD). On phones, they are Android, which is built on the Linux kernel, and iOS.
Container and Virtual Machine
Two layers wrap around the operating system. Above it, a container hides the differences between machines and gives an application the same environment everywhere. Docker explains that containers “virtualize the operating system instead of hardware”. Below it, a virtual machine hides the physical server and gives a whole operating system virtual hardware. Cloud IaaS sells this layer. NIST defines IaaS as provisioning “processing, storage, networks, and other fundamental computing resources”, where the consumer can run “operating systems and applications”.
Hardware
The map ends where the first diagram began. Hardware is layered too: the instruction set architecture (ISA) hides the microarchitecture, logic and devices, and gives software a fixed set of instructions. The ISA is one of the slowest layers in computing, which is why software built for x86 decades ago still runs today. It also shows that every layer leaks. The Spectre attacks of 2018 used speculative execution, a detail of the microarchitecture, to read data that the ISA’s rules say a program cannot read.
Why Layers: Fast and Slow
Jess Martin’s note borrows its name from Stewart Brand’s pace layers. In The Clock of the Long Now, Brand describes a civilization as layers that change at different speeds: fashion, commerce, infrastructure, governance, culture and nature. His summary is: “Fast learns, slow remembers. Fast proposes, slow disposes.”
Software works the same way. Scripts and applications change every week. They are where people try new ideas. Programming languages and protocols change over years, and they constrain everything above them. An idea moves down the stack only after it proves itself. A pattern repeated in many applications becomes a library. A library that everyone uses becomes part of a platform or a language.
This is also why a good layer is “deep”, in Ousterhout’s word: a simple interface that hides a lot of functionality. And it is why every layer leaks a little. In Joel Spolsky’s Law of Leaky Abstractions: “All non-trivial abstractions, to some degree, are leaky.” When a lower layer fails or runs slowly, you still have to understand it.
The AI Layer
Everything above was written for programs that people write. Large language models change that, and the clearest way to see how is to use the same layered thinking.
Software 1.0, 2.0 and 3.0
Andrej Karpathy describes three kinds of software. In Software 2.0, the program is the weights of a neural network, learned from data instead of written by people. In his talk Software Is Changing (Again), he calls prompts in English Software 3.0: the program is written in natural language and run by an LLM.
The LLM as an Operating System
In September 2023, Karpathy wrote that “a more complete picture is emerging of LLMs not as a chatbot, but the kernel process of a new Operating System.” Looking at LLMs as chatbots, he added, “is the same as looking at early computers as calculators.” Two months later he drew the picture.
The LLM is the CPU. The context window is the RAM. In his Intro to Large Language Models talk, he calls it “your finite precious resource of your working memory”, and describes the LLM “trying to page relevant information in and out of its context window”, the way an operating system pages memory. Files and embeddings are the disk. The browser is the network card. Calculators, Python interpreters and terminals, the Software 1.0 tools, become programs that the new CPU can run.
The analogy goes further than hardware. Here is the mapping, layer by layer. The rows for drivers and system calls are my own extension; the rest come from Karpathy’s talks.
Three rows deserve a sentence each:
- The ecosystem. Karpathy compares Windows and macOS to closed models, and Linux to open-weight models such as Llama. Mark Zuckerberg made the same argument for Llama in Open Source AI Is the Path Forward, comparing it to Linux beating the closed versions of Unix.
- The era. In Software Is Changing (Again), Karpathy says we are in “this 1960s-ish era” of computing. Models are expensive and centralized in the cloud, and we share them by time-sharing, each of us “a dimension of the batch”. The personal computing revolution for LLMs has not happened yet. Chat is the terminal; nobody has invented the GUI.
- The drivers. An operating system talks to devices through drivers with a standard interface. The Model Context Protocol (MCP) plays that role for tools and data: “a universal, open standard for connecting AI systems with data sources, replacing fragmented integrations with a single protocol.” Google’s A2A (Agent2Agent) standardizes the layer beside it, so that agents from different vendors can work together.
Where the AI Layer Sits in the Map
The new pieces fit the old map more neatly than it first seems.
- An AI application is still an application.
- A model behind an API is a serverless platform. You call it and pay per million tokens. An open-weight model is a platform you install and operate yourself.
- An agent framework is a framework in Fowler’s exact sense. You write the tools, and the model’s loop decides when to call them. Inversion of control returns, except that the thing calling your code is now a model.
- MCP and A2A are protocols, the slowest layer in Jess Martin’s list. That is why they matter more than any single agent built on top of them.
The Industry Stack
The same layers show up as an industry, and each layer fights for the value.
In January 2023, a16z asked Who Owns the Generative AI Platform? Their answer: “infrastructure vendors are likely the biggest winners in this market so far, capturing the majority of dollars flowing through the stack”, and “there don’t appear, today, to be any systemic moats in generative AI.” In 2024, David Cahn at Sequoia turned that into AI’s $600B Question: how much revenue the layers above must earn to pay for the chips and data centers below.
The open question is whether models become a commodity layer, like CPUs. In 2024, Ben Thompson argued that “the most value will be derived from building platforms that treat models like processors”. Later, in Agents Over Bubbles, he argued that “the integration between model and harness is where true agent differentiation is found.” Karpathy’s own analogies cover both sides. Model labs look like utilities, with large CAPEX to build the grid and metered OPEX to serve it. They also look like chip fabs, with deep technology trees and secrets concentrated in a few labs.
Pace Layers Under Pressure
Drew Breunig applied pace layers to AI in Understanding the Dynamics of the AI Ecosystem with Pace Layers. His stack runs from prompts, which change in days, through models, which change in months, and chips, which change in years, down to energy and organic human-made data, which change over decades.
His point is that AI breaks Brand’s rule. Normally the slow layers set the pace. Here, “massive investment” is “forcing lower layers to move faster than normal”: chips, data centers and energy are being rebuilt at the speed of software.
The rule still works in the other direction. Ideas still move down the stack after they prove themselves. Prompt tricks become model features. Hand-built tools become standard protocols. Rich Sutton’s The Bitter Lesson predicts this: knowledge that people build into AI systems “always helps in the short term”, but “in the long run it plateaus”, and general methods that scale with computation win.
And the new layer leaks more than any layer before it. Karpathy calls LLM abilities “jagged intelligence”: a model that solves hard problems can still claim that 9.11 is greater than 9.9. Spolsky’s law applies again. To build on this layer, you have to understand what it hides.