Computers are constantly moving data around. A CPU needs information from memory, a website needs information from a database, a GPU needs data to process, and your browser needs to download files from the internet. The problem is that fetching data is one of the most time-consuming operations in computer systems.
Now imagine doing those data-fetching operations thousands, millions, or even trillions of times.
This is where caching comes in. The basic idea behind caching is surprisingly simple: if you are going to need something again, keep it somewhere you can access quickly.
Caching is one of those ideas that appears almost everywhere in computing. There are tiny caches inside CPUs, caches inside GPUs, caches in web browsers, caches in CDNs, and software systems such as Redis that can keep frequently requested data in memory. The technology changes, but the basic principle remains the same.
Caching Is Like Military Logistics
Imagine a military operation where units are spread across a large area. Those units need ammunition, fuel, food, medical supplies, and equipment. Somewhere far behind the front lines is a massive warehouse containing huge quantities of supplies.
If a soldier needs something, you could send a vehicle all the way back to the warehouse, collect the supplies, and bring them back. That works, but it is obviously inefficient if the same supplies are needed repeatedly.
Instead, the military can create smaller supply depots closer to the units. The main warehouse still contains the majority of the supplies, but commonly needed items can be stored at regional depots and even smaller forward supply bases.
Now, when a unit needs something, it doesn't have to travel all the way back to the main warehouse. It can get the supplies from somewhere much closer.
This is essentially what caching does.
The original source of the data is like the large warehouse. A cache is like a supply depot positioned closer to the people or machines that need the supplies. If the requested data is already in the cache, we call that a cache hit. If it isn't there, the system has to go back to the original source. We call that a cache miss.
CPU Caches: L1, L2 and L3
A modern CPU can process instructions incredibly quickly, but the computer's main memory, or RAM, is much slower than the CPU. If the CPU had to wait for RAM every time it needed a piece of data, a significant amount of its potential processing power would be wasted waiting around.
So CPUs have caches.
Most modern CPUs have several levels, commonly referred to as L1, L2, and L3. Chat, "L" means level. But if it were to mean "Loser", L3 woud take the biggest "L"...I digress.
L1 is the smallest and fastest cache, sitting extremely close to the CPU core and designed to provide data with very little delay, like supplies kept directly with the unit, where there isn't much room, but when something is needed, it is immediately available. L2 is larger but somewhat slower, more like a nearby supply depot that can hold more than the unit itself, though getting something from it takes a little longer. L3 is larger again and slower than both L1 and L2, and on many modern CPUs, it is shared between multiple cores, functioning more like a larger regional depot serving several units at once.
Eventually, if the data isn't found in any of these caches, the CPU has to go further out to main memory.
So you get a hierarchy that roughly looks like this:
L1 → L2 → L3 → RAM
The further away we move from the CPU, the more expensive accessing data generally becomes. This same pattern continues beyond RAM. Storage such as SSDs is slower to access than RAM, hard drives are slower still, and fetching something across a network can introduce even more latency.
It is the same trade-off the military faces when deciding where to position supplies. A forward base can respond quickly, but it cannot realistically store everything. A massive warehouse can hold enormous quantities, but it is much farther away.
LRU: Deciding What Stays
Another problem with caches is that they have limited space. Eventually, they will fill up. When new data needs to be stored, something else has to be removed. The question is, what should go?
One common approach is called LRU, or Least Recently Used. If the cache is full, remove the thing that hasn't been used for the longest time.
Imagine a forward supply base with limited storage. If there is equipment sitting in the warehouse that hasn't been requested in months, while another type of equipment is being requested every day, it makes sense to make room by removing the less recently used item.
LRU applies a similar principle to software caches. Recently used data is considered more likely to be useful again, while data that hasn't been used for a long time becomes a candidate for removal.
LRU is a strategy for managing the contents of a cache, not another level of cache like L1 or L2. It answers the question of what should be removed when the cache needs space.
GPUs can Cache Out Too!
Much like CPUs, GPUs have their own memory hierarchies and caching mechanisms.
Unlike CPUs, GPUs are designed to process enormous amounts of data, often in parallel. Moving that data from slower memory every time it is needed can become a major bottleneck, so GPUs use faster forms of memory and caches to keep useful data closer to the processing hardware.
The exact architecture can differ between GPU designs, but the basic concept of caching still applies.
Once again, we have our military supply chain. The processing units are the soldiers, the data is the equipment they need, slower memory is the distant warehouse, and caches are the smaller supply depots positioned closer to the action.
The goal is still the same: reduce unnecessary trips.
Caching at the Application Level
So far, we've been talking about the hard stuff, aka hardware. But caching is just as useful in the soft stuff, aka software.
An application may need information from a database. A database query might involve searching through data, reading from storage, performing calculations, and returning a result. Doing that once might not be a problem.
But what if thousands of users ask for the exact same information?
It would be wasteful for the application to perform the same expensive operation over and over if the answer hasn't changed.
This is where in-memory data stores become useful. If you're a techie, you might know of Redis. An application can put frequently requested information into an in-memory data store so that future requests can retrieve it quickly instead of repeatedly querying the database or performing the same expensive calculation.
If thousands of units are asking for the same supplies, it makes sense to keep those supplies nearby rather than sending every request back to the main warehouse.
The same principle can apply to application data. A website might repeatedly need the same information, such as a frequently viewed product, configuration, leaderboard, or other piece of data. Instead of calculating or retrieving it from the original source every time, the application can temporarily keep the result in a cache. The next request can simply retrieve the cached version.
The Web Can Cache Out Too.
Caching doesn't stop at the CPU or application server. Why should it?
Your browser has a cache. When you visit a website, your browser may store files such as images, stylesheets, and JavaScript locally. When you visit the same website again, it may not need to download everything from the server again because some of those files are already on your computer.
Then there are CDNs, or Content Delivery Networks. A CDN can store copies of content at servers distributed around the world. Instead of every user having to retrieve content from the original server, the content can often be delivered from a server that is geographically closer.
The military analogy continues to work surprisingly well. Instead of having one enormous warehouse serve every unit in the entire area of operations, you establish multiple regional depots. Each one keeps commonly requested supplies nearby.
The original warehouse still exists. The regional depots simply reduce the distance that most requests need to travel.
This Is Almost Too Good to Be True
Keep things nearby and everything becomes faster?
Well, just like your past relationships, it's more complicated than that.
What happens when the original person...sorry, data changes?
Imagine that a supply depot has a large stock of equipment. Let's say the stock majorly consists of drones, because modern warfare has shown just how useful relatively inexpensive drones can be for reconnaissance, targeting, and other battlefield roles. Now imagine that central command introduces a newer and more effective type of drone, but the forward depot is still handing out the old ones.
The depot has supplies. The problem is that those supplies are no longer the best or most current version.
Cached data can have the same problem.
A cache can contain information that is no longer current. This is known as stale data.
One common solution is to give cached data an expiration time. This is often called a TTL, or Time To Live. You might tell a cache to keep a particular piece of information for five minutes. After those five minutes, the cached copy expires and the application retrieves a fresh version.
Another approach is to explicitly invalidate the cache when the underlying data changes.
This is one of the reasons caching can become complicated. Getting data quickly is easy. Knowing exactly when the cached copy should no longer be trusted is much harder.
Why You Should Think About Caching
Whenever something is expensive to retrieve, calculate, or transfer, it is worth asking whether that work really needs to happen every time.
Could the result be reused? Could it be stored somewhere closer? Could several requests share the same cached result? Could it safely remain cached for a few seconds, minutes, or hours?
A well-designed cache can reduce database queries, network traffic, CPU usage, latency, and infrastructure costs.
But caching isn't automatically a good idea everywhere. Every cache introduces another layer of complexity. You now have to think about expiration, invalidation, memory usage, consistency, and what happens when the cache is empty.
Sometimes adding a cache solves a performance problem. Sometimes it simply creates a more complicated system without solving anything meaningful.
The ability to know when to keep things close and when not to is one of those things that can make you a good software engineer, a good military strategist, or, apparently, a good relationship expert.
Fun Fact: Learning keyboard shortcuts is a form of caching. Instead of repeatedly navigating through menus to perform the same action, you keep the command close at hand in your muscle memory.
Keep caching!
