WEBVTT

00:00:00.500 --> 00:00:03.930
So, how can the memory system
arrange for the right data

00:00:03.930 --> 00:00:07.060
to be in the right
place at the right time?

00:00:07.060 --> 00:00:09.560
Our goal is to have the
frequently-used data

00:00:09.560 --> 00:00:12.030
in some fast SRAM.

00:00:12.030 --> 00:00:13.630
That means the
memory system will

00:00:13.630 --> 00:00:16.340
have to be able to predict
which memory locations will

00:00:16.340 --> 00:00:17.770
be accessed.

00:00:17.770 --> 00:00:20.270
And to keep the overhead
of moving data into and out

00:00:20.270 --> 00:00:24.040
of SRAM manageable, we’d like
to amortize the cost of the move

00:00:24.040 --> 00:00:25.610
over many accesses.

00:00:25.610 --> 00:00:27.520
In other words we
want any block of data

00:00:27.520 --> 00:00:31.920
we move into SRAM to
be accessed many times.

00:00:31.920 --> 00:00:36.770
When not in SRAM, data would
live in the larger, slower DRAM

00:00:36.770 --> 00:00:38.580
that serves as main memory.

00:00:38.580 --> 00:00:40.310
If the system is
working as planned,

00:00:40.310 --> 00:00:43.280
DRAM accesses would
happen infrequently, e.g.,

00:00:43.280 --> 00:00:45.710
only when it’s time to
bring another block of data

00:00:45.710 --> 00:00:48.240
into SRAM.

00:00:48.240 --> 00:00:50.720
If we look at how
programs access memory,

00:00:50.720 --> 00:00:54.500
it turns out we *can* make
accurate predictions about

00:00:54.500 --> 00:00:58.040
which memory locations
will be accessed.

00:00:58.040 --> 00:01:01.790
The guiding principle is
“locality of reference” which

00:01:01.790 --> 00:01:06.090
tells us that if there’s an
access to address X at time t,

00:01:06.090 --> 00:01:09.310
it’s very probable that the
program will access a nearby

00:01:09.310 --> 00:01:11.830
location in the near future.

00:01:11.830 --> 00:01:15.380
To understand why programs
exhibit locality of reference,

00:01:15.380 --> 00:01:19.790
let’s look at how a running
program accesses memory.

00:01:19.790 --> 00:01:22.290
Instruction fetches
are quite predictable.

00:01:22.290 --> 00:01:24.770
Execution usually
proceeds sequentially

00:01:24.770 --> 00:01:27.250
since most of the time
the next instruction is

00:01:27.250 --> 00:01:29.050
fetched from the
location after that

00:01:29.050 --> 00:01:31.430
of the current instruction.

00:01:31.430 --> 00:01:33.500
Code that loops will
repeatedly fetch

00:01:33.500 --> 00:01:35.650
the same sequence
of instructions, as

00:01:35.650 --> 00:01:39.030
shown here on the
left of the time line.

00:01:39.030 --> 00:01:41.590
There will of course be branches
and subroutine calls that

00:01:41.590 --> 00:01:44.970
interrupt sequential execution,
but then we’re back to fetching

00:01:44.970 --> 00:01:47.720
instructions from
consecutive locations.

00:01:47.720 --> 00:01:49.950
Some programming
constructs, e.g.,

00:01:49.950 --> 00:01:52.800
method dispatch in
object-oriented languages,

00:01:52.800 --> 00:01:56.370
can produce scattered references
to very short code sequences

00:01:56.370 --> 00:01:58.570
(as shown on the right
of the time line)

00:01:58.570 --> 00:02:01.690
but order is quickly restored.

00:02:01.690 --> 00:02:05.150
This agrees with our intuition
about program execution.

00:02:05.150 --> 00:02:07.680
For example, once we execute
the first instruction

00:02:07.680 --> 00:02:10.410
of a procedure, we’ll almost
certainly execute the remaining

00:02:10.410 --> 00:02:12.500
instructions in the procedure.

00:02:12.500 --> 00:02:15.890
So if we arranged for all the
code of a procedure to moved

00:02:15.890 --> 00:02:19.300
to SRAM when the procedure’s
first instruction was fetched,

00:02:19.300 --> 00:02:22.550
we’d expect that many subsequent
instruction fetches could be

00:02:22.550 --> 00:02:24.830
satisfied by the SRAM.

00:02:24.830 --> 00:02:28.160
And although fetching the first
word of a block from DRAM has

00:02:28.160 --> 00:02:32.170
relatively long latency, the
DRAM’s fast column accesses

00:02:32.170 --> 00:02:34.950
will quickly stream the
remaining words from sequential

00:02:34.950 --> 00:02:36.690
addresses.

00:02:36.690 --> 00:02:39.810
This will amortize the
cost of the initial access

00:02:39.810 --> 00:02:43.900
over the whole
sequence of transfers.

00:02:43.900 --> 00:02:45.790
The story is
similar for accesses

00:02:45.790 --> 00:02:48.360
by a procedure to its
arguments and local variables

00:02:48.360 --> 00:02:50.320
in the current stack frame.

00:02:50.320 --> 00:02:53.460
Again there will be many
accesses to a small region

00:02:53.460 --> 00:02:55.970
of memory during the span
of time we’re executing

00:02:55.970 --> 00:02:59.090
the procedure’s code.

00:02:59.090 --> 00:03:02.760
Data accesses generated by
LD and ST instructions also

00:03:02.760 --> 00:03:04.410
exhibit locality.

00:03:04.410 --> 00:03:07.490
The program may be accessing
the components of an object

00:03:07.490 --> 00:03:08.730
or struct.

00:03:08.730 --> 00:03:12.080
Or it may be stepping through
the elements of an array.

00:03:12.080 --> 00:03:14.770
Sometimes information
is moved from one array

00:03:14.770 --> 00:03:17.600
or data object to another,
as shown by the data

00:03:17.600 --> 00:03:21.110
accesses on the right
of the timeline.

00:03:21.110 --> 00:03:23.800
Using simulations we
can estimate the number

00:03:23.800 --> 00:03:26.210
of different locations
that will be accessed

00:03:26.210 --> 00:03:28.210
over a particular span of time.

00:03:28.210 --> 00:03:30.930
What we discover when we do
this is the notion of a “working

00:03:30.930 --> 00:03:34.690
set” of locations that
are accessed repeatedly.

00:03:34.690 --> 00:03:36.580
If we plot the
size of the working

00:03:36.580 --> 00:03:39.230
set as a function of the
size of the time interval,

00:03:39.230 --> 00:03:42.930
we see that the size of
the working set levels off.

00:03:42.930 --> 00:03:46.490
In other words once the time
interval reaches a certain size

00:03:46.490 --> 00:03:48.710
the number of
locations accessed is

00:03:48.710 --> 00:03:51.200
approximately the
same independent

00:03:51.200 --> 00:03:55.420
of when in time the
interval occurs.

00:03:55.420 --> 00:03:57.520
As we see in our
plot to the left,

00:03:57.520 --> 00:04:00.620
the actual addresses
accessed will change,

00:04:00.620 --> 00:04:03.430
but the number of *different*
addresses during the time

00:04:03.430 --> 00:04:06.620
interval will, on the average,
remain relatively constant

00:04:06.620 --> 00:04:10.050
and, surprisingly,
not all that large!

00:04:10.050 --> 00:04:12.470
This means that if we
can arrange for our SRAM

00:04:12.470 --> 00:04:15.560
to be large enough to hold the
working set of the program,

00:04:15.560 --> 00:04:20.370
most accesses will be able
to be satisfied by the SRAM.

00:04:20.370 --> 00:04:23.850
We’ll occasionally have to move
new data into the SRAM and old

00:04:23.850 --> 00:04:28.090
data back to DRAM, but the
DRAM access will occur less

00:04:28.090 --> 00:04:30.960
frequently than SRAM accesses.

00:04:30.960 --> 00:04:33.420
We’ll work out the
mathematics in a slide or two,

00:04:33.420 --> 00:04:36.750
but you can see that thanks
to locality of reference we’re

00:04:36.750 --> 00:04:40.020
on track to build a memory
out of a combination of SRAM

00:04:40.020 --> 00:04:44.190
and DRAM that performs like
an SRAM but has the capacity

00:04:44.190 --> 00:04:45.790
of the DRAM.

00:04:45.790 --> 00:04:49.240
The SRAM component of our
hierarchical memory system is

00:04:49.240 --> 00:04:51.090
called a “cache”.

00:04:51.090 --> 00:04:54.670
It provides low-latency access
to recently-accessed blocks

00:04:54.670 --> 00:04:56.130
of data.

00:04:56.130 --> 00:04:58.360
If the requested
data is in the cache,

00:04:58.360 --> 00:05:03.350
we have a “cache hit” and the
data is supplied by the SRAM.

00:05:03.350 --> 00:05:05.780
If the requested data
is not in the cache,

00:05:05.780 --> 00:05:09.110
we have a “cache miss” and
a block of data containing

00:05:09.110 --> 00:05:11.940
the requested location will
have to be moved from DRAM

00:05:11.940 --> 00:05:13.980
into the cache.

00:05:13.980 --> 00:05:15.980
The locality principle
tells us that we

00:05:15.980 --> 00:05:18.330
should expect cache
hits to occur much more

00:05:18.330 --> 00:05:21.820
frequently than cache misses.

00:05:21.820 --> 00:05:24.500
Modern computer systems
often use multiple levels

00:05:24.500 --> 00:05:26.740
of SRAM caches.

00:05:26.740 --> 00:05:30.190
The levels closest to the CPU
are smaller but very fast,

00:05:30.190 --> 00:05:32.350
while the levels further
away from the CPU

00:05:32.350 --> 00:05:35.330
are larger and hence slower.

00:05:35.330 --> 00:05:38.030
A miss at one level of the
cache generates an access

00:05:38.030 --> 00:05:42.130
to the next level, and so on
until a DRAM access is needed

00:05:42.130 --> 00:05:45.580
to satisfy the initial request.

00:05:45.580 --> 00:05:47.780
Caching is used in
many applications

00:05:47.780 --> 00:05:51.990
to speed up accesses to
frequently-accessed data.

00:05:51.990 --> 00:05:53.880
For example, your
browser maintains

00:05:53.880 --> 00:05:56.210
a cache of
frequently-accessed web pages

00:05:56.210 --> 00:05:58.610
and uses its local
copy of the web page

00:05:58.610 --> 00:06:01.040
if it determines the
data is still valid,

00:06:01.040 --> 00:06:03.120
avoiding the delay
of transferring

00:06:03.120 --> 00:06:05.220
the data over the Internet.

00:06:05.220 --> 00:06:07.830
Here’s an example memory
hierarchy that might be found

00:06:07.830 --> 00:06:09.430
on a modern computer.

00:06:09.430 --> 00:06:12.190
There are three levels
on-chip SRAM caches,

00:06:12.190 --> 00:06:16.240
followed by DRAM main memory
and a flash-memory cache

00:06:16.240 --> 00:06:18.430
for the hard disk drive.

00:06:18.430 --> 00:06:20.460
The compiler is
responsible for deciding

00:06:20.460 --> 00:06:23.060
which data values are
kept in the CPU registers

00:06:23.060 --> 00:06:27.220
and which values require
the use of LDs and STs.

00:06:27.220 --> 00:06:29.370
The 3-level cache
and accesses to DRAM

00:06:29.370 --> 00:06:33.190
are managed by circuity
in the memory system.

00:06:33.190 --> 00:06:35.810
After that the access
times are long enough

00:06:35.810 --> 00:06:37.840
(many hundreds of
instruction times)

00:06:37.840 --> 00:06:40.260
that the job of managing
the movement of data

00:06:40.260 --> 00:06:42.380
between the lower
levels of the hierarchy

00:06:42.380 --> 00:06:45.370
is turned over to software.

00:06:45.370 --> 00:06:49.370
Today we’re discussing how
the on-chip caches work.

00:06:49.370 --> 00:06:52.590
In Part 3 of the course,
we’ll discuss how the software

00:06:52.590 --> 00:06:57.240
manages main memory and
non-volatile storage devices.

00:06:57.240 --> 00:06:59.480
Whether managed by
hardware or software,

00:06:59.480 --> 00:07:00.990
each layer of the
memory system is

00:07:00.990 --> 00:07:03.810
designed to provide
lower-latency access

00:07:03.810 --> 00:07:07.570
to frequently-accessed locations
in the next, slower layer.

00:07:07.570 --> 00:07:10.070
But, as we’ll see, the
implementation strategies will

00:07:10.070 --> 00:07:13.650
be quite different in the
slower layers of the hierarchy.