High-Dimensional Spaces Are Counterintuitive, Part Two

The volume of an n-cube of edge length s is easy to work out. A 2-cube has s2 units of area. A 3-cube has s3 units of volume. A 4-cube has s4 units of 4-volume, and so on — an n-cube has sn units of n-volume. If the n-cube has edge of s>1, say s=2, then clearly the n-volume dramatically increases as the dimensionality increases — each dimension adds a lot more “room” to the n-cube.

A 2-sphere (circle) is pretty close in area to the smallest 2-cube (square) that encloses it — sure, you lose some area at the four corners, but not a whole lot. Though the circle is far from the square at the four corner, it is very close to the square at the four sides. A circle has about 75% the area of its enclosing square. But a 3-sphere inside a the smallest 3-cube that encloses it is far from eight corners and close to only six sides. A 3-sphere is about half the volume of the 3-cube. As you go up in dimensions, you get more and more corners that are far from the n-sphere — there are 2n corners and only 2n sides so the comparative volume of the sphere goes down.

In fact, you don’t even need to compare n-spheres to n-cubes — after you reach 5 dimensions, the n-volume of an n-sphere starts going down, not up, as dimensionality increases. With some pretty easy calculus you can show that the n-volume of an n-sphere of radius r is:

V1 = 2 r
V2 = π r2
Vn = Vn-2 2 π r2 / n

For any fixed radius this rapidly approaches zero as n gets big. Pick a big n, say 100. The volume of a 100-sphere is going to be (π r2 /50) x (π r2 /49) x (π r2/48) … (π r2/1). Suppose r is 1 — then all of those terms except for the last few are going to be quite a bit smaller than 1. Every time you add more dimensions, the n-volume of the unit n-sphere gets smaller and smaller even as the n-volume of the smallest n-cube that encloses the unit n-sphere gets exponentially larger and larger!

Here’s another weird fact about the volume of hypersolids. Consider two squares, one inside the other. How big does the small square have to be in order to have, say, 1% the area of the larger square? That’s pretty easy. If the inner square has 10% the edge length of the outer square, then it has 1% of the area of the outer square.

What about nested 3-cubes? An inner 3-cube with edges 10% the length of the edge of the outer 3-cube would have 0.1% the volume, too small. Rather, it needs to havean edge about 21% of the edge of the outer 3-cube, because .21 x .21 x .21 = about 0.01.

What about a nested n-cube? In order to have 1% the n-volume of the outer n-cube, the inner n-cube needs to have an edge of (0.01)1/n of the outer n-cube side. For n=100, that’s 0.955. Think about that for a moment. You’ve got two 100-cubes, one has edges 2 units long, the other has edges 1.91 units long. The larger n-cube contains ONE HUNDRED TIMES more volume.

Try to visualize the smaller n-cube being entirely inside the larger n-cube, the two n-cubes having the same central point. Now wrap your mind around the fact that the smaller n-cube is 1% the volume of the larger. The conclusion is unavoidable: in high dimensions the vast majority of the volume of a solid is concentrated in a thin shell near its surface! Remember, there’s 2100 corners in a 100-cube, and that makes for a lot of space to put stuff.

It’s counterintuitive because the very idea of “near” is counterintuitive in higher dimensions. Every time you add another dimension, there’s more room for points to be farther apart.

The distance between opposite corners of a square of edge 2 is 2√2. The distance between opposite corners of a 3-cube is 2√3, quite a bit bigger. The distance between opposite corners of a 100-cube is 2√100 = 20 units! There are a whole lot of dimensions to move through, and that adds distance.

We could make the same argument for an n-sphere and show that the vast majority of its (comparatively tiny) volume is also in a thin shell near the surface; I’m sure you can see how the argument would go, so I won’t bother repeating myself.

Because distance is so much more “expensive” in higher dimensions, this helps explain why n-spheres have so much less volume than n-cubes. Consider a 100-cube of edge 2 centered on the origin enclosing a 100-sphere of diameter 2, also centered on the origin. The point (1,0,0,0,0,0…,0) is on both the 100-cube and the sphere, and is 1 unit from the origin. The point (1,1,1,…,1) is on the 100-cube and is ten units away from the origin. But a 100-sphere by definition is the set of points equidistance from the origin, and distance is expensive in high dimensions. The nearest point on the 100-sphere to that corner is (0.1, 0.1, 0.1, …, 0.1), 9 units away from the corner of the 100-cube. Now its clear just how tiny the 100-sphere is compared to the 100-cube.

OK, so far we’ve been considering n-cubes that entirely enclose n-spheres, ie, an n-cube of edge length 2 that encloses a unit n-sphere, kissing the sphere at 2n points. But we know that this n-cube has ginormously more volume than the n-sphere it encloses and that most of that volume is near the edges and corners. What if we abandon the constraint that the n-cube contains 100% of the n-sphere’s volume. After all, there are only 200 points where the 100-sphere kisses the 100-cube, and that’s not very many at all.

Suppose we want an 100-cube that contains 99.9% of the volume of the unit 100-sphere. We can cover virtually all of the volume of the 100-sphere with an 100-cube of edge 0.7 instead of 2. Sure, we’re missing (1,0,0,0,…,0), but we’re still hitting (0.1,0.1,0.1,…) with huge amounts of room to spare. Most of the volume inside the 100-sphere isn’t near the 200 points with coordinates near the axes.

How much do we reduce the volume of the 100-cube by shrinking it from 2 on the edge to 0.7? We go from 2100 n-units of volume to 0.7100, a factor of around 4×1045 times smaller volume! And yet we still enclose virtually all the volume of the 100-sphere. The corner of the smaller 100-cube at (0.35, 0.35, 0.35, …) is now only 2.5 units away from (0.1, 0.1, …) instead of 9 units away. This is a much better approximation of the unit 100-sphere. It’s still hugely enormous compared to the unit 100-sphere in terms of sheer volume, but look at how much volume we save by approximating the 100-sphere as a small 100-cube!

Feeling dizzy yet? Next time we’ll see that these facts

  • n-spheres are tiny compared to n-cubes
  • hypersolids have most of their volume close to their surfaces
  • you can enclose almost all the volume of an n-sphere with a small n-cube

have major repercussions when we throw probability theory into the mix.

High-Dimensional Spaces Are Counterintuitive, Part One

A friend of mine over in Microsoft Research pointed out to me the other day that high-dimensional spaces are really counterintuitive. He’d just attended a lecture by the research guys who wrote this excellent paper and we were geeking out at a party about it. I found this paper quite eye-opening and I thought I might talk a bit about some of the stuff that’s in here at a slightly less highfalutin level — the paper assumes a pretty deep understanding of high-dimensional spaces from the get-go.

**************************

It’s hard to have a geometrical picture in your head of a more than three-dimensional space. I usually have to use one of two analogies. The first analogy I like goes like this: think of a line — one dimensional. Imagine that you have a slider control that determines your position on that line, from, say, -1 to 1, left-to-right. That’s pretty visually clear.

Add another slider that determines your up-and-down position, and you’ve got a square. Each point on the square has a unique set of slider positions.

Add another slider that determines your out-and-in position, and you’ve got a cube. Again, these are easy to picture. Every point in the cube has a unique combination of slider positions that gets you to that point.

Now think of a cube with a slider control below it that lets you slide from intense red on one end through dark red and to black on the other. Now you’ve got four axes you can move around — height, width, depth and redness. The top right front corner of the bright red cube is a certain “colour distance” from the corner of the top right front black cube. That this is not a spatial dimension isn’t particularly relevant; we’re just picturing a dimension as redness for convenience.

Every time you want to add another dimension, add another slider — just make sure that whatever is sliding is completely independent of every other dimension. Once you’ve added green and blue sliders then you’ve got a six-dimensional hypercube. The “distance” between any two 6-d points is a function of how much you have to move how many sliders to get from one to the other.

That analogy gets across one of the key ideas of multi-dimensional spaces: that each dimension is simply another independent degree of freedom through which you can move. But this is a quite mathematical and not very geometric way of thinking about dimensionality, and I want to think about the geometry of these objects. Let’s abandon this analogy.

The second analogy is a little bit more geometrical. Think of a line, say two units long. Now associate every point on that line with another line, also two units long “crossing” it at the new line’s center. Clearly that’s a filled-in square — after all, every point along one side of a square has a straight line coming out from it perpendicularly. In our slider analogy, one slider determines the point along the “main line”, and the second determines how far to go along its associated line.

Think of another line, but this time, associate every point on it with a square. That’s a solid cube.

Now think of yet another line. Associate every point on it with a cube, and you’ve got a 4-cube. At this point it gets hard to visualize, but just as a cube is an infinite number of equally-sized squares stuck together along a line, so is a 4-cube an infinite number of 3-cubes stuck together along a line. Similarly, a 5-cube is a line of 4-cubes, and so on.

Where things get weird is when you start to think about hyperspheres instead of hypercubes. Hyperspheres have some surprising properties that do not match our intuition, given that we only have experience with two and three dimensional spheres. (2-spheres are of course normally called “circles”.)

The definition of a hypersphere is pretty simple — like a 2-sphere or 3-sphere, a hypersphere is the collection of points that are all the same distance from a given center point. (But distance works strangely in higher dimensions, as we’ll see in future episodes!)

Its hard to picture a hypercube; it’s even hard to picture a hypersphere. The equivalent analogy for n-spheres requires us to think about size. Again, imagine a line two units long. Associate with each point on the line another line crossing at the middle. But this time, the associated lines are of different lengths. The lines associated with the end points are tiny, and the lines associated with the middle are longer. This describes a circular disk — for each point along the diameter of a circle you can draw a perpendicular line through the point extending to the boundaries of the disk on each side.

Now do the same thing again. Take a line, and associate each point on the line with a circle. If the circles are all the same size, you have a cylinder. But if they vary from small at the ends to big in the middle, you’ve got a sphere. Successive cross-sections of a sphere are all circles, but they start small and get big and then get small again.

Now do the same thing again. Take a line and associate each point on the line with a sphere, small at the ends and big in the middle, and you’ve got a 4-sphere. Successive “cross sections” of a 4-sphere are 3-spheres of varying size. Keep going to 5-, 6-, etc, spheres.

A circle of diameter 2 fits into a square of edge length 2, and a sphere of diameter 2 fits into a cube of edge length 2. Clearly an n-sphere of diameter two fits exactly into an n-cube of edge length two — the n-sphere “kisses” the center of each face of the n-cube. You can’t make the n-cube smaller without the n-sphere poking out of it somewhere.

But things start getting weird when you consider the volume of an n-sphere. Tomorrow we’ll compare the volume of an n-sphere to the volume of an n-cube, and discover some surprising and counterintuitive things about where that volume is.