Non-empty mount points

Why does Unix allow non-empty directories to be used as mount points?

A few weeks back, I managed to get myself horridly confused, and at the core of the confusion was a directory that I was using as a mount point, and that directory was not empty.

Of course, I didn't know that at the time. Instead, I got a call:

You know that disk we were having trouble with, and you worked on it? Well, something is really weird, because that disk is now showing a 2-year-old backup? Did you somehow restore some ghost files?

Here's the overall sequence of actions that caused my confusion:

• I have some custom backup tools, which back up my system to an external disk, which is mounted as /mnt/backup


• I had some problems with that external disk.


• After a certain amount of investigation, I decided that the external disk had become scrambled irretrievably, and I re-formatted the disk (that is, I ran mkfs on it)


• I rebooted the system, and got ready to run a new backup to verify that my /mnt/backup file system was now happy, and was astonished to see that /mnt/backup was non-empty; moreover, it contained valid backup information from 2007!

It took me a while, but I figured out what was going on: it was due to the iteraction of two behaviors, one which is fairly new (to me), and one which is extremely old:

• My Ubuntu system, and, I think, most modern Linux systems, is now using UUID identification for file systems. I'm not exactly sure when this changed; I don't think that Dapper Drake behaved this way. This particular system was upgraded from Drake, and that's probably when the UUID filesystem identification was switched on, and I didn't notice.


• My Ubuntu system, and, I think, most Linux systems, allowed me to use a non-empty directory as a mount point

Due to the first behavior, the newly-formatted file system was not getting mounted at /mnt/backup, because its UUID had changed, and my /etc/fstab was still specifying the old UUID for my external drive. So Ubuntu had mounted my new disk at something like /media/my-new-uuid.

Due to the second behavior, since I didn't have an external file system mounted at /mnt/backup, my system was happily using the old contents of the directory.

I think that, during the early development of my backup scripts, I had at some point run them without the external filesystem being mounted, so the scripts had happily written a backup into the real /mnt/backup directory. Then, later, I had arranged for the external filesystem to always be mounted, and so all subsequent backups were going to the external filesystem.

I didn't realize that I was mounting the disk on a non-empty directory, so I didn't realize that my system had been operating for several years with all these hidden files, which were present on the root directory, but which were hidden by the mounted file system and thus effectively invisible (yes, the overall disk space usage on the root filesystem was higher than it should have been, but I wasn't paying close enough attention to notice that).

It seems like it would be a nice behavior if the mount command didn't allow mounting an external filesystem onto a non-empty directory mount point, but that's not the sort of thing that Unix systems tend to worry about. The user is always right, they say.