• Part 3: Computers In Depth
• Programming

You may already be familiar with the term “operating system”. It’s the thing that usually comes pre-installed when you buy a new computer. Apple’s OS X and Microsoft’s Windows are probably the ones you’ve heard of. But what are they?

An operating system or OS is a collection of programs that makes a computer practical to use.

I admit that this definition is pretty vague. But the main point here is that a kernel alone is not an OS. A kernel is an essential part of every modern OS, but on its own it does basically nothing—it needs other programs to run in order to be really useful. So an operating system is really just a kernel plus those “other programs”. But which other programs? That varies between different OSes and is often a selling point for the companies that make them. Some common inclusions are a program for managing separate user accounts, a program for finding and organizing files, programs for opening common file formats like text and video, a web browser, things like that.

Next we’re going to play around with an OS to really get a feel for how they work. But we need to do a little preparation so that you can follow along with the book easily.

Virtual Machine

It would be great if you could just start fooling around on your own computer in order to learn about being a superuser, but I’m afraid that wouldn’t be very responsible of me. It’s easy to screw things up as a superuser—especially an inexperienced one—and I wouldn’t want you damaging your property on account of me. Also I don’t know what kind of computer you’re using so it’s hard for me to give you any specific tips. So for this chapter we’re going to use one standard computer with a standard OS so that everyone gets the same experience. As a bonus, this standard computer also lets us save our work in case anything goes awry. Don’t worry though: you don’t need to go out and buy a new computer.

Instead you will download a computer. This is (surprisingly) easy to do thanks to Turing completeness. All you need is a program which simulates a computer inside your actual computer.

A simulated computer is often called a virtual machine or VM because it exists only in the digital memory of your computer. But other than that it works just like a normal one.

The program we’re going to use to do this is VirtualBox, a free and open source VM manager. Steel yourself for possibly the hardest exercise in the book so far:

Install and configure VirtualBox. Hint: the VirtualBox User Manual has sections on installing and starting VirtualBox

Once you have that installed, we’re going to create a computer (a virtual machine) to play around with.

Once you finish, notice that your new virtual machine has appeared in the left panel. Left’s give it a spin: click your virtual machine in the left panel and then click the big green “Start” arrow at the top.

Now VirtualBox will pop up a window saying “Select start-up disk” but we don’t care about this right now so just click “Cancel”. A new window has opened: this represents the monitor of your VM. And after a few seconds… tragedy. Something FATAL has occurred. But we have actually learned enough already to understand what has gone wrong.

We know that the most important program on a computer is its kernel since it’s the program that we use to run all other programs on the computer and manage all of the hardware connected to the computer. But we haven’t provided a kernel for our computer to run! In particular, the virtual machine is complaining that it didn’t find any “bootable medium”. “Medium” here is just a generic term for data storage; it could be a hard disk drive, solid state drive, DVD, CD-ROM, floppy disk, etc. “Bootable” is refering to a process called “bootstrapping”.

Bootstrapping (as in “pulling yourself up by your own bootstraps”) is when a simpler program is used to run a more complicated program. It is usually shortened to just booting. That simpler program is called a boot loader.

See, a kernel is usually such a huge and complex program that your computer’s BIOS isn’t sophisticated enough to get it running (recall that the BIOS is the component of the motherboard responsible for starting your computer). Instead you need a simpler program called a boot loader which the BIOS can run and it is the boot loader that gets the kernel running. So the error message we see here is the BIOS complaining that it searched all of the storage devices attached to the virtual machine and couldn’t find any that had a boot loader on them.

We’ll solve this problem by downloading an operating system. Every OS includes a boot loader and a kernel, so that will get us past this particular problem.

Go to archlinux.org and download Arch Linux. If you’re feeling up to a challenge, you can use the recommended BitTorrent download. Otherwise just scroll down and use one of the HTTP direct downloads.

The Arch Linux download includes a .iso file. This file represents a generic storage medium—like the kind our BIOS was looking for—containing a boot loader and the Arch Linux operating system. Now let’s hook it up to the virtual machine in VirtualBox.

Once again, click the big green arrow to start the machine. Something different has happened! The BIOS automatically detected a boot loader on our virtual CD drive so we’re now looking at the interface for the Arch Linux boot loader. We have several options, but we just want the default one: “Boot Arch Linux (x86_64)”, so hit Enter. A bunch of text will fly by the screen as first the boot loader gets ready to run the kernel and then more text as the kernel takes over and starts even more programs running.

Eventually you’ll reach a screen looking something like this

Arch Linux 3.19.3-3-ARCH (tty1)

archiso login: root (automatic login)
root@archiso ~ # _

This is what success looks like. The VM has booted successfully and we are now running the Arch Linux operating system!

If the VM’s window turns completely black, this is just the virtual monitor trying to save power. Clicking in the window and pressing any keyboard key should wake it up.

UNIX

This virtual machine is running an OS called Arch Linux which (you might guess) uses Linux for its kernel. Linux is based on the kernel of an old OS called UNIX which was developed by AT&T in the 70s. UNIX no longer exists as such, but it lives on today through the many OSes which imitate it, including the Linux and XNU-based OSes. These OSes try to be as UNIX-like as possible and are often reffered to as just *nix, meaning “some variety of OS that works kinda like UNIX”. So even though we’re working with Arch Linux specifically, many of the things you will learn here will apply to many other OSes.

Except for Windows. Microsoft likes to be arbitrarily different whenever possible, so most of the things in this chapter will not work as-is on a Windows computer. Despite that, Windows does have some things in common with UNIX conceptually, so the general principles from this section still apply.

Cygwin is an open source project that tries to make Windows more UNIXy. If you like what you learn in this chapter and are stuck with Windows, Cygwin can be a practical way to apply your knowledge.

The screen we’re at right now is telling us that the name of this operating system is archiso and we are logged in as a user called root. Arch Linux is a multi-user OS, meaning that it is designed for multiple people to use the computer simultaneously. So any work that you do on the computer has to be done after logging in with a certain user name. Normally you would have your own user name just for you, but in this case the computer has automatically logged us in as the root user. This isn’t an ordinary user we’re logged in as: root is the superuser!

So we’re logged in as the superuser and ready to roll. First, let’s install some additional tools that weren’t included with the Arch Linux downoad.

Before I start explaining more, let’s become familiar with VirtualBox’s ability to save and load our work. This will be handy moving forward in case you screw something up and want to return to an earlier state where things were working. Simply click “Take Snapshot…” in the “Machine” menu of the VM’s window. You can give your snapshot a name and a description to remind yourself later why you saved your work, then click OK. After a short wait, you should be back to the VM as though nothing happened. But what just occured is VirtualBox created a file storing all of the information about the current state of the VM.

Now to really appreciate how useful snapshots are, type in the following and then press Enter:

rm -rf --no-preserve-root /

If you get a message like “unrecognized option” or “command not found” or it just prints another root@archiso ~ # line right away, you probably typed it wrong. Just carefully try again.

As lots of text starts flying across the screen, you might be wondering, “What did I just do?” Well, right now Arch Linux is happily destroying itself. And by now it has probably already damaged itself beyond repair. Fun! This is the kind of power that a superuser wields. Thankfully this isn’t a real computer and we just took a snapshot so nothing is really lost. To restore the VM, first click “Close” under the “Machine” menu, select “Power off the machine” and click “OK”. This forces the VM to stop.

Now the right pane of the VirtualBox window shows the snapshot we just created for the machine. Select it and then click the “Restore Snapshot” button. It may ask if you want to make a snapshot of the current state, but the current state is terrible so you can skip that. Then just click the “Start” arrow again and voilà! We’re back in business.

Shell

Now that the VM snapshot is restored, we’re back to where we’ve just logged in. After a user logs in to Arch Linux, the kernel automatically runs a program called a “shell” for that user. While the kernel is a program that works like an interface for other programs, the shell is a program that works like an interface for humans. Similar to a kernel, it runs in an infinite loop, waiting for input. But instead of waiting for program instructions to run, it waits for text commands from the user. The root@archiso ~ # text with the flashing cursor is the shell’s way of saying “you just logged into the computer named archiso as the root user and I’m waiting for a command”—this is called the shell’s “prompt”.

A text interface (like the one provided by the shell) is often called a command line or a terminal. Command line because of the way you type commands line-by-line, and terminal because older computers used a special machine called a terminal for inputting commands and reading output.

Let’s type a few commands to get a feel for what they look like. The shell can be pretty picky about the input you give it, so try to type things exactly as you see them here.

# foo bar

# ls -l /usr

# find /etc -name system.conf

# grep -i ps /etc/bash.bashrc

# cp /usr/share/licenses/common/GPL3/license.txt /root

# head --lines=20 /root/license.txt

If you ever get stuck at a point where the shell won’t prompt you for another command, you can press Ctrl+C. This “interrupts” whatever the shell is doing and should force the shell to show another prompt. As a last resort, you can always shut down the VM and restore a prior snapshot.

After each shell command you should see the shell print some text and then another prompt for the next command. This is sometimes called a “read, evaluate, print, loop” or REPL. The shell reads your command, evaluates it (i.e. does something on the computer with the command), prints the result of the command to the screen, and then loops back to the beginning to read another command.

How does the shell read these commands? First the shell looks at the “words” of the command by splitting up the text on spaces. So grep -i ps /etc/bash.bashrc/ becomes four words: grep, -i, ps, /etc/bash.bashrc. The first word is the name of the command and it is usually the most significant word because it indicates what sort of action the shell is going to perform. So, for this example, the shell is performing an action called “grep”. The rest of the words are called the “arguments” of the command and they determine the specific behavior of the action. The arguments that you are allowed to use depend entirely on the name of the command being run, which is why the five different commands above have completely different arguments.

We’ll learn more about these commands and their arguments in a little bit, but as a quick guide, you can generally group arguments into two categories: options and filenames. Arguments that start with a - or -- are options: they tell the specific command (e.g. grep) what sort of thing to do. Options are (usually) optional and the command will normally perform some default task if you leave them out. The other arguments (which often contain a /) are the filenames: they indicate not what the command should do, but where the command should get its data from. These aren’t hard rules though; sometimes options don’t start with a - or may contain / like a filename—it depends entirely on the specific command.

You can think of commands like programs: they take input (their arguments) and produce output (the text printed by the shell) and the output depends only on the input.

But what’s that about “getting data” from a filename. What does that even mean?

File Systems

We already know that computers store data using binary and data formats, but how does a computer distinguish between different sets of data? A hard drive can contain data in many different formats, so how do we say “this audio data” or “that image data” or “the other text file” to the computer?

The solution is another data format called a “file system”.

A file system is a data format which wraps other data formats such that specific chunks of data can be referred to. These chunks of data are called files.

The file system format usually stores information associated with each file (such as a name for the file and its size in bytes) so that you can view a summary of the data at a high level without needing to examine the data format of each individual file. Interacting with the file system is grouped under the I/O category, which means it is managed by the kernel and the kernel provides an interface for other programs to work with the file system.

Let’s jump right in and explore the file system of our VM.

The shell has many commands for working with the file system, but one of the most important is ls, which lists files in the file system. Try the following command:

# ls /

You should see this output:

bin boot dev etc home lib lib64 mnt opt proc root run sbin srv sys tmp usr var

Now let’s break down what happened. From what we just learned, we know that this command is called ls and it has one argument: the filename /. In a *nix file system, / is a special filename which we call the “root” of the file system (no relation to the root we logged in as). The root of the file system is special because it is how we refer to the entire file system at once. So in English we could read the previous command as “list the files on the file system”. From the output in the shell, we can see that there are 18 files.

Wait a second, only 18 files? That seems pretty small. In fact there are many more (thousands) of files in the file system. But rather than storing all of these files in the root, the file system organizes them so that we don’t get pages and pages of output when we do ls /. This organization is done using special files called directories.

A directory is a file which contains no data of its own but instead contains a list of other files on the file system.

ls helps us out here by coloring all of the directories dark blue. That’s where our other files are squirreled away. And, in fact, we already know how to list them. I didn’t mention this earlier but that special / filename is actually the root directory. So now we really understand how ls works: its argument is the filename of a directory and it outputs the filenames of the files contained in that directory. To list our other files, we can just use ls again with the name of one of these other directories.

But if we try this:

# ls etc

we get an error

ls: cannot access etc: No such file or directory

This is because we didn’t specify where in the file system etc is. Remeber that we found etc by listing /, so the solution is to specify that:

# ls /etc

You should see quite a lot of files in there along with a few directories, for example systemd. So let’s see what files are in there. Again, we need to specify where the directory is: systemd is in etc, and etc is in / so the command is:

# ls /etc/systemd

Notice how we use another / to indicate that systemd is in etc.

In /etc/systemd we see another directory system. Use ls to see what files are in there, and keep going deeper if some of them are directories. You can also try exploring some other directories in /.

If any command prints a lot of output, you can scroll through the result by pressing Shift+Page Up and Shift+Page Down.

File systems are a little different in Microsoft’s Windows operating systems. A minor difference is that they use backslashes \ instead of forward slashes / to separate directory names. More significantly, the root directory isn’t just called \, it has a letter name depending on which HDD the files are stored. So in Windows, C:\Program Files is the path for the directory Program Files stored in the root of the file system on the HDD named C.

A common analogy for a file system is a drawer in a filing cabinet. In this analogy, files are files, directories are folders, and the root directory is like the drawer itself in which all files and folders are stored.

Working Directories

Now let’s look at /etc, for example. If you ls it, you’ll see that it has lots of files and lots of directories. What if we want to spend some time exploring /etc? Won’t it get really tedious typing out /etc over and over again as we list different directories? Thankfully, the shell can help us out here. The shell lets us “go into” a directory so that when we refer to filenames in that directory we don’t need to mention the file system location of the directory we are “in”. This is purely for convenience, but it really saves a lot of effort.

The command to “go into” a directory is cd, for change directory. Let’s do it:

# cd /etc

This doesn’t print any output, but notice that now the shell prompt is different.

root@archiso /etc # _

Now that we are “in” /etc the shell reminds us by putting the directory’s filename in the prompt.

The directory we “go into” using cd is called the working directory.

So how does the working directory save us time? Well now we can use the special filename . which means “the working directory”. So the following two commands are now equivalent:

# ls /etc/kernel
# ls ./kernel

Now you can explore deeper and deeper like so:

# cd ./kernel
# ls .

If you ever lose track of where you are, you can use the command pwd (with no arguments) to print the working directory.

# pwd
/etc/kernel

So now we can easily explore deeper, but is there an easy way to go back? Yep. The special filename .. refers to the directory containg our working directory (sometimes called the “parent directory”). So if you cd somewhere you don’t want to be, you can back up with just:

# cd ..

Now where it gets really crazy is that you can chain these special filenames together. So ../.. means the parent directory of our parent directory. And /etc/../usr/.. is just a really convoluted way of referring to / (verify it yourself using ls)! Similarly ./././././. is the same as .. To think about these chains, you can just read them from left to right, and imagine stepping through the file system as you do so. So when you see ls /etc/kernel/../netctl/./hooks you can think “go into /etc, then go into kernel, then go back to the parent directory, then go into netctl, then go into the working directory, then go into hooks and list the files there”.

A file path (or just path) is a chain of filenames in a file system that describe how to locate a particular file in the file system directories.

Revisiting the cd and ls commands, we see that their first argument is really a file path.

There is another convenience for making paths just a little nicer to work with. Previously, all of our paths started with either / or . or ... That’s because these things tell the computer exactly where to start looking in the filesystem (the root directory, working directory, or parent directory). But what if a path starts with none of these things? In that case, the assumption is that the computer should look in the working directory. So whenever you would start a path with . you can just leave that part out because that’s where the computer will look by default. So the following sequence of commands works just fine:

# cd /
# cd etc
# cd kernel
# ls

root@archiso ~ # _

A file path that starts with the root / is called an absolute path because it says exactly where a file is in the file system. File paths that start with . or .. are relative paths because they locate files relative to the working directory.

Commands

Now that we know how to travel about the file system, let’s look again at some shell commands.

man

Perhaps the most important command of all is man, short for manual. Arch Linux comes with an extensive manual describing all of the commands (among other things). To view the manual, simply use the man command with the name of the command you want to learn about as its argument. For example, to learn about ls:

# man ls

Or to learn about man itself:

# man man

When you view the manual, the shell switches to a different view called a “pager” which makes reading easier. You can scroll up and down with ↑, ↓, Page Up, Page Down and quit with Q.

Most manual pages are split up into similar sections:

• Name: a very short description of the command
• Synposis: a (confusing) description of all the different ways you can enter the command
• Description: a very long description of the command
• Options: describes all of the options arguments

You may find that some commands organize their manuals differently or have fewer or more sections. Commands can differ in a lot of ways and their manuals often reflect that. As we learn about other commands remember that you can always learn way more by reading the manual.

Additionally, many (but not all) commands accept a --help or -h option which prints some basic information about the command. It’s worth a shot if you don’t feel like reading the manual:

# ls --help

ls

We already know a lot about ls but here are a few common features we didn’t mention. You can list multiple directories at once just by adding them all to the arguments:

# ls / /etc /usr

Adding the -l option lists one file on each line with additional information such as the size (in bytes) of the file and when it was last modified:

# ls -l /etc

find

If you’re looking for a specific file somewhere in the file system, it can be pretty tedious to find it using just ls and cd. find simplifies this task by searching the file system for you. The simplest usage looks like this:

# find /etc/pacman.d

which simply lists all files in the directory pacman.d and any files in directories in pacman.d and then any files in directories in directories in pacman.d, etc. We can then find specific files among these by adding options that specify what sort of file we’re looking for. For example we can find all files that have gpg in the filename like so:

# find /etc/pacman.d -name '*gpg*'

Read the manual if you want to know more about it.

cat

Commands like ls and cd care mainly about directories because they focus on navigating the file system. But most commands are actually designed for regular (non-directory) files, which is where the computer actually stores data.

In particular, most commands are designed to work with human-readable text data (in a format like ASCII). cat, short for concatenate, is one of the simplest such commands. You give it several files as arguments and it outputs the contents of the files one after another.

# cat /etc/resolv.conf /etc/nsswitch.conf

head and tail

For large files, cat can spew out way too much information. head is like cat but it only prints the first 10 lines of each file.

# head /etc/services

In the earlier examples we saw how to use an option to print a different number of lines. The command tail is similar to head but it prints the last few lines of each file instead.

less

If you have a large file that you want to read, you can use less which takes a file as its argument and shows its contents in a pager the same way man does. You can scroll up and down and quit using the same keyboard keys as in man.

less /etc/makepkg.conf

cp and mv

Short for copy, cp takes two paths as arguments, copies the contents of the file at the first path, and saves the contents at the second path. If the second path is a path to a directory, it creates a file in that directory with the same filename as the original file. Otherwise it uses the second path as the new file name. We used this earlier to copy a text file into /root.

Copying a file to a directory:

cp /etc/fstab /root

Copying a file to a directory but with a different file name:

cp /etc/fstab /root/foobar

mv is a similar command, but it moves the file from one path to another rather than creating a copy.

mv /root/foobar /root/fizzbuzz

grep

grep is short for global regular expression print, which is really not a useful mnemonic. What it does is search for text in files. The simplest use-case looks like this:

# grep word /etc/inputrc /etc/nanorc

which simply prints any lines from /etc/inputrc and /etc/nanorc that have “word” in them. grep has a lot of useful options that let it do all kinds of neat stuff, but you should read the manual to learn more about it.

In actuality, grep doesn’t just search for text, it searches for patterns of text. These patterns are called “regular expressions”, hence the name. The simplest patterns are plain text like word which just match that exact text, but we can also use complicated patterns like '\bw\w*d\b' which searches for any words that start with w and end with d.

Command Types

Now we have a broad idea of what some commands do, but we don’t know how they do these things. To learn more about them, let’s look at the type command. Shell commands fall into different categories and the commands in each category work in different ways. type takes another command as its argument and outputs the category that command is in (if you include the -a option, it will tell you if a command is in multiple categories). Try this:

# type -a cd
cd is a shell function from /etc/zsh/zshrc
cd is a shell builtin

What this is telling us is that cd is a special command understood by the shell program. This makes sense because the working directory is something that the shell keeps track of for our convenience. When you type cd foo you are commanding the shell itself to change its working directory—hence “shell builtin”. But what about other commands?

# type -a man
man is /usr/bin/man

Interesting. That looks like a file path. Let’s do a little test and type these two commands:

# man ls
# /usr/bin/man ls

Weird, right? We typed a file path instead of a command name, but the output was the same. What’s going on here? When you type a command into the shell, the shell looks up the command name in each of the command categories it knows. So, for example, first it might check if man is a shell builtin. If it can’t find the command in any of its categories, the shell tries one last thing: it looks to see if the command name is actually the file name of a program. If it is, the shell tells the kernel the to run that program and passes along the arguments you provided—which is exactly what happens here with man. The reason why we can type either man or /usr/bin/man is because the shell keeps track of which directories in the file system contain programs and automatically searches all of those directories for the command name we provided. In fact, most programs on the computer are located in /usr/bin (bin is short for binary, which is another word for program), so that’s a good place to start if you want a comprehensive list of programs.

# echo $PATH

As an example of another command category, try

# type -a ls
ls is aliased to `ls --color=auto'
ls is /usr/bin/ls

An alias is another time-saving feature of the shell. What this tells us is that when we type ls foo, the shell automatically replaces that with ls --color=auto foo and then runs the program /usr/bin/ls with those arguments. This alias is what nicely colors file names for us when we list directories.

So now we know that most shell commands are in fact just names of programs that we want the kernel to run. In the next chapter we’ll take a closer look at how these programs work.

Another convenient feature of the shell is that you can press ↑ and ↓ to scroll through previous commands. This is helpful if you want to correct a typo in a command you ran.

# cd /sys/devices/system/cpu
# cat kernel_max modalias offline online possible present uevent

# cat /sys/devices/system/cpu/*

# ls /usr/*/share
man

# cat /sys/devices/system/cpu/o*

# head /etc/nanorc

# head /etc/naTab

# head /etc/mkiTab

# head /etc/mkinitcpioTabTab
completing file
mkinitcpio-archiso.conf mkinitcpio.conf mkinitcpio.d/

# ls /usr/*/shareTab
# cat /etc/ge*TabTab

# gTabTab
Display all 126 possibilities? (y or n)

Graphics

Perhaps you still feel a little weird about the commands and programs we are running because they are so unlike the programs you are used to on your computer. How are these weird little programs with their arguments and text output related to big, fancy programs like your web browser or video games that have nothing to do with a text interface like the shell?

They are actually pretty closely related! Just like how the shell is a REPL running in a loop, reading text input that you type and outputting text to the screen, programs that display graphics (like a web browser) are also a kind of REPL. They run in a loop reading not just text input from the keyboard but also mouse input (so they can see where/what you click on). And rather than waiting for an entire command before evaluating and printing, they run in a very fast loop reading and evaluating input dozens of times per second. This is what creates the illusion that these programs are responding to your input in “real time” even though they are really running in discrete steps just like a shell.

Exercises

• Part 3: Computers In Depth
• Programming