Exclude Linux server failures with strace

strace is a useful gadget – most Linux systems have been installed by default – you can track system calls to let you know what a program is doing in the background. Strace is a basic debugging tool; but it's great software even if you're not tracking a problem. It can tell you a lot about how a Linux program works.

A system call is a message from the application to the kernel. User programs in modern computer systems run in a sandbox: they don't allow direct interaction with the computer (so you can't stuff some data into registers to do some work as before). Instead, whenever a program needs to interact with the system, it sends a request (system call) to the kernel. Strace is used to track these messages. So keep in mind that if you don't see any strace output for a while, that doesn't mean your program is blocking. It's very likely that the program does something in its own sandbox, and these things don't need to communicate with the rest of the system.

Usage

Strace programs can do these things, but it always outputs everything directly to the standard error file (that is, the screen). As you will see, it produces a lot of output; so usually you'd better set an output file with the -o option:

strace -o outputfile.txt program has some editors (eg vim) can syntax highlight the output of strace. This means that different parts of the file, as well as different parts of each line, are displayed in different colors. This feature is quite useful, and I highly recommend using an editor like this to view the output of strace.

Command output explanation

Try strace -o strace.out ls –l, then open strace.out with your favorite editor and enable syntax highlighting.

Before diving into the details, let's take a look at the basic structure of each line. Strace records every system call made by the program and is displayed on a separate line. The name of the system call appears at the beginning of each line, the parameters appear in parentheses, and the return value is after the equal sign, which is the end of a line. The first few lines of the command ls –l are basically the following:

execve("/bin/ls", ["ls", "-l"], [/* 21 Vars */]) = 0 brk(0) = 0x619000 mmap(NULL, 4096, PROT_READ