How Linux uses Sysdig to troubleshoot system faults

Sysdig is a system debugging tool that can troubleshoot and monitor the system. It is very useful when the system is faulty. The following small series will introduce you to Linux using Sysdig to troubleshoot. Method, interested friends can come to understand.

On Linux, use the following command to install Sysdig:

curl -s https://s3.amazonaws.com/download.draios.com/stable/install- Sysdig |  Sudo bash

This will install Sysdig to an rpm or deb Linux system.

Capture System Activity

Capture in real time, print the results to standard output:

sysdig

Save the captured results to the file system.scap for later Analysis:

sysdig -w system.scap

Captures the specified number of events 200 and saves them to a file:

sysdig -n 200 -w system.scap

Read the captured file:

sysdig -r system.scap

Interpret the capture result

(1) (2) (3) (4) (5 (6) (7) (8)

1 10:54:50.462463956 0 sysdig (29043) sysdigevent event_type=1 event_data=0

2 10:54:50.462603110 0 sysdig ( 29043) sysdigevent event_type=1 event_data=0

3 10:54:50.462729565 0 sysdig (29043) sysdigevent event_type=1 event_data=0

4 10:54:50.462859521 0 sysdig ( 29043) 》sysdigevent event_type=1 event_data=0

5 10:54:50.463206317 0 sysdig (29043) 》 switch next=0 pgft_maj=0 pgft_min=1 790 vm_size=35748 vm_rss=7164 vm_swap=0

6 10:54:50.464246835 0 "NA" (0) 》 switch next=7 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0< Br>

7 10:54:50.464249707 2 "NA" (0) 》 switch next=8374 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

8 10:54:50.464255940 0 "NA" (7) 》 switch next=0 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

9 10:54:50.464264256 2 "NA" (8374) 》 switch next= 0 pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

10 10:54:50.464358113 2 "NA" (0) 》 switch next=854(mlnet) pgft_maj=0 pgft_min=0 vm_size =0 vm_rss=0 vm_swap=0

11 10:54:50.464370099 2 mlnet (854) << poll res=0 fds=

12 10:54:50.464378193 2 mlnet (854) 》 Poll fds= timeout=5

13 10:54:50.464385400 2 mlnet (854) 》 switch next=0 pgft_maj=216 pgft_min=3386 vm_size=162608 vm_rss=12196 vm_swap=2716

14 10:54:50.464950541 0 "N A》 (0) 》 switch next=1105(memcached) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 vm_swap=0

15 10:54:50.464954692 0 memcached (1105) epoll_wait res=0

16 10:54:50.464976007 0 memcached (1105) 》 epoll_wait maxevents=32

17 10:54:50.464984030 0 memcached (1105) 》 switch next=0 pgft_maj=3 pgft_min=247 Vm_size=327412 vm_rss=1860 vm_swap=468

18 10:54:50.465256687 2 "NA" (0) 》 switch next=2181(plugin-containe) pgft_maj=0 pgft_min=0 vm_size=0 vm_rss=0 Vm_swap=0

19 10:54:50.465261465 2 plugin-containe (2181) << poll res=0 fds=

20 10:54:50.465297692 2 plugin-containe (2181) 》 getrlimit Resource=3(RLIMIT_STACK)

The results captured by Sysdig are as shown above, and the meaning of each column is:

Event Number

Timestamp

CPU number

Process name

Thread ID

Event direction," is the entry event, To exit the event

event type, such as open, read, etc.

event parameter list

filter capture result

By default, Sysdig captures information There are so many things to find that we are interested in, which requires filtering like grep.

Filter by field category:

sysdig -r system.scap proc.name=sysdig

This command filters out system events with the process name sysdig and the result is :

1 10:54:50.462463956 0 sysdig (29043) sysdigevent event_type=1 event_data=0

2 10:54:50.462603110 0 sysdig (29043) sysdigevent event_type=1 event_data= 0

3 10:54:50.462729565 0 sysdig (29043) sysdigevent event_type=1 event_data=0

4 10:54:50.462859521 0 sysdig (29043) sysdigevent event_type=1 event_data= 0

5 10:54:50.463206317 0 sysdig (29043) 》 switch next=0 pgft_maj=0 pgft_min=1790 vm_size=35748 vm_rss=7164 vm_swap=0

Sysdig provides fd, process Field categories such as evt, user, group, and syslog can be queried by sysdig -l.

In addition to =, Sysdig's filter expressions are also supported! Comparison operators such as =, ", ", ", "= and contains.

At the same time, you can also use Boolean operators such as and, or, not. For example:

sysdig -r system.scap proc.name=sysdig and evt.type=switch

Chisels

In Sysdig, chisels is a script written in Lua. Can be used to extend the filtering capabilities of Sysdig.

For example, if we want to see the most frequent processes for reading and writing disk files, we can use the topprocs_file chisels:

sysdig -c topprocs_file

The result is:

Bytes Process

------------------------------

448.36KB mozStorage

220.38KB perl

1.69KB tmux

1.62KB sh

1.59KB Xorg

1.30KB urxvtd

Chisels, available through sysdig -cl. Of course, if you are familiar with Lua, you can also write your own chisels.

The above is the introduction of Linux using Sysdig method. By using the Sysdig command, the fault information of the Linux system is captured in time, and related problems are fixed to ensure that the performance of the Linux system is kept at an optimal state.