How Linux can improve the copying efficiency of large files

In Linux system operation, when you are making a direct copy of the machine, it is a waste of time when encountering large files, especially when copying files to multiple machines at the same time. So, is there any good way to improve the copying efficiency of large files? Let's learn together.

You can copy one by one by one from the source machine to the target machine, but often time is doubled. Or you can copy from the source machine to multiple target machines at the same time, but depending on the bandwidth of the source machine, the speed is not really fast.

Fortunately, with some UNIX tools you can do better. Combine tee and FIFO to form a fast distribution chain of files: each machine in the distribution chain saves the file and distributes it to its next ring.

First, select a target machine as the last link of the distribution chain. On this machine, you only need to use nc to listen (assuming the port is 1234), then decompress it by pipez through the pipeline and continue to pass. The pipeline divides the data into tar to decompose.

nc -l 1234|  Pigz -d |  Tar xvf -

Then, go up from the end of the distribution chain, set up other target machines, and also listen, decompress, and decompose, but before decompression we output the data to the named pipe through the tee command ( FIFO), another shell pipeline will distribute these uncompressed data to the next ring of the distribution chain:

mkfifo myfifo

nc hostname_of_next_box 1234 nc -l 1234|  Tee myfifo |  Pigz -d |  Tar xvf -

Finally, start the distribution chain on the source machine and transfer the data to the first ring of the distribution chain:

tar cv some_files |  Pigz |  Nc hostname_of_first_box 1234

In my tests, each machine in the distribution chain lost about 3%-10% of its performance (as opposed to a one-to-one copy), but relatively one by one. Or a single machine can be distributed to multiple machines at the same time, and the efficiency improvement is obvious.

The above is the introduction of Linux to improve the efficiency of large file copying, mainly through the UNIX tool to form a file fast distribution chain to achieve fast copy of large files, the efficiency has improved a lot.