Gnu-Parallel

GNU 可以在程序退出之前並行輸出標準輸出嗎?

  • December 4, 2020
echo 'echo "hello, world!";sleep 3;' | parallel

此命令在完成之前不會輸出任何內容。Parallel 的手冊頁聲稱:

GNU 並行確保命令的輸出與按順序執行命令時的輸出相同。

我猜魔鬼在措辭中:你得到的輸出與正常執行它相同,但輸出與正常執行它不同。例如,我已經尋找了一個可以執行此操作的選項--results /dev/stdout,但這不起作用。

我的案例是查看我正在執行的命令的實時進度輸出。這不是關於已經完成了多少任務,哪些並行可以為我顯示,而是關於我想要查看的每個命令的進度輸出。

我會使用 bash 循環for i in $x; do cmd & done;

是否可以並行執行此操作?如果沒有,還有其他工具嗎?

我想你正在尋找--ungroup. 手冊頁說:

--group  Group output. Output from each jobs is grouped 
        together and is only printed when the command is finished. 

        --group is the default. Can be reversed with -u.

-u當然是 的同義詞--ungroup

要查看一些並行作業的進度,請嘗試--tmuxpane --fg

parallel --tmuxpane --fg seq {} 10000000 ::: {1..100}

您也可能正在尋找-uor (更有可能) --lb。來自man parallel

  --line-buffer
  --lb
      Buffer output on line basis. --group will keep the output together
      for a whole job. --ungroup allows output to mixup with half a line
      coming from one job and half a line coming from another job.
      --line-buffer fits between these two: GNU parallel will print a full
      line, but will allow for mixing lines of different jobs.

      --line-buffer takes more CPU power than both --group and --ungroup,
      but can be much faster than --group if the CPU is not the limiting
      factor.

      Normally --line-buffer does not buffer on disk, and can thus process
      an infinite amount of data, but it will buffer on disk when combined
      with: --keep-order, --results, --compress, and --files. This will
      make it as slow as --group and will limit output to the available
      disk space.

      With --keep-order --line-buffer will output lines from the first job
      while it is running, then lines from the second job while that is
      running. It will buffer full lines, but jobs will not mix. Compare:

        parallel -j0 'echo {};sleep {};echo {}' ::: 1 3 2 4
        parallel -j0 --lb 'echo {};sleep {};echo {}' ::: 1 3 2 4
        parallel -j0 -k --lb 'echo {};sleep {};echo {}' ::: 1 3 2 4

      See also: --group --ungroup

$$ … $$

  --ungroup
  -u  Ungroup output.  Output is printed as soon as possible and by passes
      GNU parallel internal processing. This may cause output from
      different commands to be mixed thus should only be used if you do not
      care about the output. Compare these:

        seq 4 | parallel -j0 \
          'sleep {};echo -n start{};sleep {};echo {}end'
        seq 4 | parallel -u -j0 \
          'sleep {};echo -n start{};sleep {};echo {}end'

      It also disables --tag. GNU parallel outputs faster with -u. Compare
      the speeds of these:

        parallel seq ::: 300000000 >/dev/null
        parallel -u seq ::: 300000000 >/dev/null
        parallel --line-buffer seq ::: 300000000 >/dev/null

      Can be reversed with --group.

      See also: --line-buffer --group

一個-u閃耀的例子是 stdout 和 stderr 混合在同一行中:

echo -n 'This is stdout (';echo -n stderr >&2 ; echo ')'

這將使用--lb 和錯誤地格式化--group

但由於程序之間的半行混合,甚至-u不能保證它的格式正確:http: //mywiki.wooledge.org/BashPitfalls#Non-atomic_writes_with_xargs_-P

引用自:https://unix.stackexchange.com/questions/443539