如何將數組拆分為五個文件並並行下載？

當我在. testMachineB_testMachineC``testMachineA``testMachineA

如果該文件不在 中testMachineB，那麼它肯定應該在其中testMachineC。所以我會先嘗試複製文件testMachineB，如果它不在那裡，testMachineB那麼我會去testMachineC複制相同的文件。

PARTITIONS是我需要testMachineA在目錄中複製的文件分區號FOLDER_LOCATION。

#!/bin/bash

readonly FOLDER_LOCATION=/export/home/username/pooking/primary
readonly MACHINES=(testMachineB testMachineC)
PARTITIONS=(0 3 5 7 9 11 13 15 17 19 21 23 25 27 29) # this will have more file numbers around 400

dir1=/data/snapshot/20140317

# delete all the files first
find "$FOLDER_LOCATION" -mindepth 1 -delete
for el in "${PARTITIONS[@]}"
do
   scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 username@${MACHINES[0]}:$dir1/s5_daily_1980_"$el"_200003_5.data $FOLDER_LOCATION/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 username@${MACHINES[1]}:$dir1/s5_daily_1980_"$el"_200003_5.data $FOLDER_LOCATION/.
done

問題陳述：-

現在我要做的是 - 拆分PARTITIONS包含五個文件中的分區號的數組。所以我將複製第一組並行的 5 個文件。一旦這五個文件完成，然後我將移動到下一組，其中還有另外五個文件並再次並行下載它們並繼續這樣做，直到所有文件都完成。

我不想並行下載所有文件，一次只下載五個文件。

使用 bash shell 腳本可以做到這一點嗎？

更新：-

你在建議這樣的事情嗎？

echo $$

readonly FOLDER_LOCATION=/export/home/username/pooking/primary
readonly MACHINES=(testMachineB testMachineC)
ELEMENTS=(0 3 5 7 9 11 13 15 17 19 21 23 25 27 29)
LEN_ELEMENTS=${#ELEMENTS[@]}
X=0

dir1=/data/snapshot/20140317

function download() {
   if [[ $X < $LEN_ELEMENTS ]]; then
       (scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 username@${MACHINES[0]}:$dir1/s5_daily_1980_"${ELEMENTS[$X]}"_200003_5.data $FOLDER_LOCATION/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 username@${MACHINES[1]}:$dir1/s5_daily_1980_"${ELEMENTS[$X]}"_200003_5.data $FOLDER_LOCATION/.) && kill -SIGHUP $$ 2>/dev/null &
   fi
}

trap 'X=$((X+1)); download' SIGHUP

# delete old files
find "$FOLDER_LOCATION" -mindepth 1 -delete

# initial loop
for x in {1..5}
do
   download
done

# waiting loop
while [ $X -lt $LEN_ELEMENTS ]
do
   sleep 1
done

上面看起來對嗎？而且，現在我應該把刪除命令放在哪裡？

像這樣的東西：

# Your variable initialization
readonly FOLDER_LOCATION=/export/home/username/pooking/primary
readonly MACHINES=(testMachineB testMachineC)
PARTITIONS=(0 3 5 7 9 11 13 15 17 19 21 23 25 27 29) # this will have more file numbers around 400

dir1=/data/snapshot/20140317

# delete all the files first
find "$FOLDER_LOCATION" -mindepth 1 -delete

# Bash function to copy a single file based on your script
do_copy() {
 el=$1
 scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[0]}:$dir1/s5_daily_1980_"$el"_200003_5.data $PRIMARY/. || scp -o ControlMaster=auto -o 'ControlPath=~/.ssh/control-%r@%h:%p' -o ControlPersist=900 david@${FILERS_LOCATION[1]}:$dir1/s5_daily_1980_"$el"_200003_5.data $PRIMARY/.
}

# export -f is needed so GNU Parallel can see the function
export -f do_copy

# Run 5 do_copy in parallel. When one finishes, start another.
# Give them each an argument from PRIMARY_PARTITION
parallel -j 5 do_copy ::: "${PRIMARY_PARTITION[@]}"

了解更多：

• 觀看介紹影片以進行快速介紹： https ://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
• 瀏覽本教程（man parallel_tutorial）。你的命令行會愛你。

GNU Parallel 10 秒安裝：

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

引用自：https://unix.stackexchange.com/questions/127834