我正试图扫描一个IP块,总共有6.5万个地址。我们已经被指示使用带有bash的ICMP数据包,并找到一种并行化的方法。以下是我想出的:
#!/bin/bash
ping() {
if ping -c 1 -W 5 131.212.$i.$j >/dev/null
then
((++s))
echo -n "*"
else
((++f))
echo -n "."
fi
((++j))
#if j has reached 255, set it to zero and increment i
if [ $j -gt 255 ]; then
j=0
((++i))
echo "Pinging 131.212.$i.xx IP Block...\n"
fi
}
s=0 #number of responses recieved
f=0 #number of failures recieved
i=0 #IP increment 1
j=0 #IP increment 2
curProcs=$(ps | wc -l)
maxProcs=$(getconf OPEN_MAX)
while [ $i -lt 256 ]; do
curProcs=$(ps | wc -l)
if [ $curProcs -lt $maxProcs ]; then
ping &
else
sleep 10
fi
done
echo "Found "$s" responses and "$f" timeouts."
echo /usr/bin/time -l
done但是,我遇到了以下错误(在macOS上):
redirection error: cannot duplicate fd: Too many open files我的理解是,我正在讨论一个资源限制,如果现有进程的数量少于指定的最大值,我尝试通过启动新的ping进程来纠正这个限制,但这不能解决问题。
谢谢您的时间和建议。
编辑:下面有很多很好的建议,可以用已有的工具来完成。由于我受到学术要求的限制,我最终将ping循环分割成一个不同的过程,用于每个12.34.x.x块,尽管这个过程很难看,但在不到5分钟的时间内就成功了。这段代码有很多问题,但对于将来的一些人来说,这可能是一个很好的起点:
#!/bin/bash
#############################
# Ping Subfunction #
#############################
# blocks with more responses will complete first since worst-case scenerio
# is O(n) if no IPs generate a response
pingSubnet() {
for ((j = 0 ; j <= 255 ; j++)); do
# send a single ping with a timeout of 5 sec, piping output to the bitbucket
if ping -c 1 -W 1 131.212."$i"."$j" >/dev/null
then
((++s))
else
((++f))
fi
done
#echo "Recieved $s responses with $f timeouts in block $i..."
# output number of success results to the pipe opened in at the start
echo "$s" >"$pipe"
exit 0
}
#############################
# Variable Declaration #
#############################
start=$(date +%s) #start of execution time
startMem=$(vm_stat | awk '/Pages free/ {print $3}' | awk 'BEGIN { FS = "\." }; {print ($1*0.004092)}' | sed 's/\..*$//');
startCPU=$(top -l 1 | grep "CPU usage" | awk '{print 100-$7;}' | sed 's/\..*$//')
s=0 #number of responses recieved
f=0 #number of failures recieved
i=0 #IP increment 1
j=0 #IP increment 2
#############################
# Pipe Initialization #
#############################
# create a pipe for child procs to write to
# child procs inherit runtime environment of parent proc, but cannot
# write back to it (like passing by value in C, but the whole env)
# hence, they need somewhere else to write back to that the parent
# proc can read back in
pipe=/tmp/pingpipe
trap 'rm -f $pipe' EXIT
if [[ ! -p $pipe ]]; then
mkfifo $pipe
exec 3<> $pipe
fi
#############################
# IP Block Iteration #
#############################
# adding an ampersand to the end forks the command to a separate, backgrounded
# child process. this allows for parellel computation but adds logistical
# challenges since children can't write the parent's variables
echo "Initiating scan processes..."
while [ $i -lt 256 ]; do
#echo "Beginning 131.212.$i.x block scan..."
#ping subnet asynchronously
pingSubnet &
((++i))
done
echo "Waiting for scans to complete (this may take up to 5 minutes)..."
peakMem=$(vm_stat | awk '/Pages free/ {print $3}' | awk 'BEGIN { FS = "\." }; {print ($1*0.004092)}' | sed 's/\..*$//')
peakCPU=$(top -l 1 | grep "CPU usage" | awk '{print 100-$7;}' | sed 's/\..*$//')
wait
echo -e "done" >$pipe
#############################
# Concat Pipe Outputs #
#############################
# read each line from the pipe we created earlier, adding the number
# of successes up in a variable
success=0
echo "Tallying responses..."
while read -r line <$pipe; do
if [[ "$line" == 'done' ]]; then
break
fi
success=$((line+success))
done
#############################
# Output Statistics #
#############################
echo "Gathering Statistics..."
fail=$((65535-success))
#output program statistics
averageMem=$((peakMem-startMem))
averageCPU=$((peakCPU-startCPU))
end=$(date +%s) #end of execution time
runtime=$((end-start))
echo "Scan completed in $runtime seconds."
echo "Found $success active servers and $fail nonresponsive addresses with a timeout of 1."
echo "Estimated memory usage was $averageMem MB."
echo "Estimated CPU utilization was $averageCPU %"发布于 2017-10-01 09:52:55
这将为您提供一些关于GNU并行的想法。
parallel --dry-run -j 64 -k ping 131.212.{1}.{2} ::: $(seq 1 3) ::: $(seq 11 13)
ping 131.212.1.11
ping 131.212.1.12
ping 131.212.1.13
ping 131.212.2.11
ping 131.212.2.12
ping 131.212.2.13
ping 131.212.3.11
ping 131.212.3.12
ping 131.212.3.13-j64一次并行执行64个ping。-dry-run的意思是什么都不做,只是展示它会做什么。-k的意思是保持输出有序-(只是为了让你理解它):::引入了这些参数,我用不同的数字(1到3,然后11到13)重复它们,这样您就可以区分这两个计数器,并看到所有的排列和组合都生成了。
发布于 2017-10-01 01:51:53
发布于 2017-10-01 06:54:08
当然,它并不像上面试图构建的那样最优,但是您可以在后台启动最大允许数量的进程,等待它们结束并启动下一批进程,如下所示(但我使用的是sleep 1s):
for i in {1..20} # iterate some
do
sleep 1 & # start in the background
if ! ((i % 5)) # after every 5th (using mod to detect)
then
wait %1 %2 %3 %4 %5 # wait for all jobs to finish
fi
donehttps://stackoverflow.com/questions/46508532
复制相似问题