如何使用Python为Hadoop编写一个简单的MapReduce程序

#!/usr/bin/env python
 
import sys
 
# input comes from STDIN (standard input)
for line in sys.stdin:
# remove leading and trailing whitespace
    line = line.strip()
# split the line into words
    words = line.split()
# increase counters
for word in words:
# write the results to STDOUT (standard output);
# what we output here will be the input for the
# Reduce step, i.e. the input for reducer.py
#
# tab-delimited; the trivial word count is 1
print '%s\\t%s' % (word, 1)

#!/usr/bin/env python
 
from operator import itemgetter
import sys
 
# maps words to their counts
word2count = {}
 
# input comes from STDIN
for line in sys.stdin:
# remove leading and trailing whitespace
    line = line.strip()
 
# parse the input we got from mapper.py
    word, count = line.split('\\t', 1)
# convert count (currently a string) to int
try:
        count = int(count)
        word2count[word] = word2count.get(word, 0) + count
except ValueError:
# count was not a number, so silently
# ignore/discard this line
pass
 
# sort the words lexigraphically;
#
# this step is NOT required, we just do it so that our
# final output will look more like the official Hadoop
# word count examples
sorted_word2count = sorted(word2count.items(), key=itemgetter(0))
 
# write the results to STDOUT (standard output)
for word, count in sorted_word2count:
print '%s\\t%s'% (word, count)

——————————————————————————————————————————————
 # very basic test
 hadoop@ubuntu:~$ echo "foo foo quux labs foo bar quux" | /home/liupeng/hadoop/mapper.py
 foo     1
 foo     1
 quux    1
 labs    1
 foo     1
 bar     1
——————————————————————————————————————————————
 hadoop@ubuntu:~$ echo "foo foo quux labs foo bar quux" | /home/liupeng/hadoop/mapper.py | sort | /home/liupeng/hadoop/reducer.py
 bar     1
 foo     3
 labs    1
——————————————————————————————————————————————

 # using one of the ebooks as example input
 # (see below on where to get the ebooks)
 hadoop@ubuntu:~$ cat /tmp/liupeng/20417-8.txt | /home/liupeng/hadoop/mapper.py
 The     1
 Project 1
 Gutenberg       1
 EBook   1
 of      1
 [...] 
 (you get the idea)

 quux    2

 quux    1  

—————————————————————————————————————————————

 hadoop@ubuntu:~$ ls -l /tmp/liupeng/
 total 3592
 -rw-r--r-- 1 hadoop hadoop  674425 2007-01-22 12:56 20417-8.txt
 -rw-r--r-- 1 hadoop hadoop 1423808 2006-08-03 16:36 7ldvc10.txt
 -rw-r--r-- 1 hadoop hadoop 1561677 2004-11-26 09:48 ulyss12.txt
 hadoop@ubuntu:~$

复制本地数据到HDFS

在我们运行MapReduce job 前，我们需要将本地的文件复制到HDFS中：

 hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop dfs -copyFromLocal /tmp/liupeng liupeng
 hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop dfs -ls
 Found 1 items
 /user/hadoop/liupeng  <dir>
 hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop dfs -ls liupeng
 Found 3 items
 /user/hadoop/gutenberg/20417-8.txt      <r 1>   674425
 /user/hadoop/gutenberg/7ldvc10.txt      <r 1>   1423808
 /user/hadoop/gutenberg/ulyss12.txt      <r 1>   1561677

执行 MapReduce job

现在，一切准备就绪，我们将在运行Python MapReduce job 在Hadoop集群上。像我上面所说的，我们使用的是
HadoopStreaming 帮助我们传递数据在Map和Reduce间并通过STDIN和STDOUT，进行标准化输入输出。


hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop jar share/hadoop/tools/lib/hadoop-streaming-2.8.1.jar 
    -mapper /home/liupeng/hadoop/mapper.py -reducer /home/liupeng/hadoop/reducer.py 
    -input liupeng/* -output liupeng-output
在运行中，如果你想更改Hadoop的一些设置，如增加Reduce任务的数量，你可以使用“-jobconf”选项： 
hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop jar share/hadoop/tools/lib/hadoop-streaming-2.8.1.jar
   

hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop jar share/hadoop/tools/lib/hadoop-streaming-2.8.1.jar 
    -mapper /home/liupeng/hadoop/mapper.py -reducer /home/liupeng/hadoop/reducer.py 
    -input liupeng/* -output liupeng-output

 hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop dfs -ls liupeng-output
 Found 1 items
 /user/hadoop/liupeng-output/part-00000     <r 1>   903193  2007-09-21 13:00
 hadoop@ubuntu:/usr/local/hadoop$ 

 hadoop@ubuntu:/usr/local/hadoop$ bin/hadoop dfs -cat liupeng-output/part-00000
 "(Lo)cra"       1
 "1490   1
 "1498," 1
 "35"    1
 "40,"   1
 "A      2
 "AS-IS".        2
 "A_     1
 "Absoluti       1
 [...]
 hadoop@ubuntu:/usr/local/hadoop$

请参考:
(1) http://www.cnblogs.com/end/archive/2012/08/13/2636175.html
(2) https://www.bbsmax.com/A/nAJvA3eQzr/
(3) https://www.digitalocean.com/community/tutorials/how-to-install-hadoop-in-stand-alone-mode-on-ubuntu-16-04