今天发现测试环境的kerberos Hadoop的hive不能跑了，具体表现是select * limit这种不走mapred的job是ok的，走mapred的job就会报错，报的错比较奇怪（Unable...(LinuxTaskController.java:194) at org.apache.hadoop.mapred.TaskTracker$4.run(TaskTracker.java...(TaskTracker.java:1395) at org.apache.hadoop.mapred.TaskTracker.localizeJob(TaskTracker.java:...1310) at org.apache.hadoop.mapred.TaskTracker.startNewTask(TaskTracker.java:2727) at...initializeJob方法是对job做初始操作，传入user,jobid,token,mapred的local dir等参数，生成一个数组，并调用ShellCommandExecutor的构造方法进行实例化
hadoop-mapred-site.xml配置文件详解 mapred-site.xml配置文件简介 MapReduce配置相关 mapred-site.xml配置 name value description...Be sure to specify a default value in mapred-site.xml if the underlying filesystem is not HDFS....This must match one of the queues defined in mapred-queues.xml for the system....For Linux: $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/, $HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib...For Windows: %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce/, %HADOOP_MAPRED_HOME%/share/hadoop/mapreduce
记录一次错误： 环境：CDH5.10 jdk8 hive query 时，报错org.apache.hadoop.mapred.YarnChild: Error running child...并没有这个参数，却有mapreduce.map.java.opts， mapreduce.map.java.opts会覆盖掉 mapreduce.map.java.opts.max.heap 4.还会看到mapred.map
执行hadoop classpath 在hadoop配置文件 yarn-site中添加以下配置 <property> <name>yarn.applicatio...
.* FROM `tb_user` AS t LIMIT 1 17 18/05/18 19:32:52 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is..... using builtin-java classes where applicable 27 18/05/18 19:33:04 INFO Configuration.deprecation: mapred.jar...Instead, use mapreduce.job.jar 28 18/05/18 19:33:07 INFO Configuration.deprecation: mapred.map.tasks...statement: SELECT t.* FROM `tb_user` AS t LIMIT 1 18/05/18 19:39:27 INFO orm.CompilationManager: HADOOP_MAPRED_HOME...Instead, use mapreduce.job.jar 18/05/18 19:39:31 INFO Configuration.deprecation: mapred.map.tasks is
最近开始学习mapreduce。小菌在第一次写mapreduce的时候就遇到了一个新手经常遇到的坑,具体的bug见下图
; import org.apache.hadoop.mapred.JobConf; import org.apache.hadoop.mapred.MapReduceBase; import org.apache.hadoop.mapred.Mapper...; import org.apache.hadoop.mapred.OutputCollector; import org.apache.hadoop.mapred.Reducer; import org.apache.hadoop.mapred.Reporter...mapred.MapTask: Starting flush of map output 13/07/29 02:33:03 INFO mapred.MapTask: Finished spill 0...02:33:06 INFO mapred.LocalJobRunner: 13/07/29 02:33:06 INFO mapred.Task: Task 'attempt_local_0001_m..._000000_0' done. 13/07/29 02:33:06 INFO mapred.LocalJobRunner: 13/07/29 02:33:06 INFO mapred.Merger:
1.mapred.task.timeout 任务超时时间，默认是10分钟 2.mapred.map.max.attempts mapred.reduce.max.attempts...默认任务失败重复次数为4 3.mapred.max.map.failures.percent mapred.reduce.map.failures.percent 不触发错误的失败的最大百分比...tasktracker用于默认为40的啦 9.mapred.reduce.parallel.copies reduce复制map输出的线程数，默认是5个 10.mapred.inmem.merge.threshold...mapred.reduce.tasks.speculative.execution 推测执行默认值为true， 15.mapred.job.reuse.jvm.num.tasks...，防止因为内存溢出导致的错误 17.SkipBadRecord开启跳跃模式，跳过失败的坏记录 mapred.map.max.attemps mapred.reduce.attemps 一次只能跳过一个错误记录
:19:29 INFO mapred.MapTask: record buffer = 262144/327680 line:11 15/01/28 21:19:29 INFO mapred.MapTask...mapred.TaskRunner: Task 'attempt_local_0001_m_000000_0' done. 15/01/28 21:19:29 INFO mapred.MapTask:...mapred.TaskRunner: Task 'attempt_local_0001_m_000001_0' done. 15/01/28 21:19:29 INFO mapred.MapTask:...mapred.MapTask: data buffer = 79691776/99614720 15/01/28 21:19:30 INFO mapred.MapTask: record buffer.../01/28 21:19:31 INFO mapred.JobClient: Map-Reduce Framework 15/01/28 21:19:31 INFO mapred.JobClient
/08 01:25:52 INFO mapred.JobClient: map 66% reduce 0% 14/09/08 01:25:53 INFO mapred.JobClient: map...01:25:54 INFO mapred.JobClient: Counters: 18 14/09/08 01:25:54 INFO mapred.JobClient: Job Counters 14...01:25:54 INFO mapred.JobClient: File Output Format Counters 14/09/08 01:25:54 INFO mapred.JobClient...:54 INFO mapred.JobClient: Map-Reduce Framework 14/09/08 01:25:54 INFO mapred.JobClient: Map input...09/08 01:25:54 INFO mapred.JobClient: Map output records=3 14/09/08 01:25:54 INFO mapred.JobClient
/15 15:23:15 INFO mapred.JobClient: map 100% reduce 0% 14/07/15 15:23:30 INFO mapred.JobClient: map...mapred.JobClient: FileSystemCounters 14/07/15 15:23:32 INFO mapred.JobClient: FILE_BYTES_READ=...07/15 15:23:32 INFO mapred.JobClient: Spilled Records=262 14/07/15 15:23:32 INFO mapred.JobClient...:32 INFO mapred.JobClient: Combine input records=179 14/07/15 15:23:32 INFO mapred.JobClient:...23:32 INFO mapred.JobClient: Reduce input groups=131 14/07/15 15:23:32 INFO mapred.JobClient:
: Running job: job_201209101835_0001 12/09/10 18:35:54 INFO mapred.JobClient: map 0% reduce 0%...12/09/10 18:36:17 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_000000_0, Status :...(Child.java:249) 12/09/10 18:36:20 INFO mapred.JobClient: Task Id : attempt_201209101835_0001_m_...12/09/10 18:37:12 INFO mapred.JobClient: Counters: 6 12/09/10 18:37:12 INFO mapred.JobClient:.../09/10 18:37:12 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 12/09/10 18:37:12 INFO mapred.JobClient
property> fs.defaultFS hdfs://node1:9000 保存退出 2.3.5 修改mapred-site.xml...文件 默认情况下该目录下没有mapred-site.xml文件，而是存在一个模版文件mapred-site.xml.template，因此首先将模版文件复制一份为mapred-site.xml。...然后在通过vi编辑该mapred-site.xml文件 cp mapred-site.xml.template mapred-site.xml vi mapred-site.xml [root@...node1 hadoop]# cp mapred-site.xml.template mapred-site.xml [root@node1 hadoop]# vi mapred-site.xml
15 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 15/01/2410:30:15 INFO mapred.MapTask: soft limit...INFO mapred.MapTask: kvstart = 26214396; length = 6553600 15/01/2410:30:15 INFO mapred.MapTask: Map...= 17/6553600 15/01/2410:30:15 INFO mapred.MapTask: Finished spill 0 15/01/2410:30:15 INFO mapred.Task...INFO mapred.MapTask: kvstart = 26214396; length = 6553600 15/01/2410:30:15 INFO mapred.MapTask: Map...= 17/6553600 15/01/2410:30:15 INFO mapred.MapTask: Finished spill 0 15/01/2410:30:15 INFO mapred.Task
/18 10:12:32 INFO mapred.JobClient: map 100% reduce 0% 14/10/18 10:12:39 INFO mapred.JobClient: map...mapred.JobClient: Job complete: job_201410181754_0004 14/10/18 10:12:40 INFO mapred.JobClient: Counters...:40 INFO mapred.JobClient: Map input records=2 14/10/18 10:12:40 INFO mapred.JobClient: Reduce...mapred.JobClient: Map output bytes=78 14/10/18 10:12:40 INFO mapred.JobClient: CPU time spent...10:12:40 INFO mapred.JobClient: Reduce input groups=4 14/10/18 10:12:40 INFO mapred.JobClient:
INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100 15/05/27 17:20:00 INFO mapred.MapTask: soft limit...00 INFO mapred.MapTask: kvstart = 26214396; length = 6553600 15/05/27 17:20:00 INFO mapred.MapTask: Map...: 15/05/27 17:20:01 INFO mapred.MapTask: Starting flush of map output 15/05/27 17:20:01 INFO mapred.MapTask...01 INFO mapred.Task: Task 'attempt_local602619796_0001_m_000000_0' done. 15/05/27 17:20:01 INFO mapred.LocalJobRunner.../05/27 17:20:01 INFO mapred.Task: Using ResourceCalculatorProcessTree : [ ] 15/05/27 17:20:01 INFO mapred.ReduceTask
/26 04:28:01 INFO mapred.JobClient: map 29% reduce 0% 15/04/26 04:28:07 INFO mapred.JobClient: map...mapred.JobClient: map 100% reduce 0% 15/04/26 04:28:49 INFO mapred.JobClient: map 100% reduce 100%...mapred.JobClient: Counters: 29 15/04/26 04:28:50 INFO mapred.JobClient: Job Counters 15/04/26 04:28...:28:50 INFO mapred.JobClient: Data-local map tasks=1 15/04/26 04:28:50 INFO mapred.JobClient:...04:28:50 INFO mapred.JobClient: HDFS_BYTES_READ=61084325 15/04/26 04:28:50 INFO mapred.JobClient
mapred.MapTask: bufstart = 0; bufvoid = 104857600 2022-04-04 19:38:21,711 INFO mapred.MapTask: kvstart...= org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2022-04-04 19:38:21,771 INFO mapred.LocalJobRunner...INFO mapred.MapTask: Spilling map output 2022-04-04 19:38:21,771 INFO mapred.MapTask: bufstart = 0; bufend...mapred.MapTask: bufstart = 0; bufvoid = 104857600 2022-04-04 19:38:21,955 INFO mapred.MapTask: kvstart...= org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2022-04-04 19:38:21,963 INFO mapred.LocalJobRunner
就需要使用hadoop中的org.apache.hadoop.mapred.lib.KeyFieldBasedPartitioner了。...比如上面的需求用mapred.text.key.partitioner.options表示为 mapred.text.key.partitioner.options=-k1,1...=-k2,3 \ -jobconf mapred.reduce.tasks=5 结果： e.9.4 5...\ -jobconf mapred.output.key.comparator.class=org.apache.hadoop.mapred.lib.KeyFieldBasedComparator...-jobconf mapred.reduce.tasks=5 结果： e.5.1.45 e.5.1.23
mapred.job.name="xxx_test" 2. Reduce的个数 说明1 mapred.reduce.tasks 3....Job的task并发数 # mapper mapred.job.max.map.running # reducer mapred.job.max.reduce.running 4....设置Job的失败比例 # mapper mapred.max.map.failures.percent # reducer mapred.max.reduce.failures.percent 5...设置task的超时时间 mapred.task.timeout # （单位：毫秒） 7....预测执行 mapred.map.tasks.speculative.execution // 默认打开为true，关闭请显式设置为false mapred.reduce.tasks.speculative.execution
领取专属 10元无门槛券
手把手带您无忧上云
云服务器
ICP备案
对象存储
即时通信 IM
腾讯会议
