(DBInputFormat.java:170) at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:161...(DBInputFormat.java:223) at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java...(DBInputFormat.java:216) ... 11 more Caused by: oracle.net.ns.NetException: Listener refused the connection...(DBInputFormat.java:170) at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:161...(DBInputFormat.java:223) at org.apache.sqoop.mapreduce.db.DBInputFormat.setDbConf(DBInputFormat.java
java.lang.RuntimeException: java.sql.SQLRecoverableException: IO Error: Connection reset at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf...(DBInputFormat.java:164) at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62) at...java.lang.RuntimeException: java.sql.SQLRecoverableException: IO Error: Connection reset at com.cloudera.sqoop.mapreduce.db.DBInputFormat.getConnection...(DBInputFormat.java:190) at com.cloudera.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java:
(DBInputFormat.java:167) 40 at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java...(DBInputFormat.java:220) 53 at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java...(DBInputFormat.java:167) 101 at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java...(DBInputFormat.java:220) 114 at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java...(DBInputFormat.java:220) 175 at org.apache.sqoop.mapreduce.db.DBInputFormat.setConf(DBInputFormat.java
根据解析的数据不同，InputFormat的子类有DBInputFormat、DelegatingInputFormat和FileInputFormat。...其中，DBInputFormat是专门用于加载数据库中的数据的，如mysql、oracle等；FileInputFormat是专门用于处理文件中的数据的；DelegatingInputFormat是把其他各种...从功能可以看出，FileInputFormat是用途最广的，其次是DBInputFormat，再次是DelegatingInputFormat。
好了言归正传，简单的说说背景、原理以及需要注意的地方： 1、为了方便 MapReduce 直接访问关系型数据库（Mysql,Oracle），Hadoop提供了DBInputFormat和DBOutputFormat...通过DBInputFormat类把数据库表数据读入到HDFS，根据DBOutputFormat类把MapReduce产生的结果集导入到数据库表中。...2、由于0.20版本对DBInputFormat和DBOutputFormat支持不是很好，该例用了0.19版本来说明这两个类的用法。...conf.setOutputValueClass(Text.class); conf.setOutputFormat(DBOutputFormat.class); conf.setInputFormat(DBInputFormat.class...192.168.1.101:3306/test", "root", "root"); String[] fields = { "id", "name" }; // 从 t 表读数据 DBInputFormat.setInput
如果是普通文件，可以使用FileInputFormat 如果是SequeceFile（hadoop提供的一种文件格式），可以使用SequnceFileInputFormat 如果处理的数据在数据库中，需要使用DBInputFormat
:19:29 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 17:19:31 INFO db.DBInputFormat...:20:04 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 17:20:06 INFO db.DBInputFormat...:20:22 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 17:20:23 INFO db.DBInputFormat...:21:01 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 17:21:02 INFO db.DBInputFormat...:21:42 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 17:21:43 INFO db.DBInputFormat
读数据：TextInputFormat SequenceFileInputFormat DBInputFormat
:26:23 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 15:26:26 INFO db.DBInputFormat...:26:48 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 15:26:49 INFO db.DBInputFormat...:00:10 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 16:00:12 INFO db.DBInputFormat...:00:35 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 16:00:36 INFO db.DBInputFormat
除了这个格式器以外， 还有KeyValueTextInputFormat, CombineTextInputFormat, SequenceFileInputFormat, DBInputFormat
:03:40 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 15:03:43 INFO db.DBInputFormat...:03:59 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 15:04:01 INFO db.DBInputFormat
当输入为其它格式时，处理逻辑又不相同了，比如当输入格式为DBInputFormat时，会根据输入表的行数（记录数）决定mapper的数量，更多细节可以参考源代码。
:17:20 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/23 16:17:22 INFO db.DBInputFormat...:22:00 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/23 22:22:03 INFO db.DBInputFormat...:43:43 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/23 22:43:45 INFO db.DBInputFormat...:12:52 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/24 11:12:54 INFO db.DBInputFormat
set useSSL=true and provide truststore for server certificate verification. 20/02/07 10:44:21 INFO db.DBInputFormat...set useSSL=true and provide truststore for server certificate verification. 20/02/07 12:19:55 INFO db.DBInputFormat
Instead, use mapreduce.job.maps 18/04/03 10:50:36 INFO db.DBInputFormat: Using read commited transaction...:35:51 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/22 13:35:56 INFO db.DBInputFormat
比如DB输入DBInputFormat，常用的还是FileInputFormat，因为大部分MapReduce job都基于HDFS。
:05:45 INFO client.ConfiguredRMFailoverProxyProvider: Failing over to rm2 18/05/22 17:05:47 INFO db.DBInputFormat
/09/08 01:25:41 INFO mapreduce.ImportJobBase: Beginning import of project 14/09/08 01:25:42 INFO db.DBInputFormat.../09/08 01:58:08 INFO mapreduce.ImportJobBase: Beginning import of project 14/09/08 01:58:08 INFO db.DBInputFormat
client.RMProxy: Connecting to ResourceManager at slaver1/192.168.19.131:8032 18/05/18 19:39:37 INFO db.DBInputFormat
19:25 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 14/10/21 18:19:40 INFO db.DBInputFormat...30:10 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032 14/10/21 18:30:17 INFO db.DBInputFormat
领取专属 10元无门槛券
手把手带您无忧上云
云服务器
ICP备案
对象存储
腾讯会议
实时音视频
