Error reading from database: java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@27ce24aa is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
java.sql.SQLException: Streaming result set com.mysql.jdbc.RowDataDynamic@27ce24aa is still active. No statements may be issued when any streaming result sets are open and in use on a given connection. Ensure that you have called .close() on any active streaming result sets before attempting more queries.
由于mysql-connector-java的bug造成的,出错时我用的是mysql-connector-java-5.1.12-bin.jar,更新成mysql-connector-java-5.1.32-bin.jar就可以了。
但是我通过wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.32.tar.gz,下载了tar.gz包 解压之后得到mysql-connector-java-5.1.32-bin.jar
注意mysql-connector-java-5.1.32-bin.jar,看见没有是-bin.jar,maven项目的pom.xml中添加mysql依赖,会产出mysql-connector-java-5.1.32.jar 没有-bin,注意这是两个不同的包。
千万不要直接从项目中拷过来。。。最后替换掉sqoop目录原有jar包,我的目录是/opt/software/sqoop/lib/
老大看我焦头烂额却没有丝毫进展,告知我要访问国外网站,这种问题要到墙外查阅资料。我也知道,但是最近v**好笑来月经了似的翻不出去了。。。
大哥最后出马,在StackOverFlow上找到了答案:
sqoop export \
--connect $STAT_JDBC_CONNECT \
--username $JDBC_USERNAME \
--password $JDBC_PASSWORD \
--table $TABLE \
--update-mode allowinsert \
--driver com.mysql.jdbc.Driver \
--export-dir $SQOOP_HDFS_RESULT_DAY \
--columns stat_ds,command,answer,counts
我用的工具ShadowSocks,找了半天没有类似的经验,就把电脑关机。重启之后在没有改任何配置的前提下就好了!
/**
* 统计音箱型号分布情况
*
* @param dataFrame
* @return
*/
def echoModelDistributeStat(dataFrame: DataFrame, preDay: String): List[String] = {
var numdf = dataFrame.groupBy("model").agg(count("model").as("model_count"))
// 统计出总共的设备数
var sumdf = numdf.groupBy().agg(sum("model_count"))
// 计算设备型号的总数量
var modelSum = sumdf.collect()(0)(0)
// 计算出需要的的DF
var modelInfo = numdf.select(numdf("model"), numdf("model_count"), (numdf("model_count") / modelSum * 100).as("model_percent"))
modelInfo.registerTempTable("m")
// 将日期转换为时间戳
val timeStamp = TimeUtils.getTimeStamp(preDay + " " + Constant.TIME_ZERO)
val timeDf = modelInfo.sqlContext.sql("select " + timeStamp + " as stat_ds,m.model,m.model_count,round(m.model_percent,2) as model_percent from m")
// 将DF转成List[Row]的形式
val list = timeDf.collect().toList
// 将List[Row]转为List[String]
val listStr = listRowConvertListStr(list)
return listStr
}