sparksql源码系列 | ResolveReferences规则count(*)详解

数据仓库践行者

发布于 2022-06-09 13:30:14

57500

代码可运行

文章被收录于专栏：数据仓库践行者数据仓库践行者

运行总次数：0

代码可运行

本文基于spark 3.2

这篇文章是做上次源码调试分享上留的一个作业题

1、select * from TESTDATA2，分析一下【】的情况，看看是怎么把【】转化为对应字段的。匹配ResolveReferences中的这段代码：case p: Project if containsStar(p.projectList) => p.copy(projectList = buildExpandedProjectList(p.projectList, p.child))

sql:

select * from testdata2

对应astTree:

unresolved logical plan 、resolved Logical Plan 以及这中间用到的规则：

生成resolved Logical Plan用的所有规则一览


==  Parsed Logical Plan  ==
'Project [*]
+- 'UnresolvedRelation [testdata2], [], false


//*********************** 规则1************************
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveRelations
'Project [*]
+- SubqueryAlias testdata2
   +- View (`testData2`, [a#3,b#4])
      +- SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true])).a AS a#3, knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true])).b AS b#4]
         +- ExternalRDD [obj#2]

//*********************** 规则2************************
org.apache.spark.sql.catalyst.analysis.Analyzer$ResolveReferences
Project [a#3, b#4]
+- SubqueryAlias testdata2
   +- View (`testData2`, [a#3,b#4])
      +- SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true])).a AS a#3, knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true])).b AS b#4]
         +- ExternalRDD [obj#2]

== Analyzed Logical Plan ==
Project [a#3, b#4]
+- SubqueryAlias testdata2
   +- View (`testData2`, [a#3,b#4])
      +- SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true])).a AS a#3, knownnotnull(assertnotnull(input[0, org.apache.spark.sql.test.SQLTestData$TestData2, true])).b AS b#4]
         +- ExternalRDD [obj#2]