• 如果您觉得本站非常有看点,那么赶紧使用Ctrl+D 收藏吧

spark Sql语句很长时Driver端出现栈溢出 java.lang.StackOverflowError

SPARK wangting 1个月前 (10-12) 9次浏览

执行类似如下形式的Sql:

select user_id from qt_revert_for_dmp where ((attr[‘535951390’] is not null and get_json_object(attr[‘535951390′],’$.yymmdd’)>20150624) or (attr[‘202697188’] is not null and get_json_object(attr[‘202697188′],’$.yymmdd’)>20150624) or (attr[‘90893145’] is not null and get_json_object(attr[‘90893145′],’$.yymmdd’)>20150624) )…

完整的Sql很长,达到18k。执行报错:

Exception in thread “Thread-228” java.lang.StackOverflowError
at org.apache.spark.sql.hive.HiveQl$.nodeToExpr(HiveQl.scala:1096)
at org.apache.spark.sql.hive.HiveQl$.nodeToExpr(HiveQl.scala:1050)
at org.apache.spark.sql.hive.HiveQl$.nodeToExpr(HiveQl.scala:1050)

at app.ecom.dmp.haystack.ExecSql$$anonfun$run$1.apply$mcVI$sp(QueryEngine.scala:164)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at app.ecom.dmp.haystack.ExecSql.run(QueryEngine.scala:161)

 

Driver 栈溢出,可以在 spark-defaults.conf 里适当调大

spark.driver.extraJavaOptions -Xss=1024m

 


喜欢 (2)