执行类似如下形式的Sql:
select user_id from qt_revert_for_dmp where ((attr[‘535951390’] is not null and get_json_object(attr[‘535951390′],’$.yymmdd’)>20150624) or (attr[‘202697188’] is not null and get_json_object(attr[‘202697188′],’$.yymmdd’)>20150624) or (attr[‘90893145’] is not null and get_json_object(attr[‘90893145′],’$.yymmdd’)>20150624) )…
完整的Sql很长,达到18k。执行报错:
Exception in thread “Thread-228” java.lang.StackOverflowError
at org.apache.spark.sql.hive.HiveQl$.nodeToExpr(HiveQl.scala:1096)
at org.apache.spark.sql.hive.HiveQl$.nodeToExpr(HiveQl.scala:1050)
at org.apache.spark.sql.hive.HiveQl$.nodeToExpr(HiveQl.scala:1050)
…
at app.ecom.dmp.haystack.ExecSql$$anonfun$run$1.apply$mcVI$sp(QueryEngine.scala:164)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
at app.ecom.dmp.haystack.ExecSql.run(QueryEngine.scala:161)
Driver 栈溢出,可以在 spark-defaults.conf 里适当调大
spark.driver.extraJavaOptions -Xss=1024m