• 如果您觉得本站非常有看点,那么赶紧使用Ctrl+D 收藏吧

spark作业提交报InvalidClassException

SPARK wangting 1个月前 (10-12) 10次浏览

启动Spark作业报如下错误:

ERROR Remoting: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID = 1677335532749418220, local class serialVersionUID = -7366074099953117729
java.io.InvalidClassException: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID = 1677335532749418220, local class serialVersionUID = -7366074099953117729
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:560)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1599)

 

pyspark报:

in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o42.saveAsTextFile.: org.apache.spark.SparkException: Job aborted due to stage failure: Task 9 in stage 0.0 failed 4 times, most recent failure: Lost task 9.3 in stage 0.0 (TID 30, nmg01-taihang-d10196.nmg01.baidu.com): java.io.InvalidClassException: org.apache.spark.SparkHadoopWriter; local class incompatible: stream classdesc serialVersionUID = 4392444616715414383, local class serialVersionUID = 4757876622451827520

 

上述两种错误原因相同,在Standalone模式下发生。

一般均由用户使用Spark客户端同集群客户端版本不一致导致。

 

 

 


喜欢 (2)