启动Spark作业报如下错误:
ERROR Remoting: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID = 1677335532749418220, local class serialVersionUID = -7366074099953117729
java.io.InvalidClassException: org.apache.spark.storage.BlockManagerId; local class incompatible: stream classdesc serialVersionUID = 1677335532749418220, local class serialVersionUID = -7366074099953117729
at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:560)
at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1599)
pyspark报:
in get_return_value py4j.protocol.Py4JJavaError: An error occurred while calling o42.saveAsTextFile.: org.apache.spark.SparkException: Job aborted due to stage failure: Task 9 in stage 0.0 failed 4 times, most recent failure: Lost task 9.3 in stage 0.0 (TID 30, nmg01-taihang-d10196.nmg01.baidu.com): java.io.InvalidClassException: org.apache.spark.SparkHadoopWriter; local class incompatible: stream classdesc serialVersionUID = 4392444616715414383, local class serialVersionUID = 4757876622451827520
上述两种错误原因相同,在Standalone模式下发生。
一般均由用户使用Spark客户端同集群客户端版本不一致导致。