• 如果您觉得本站非常有看点,那么赶紧使用Ctrl+D 收藏吧

spark 运行时报 Asked to remove non-existent executor

SPARK wangting 1个月前 (10-12) 12次浏览

spark运行过程中打印如下信息:

15/06/23 17:50:57 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 284 15/06/23 17:50:57
INFO AppClient$ClientActor: Executor added: app-20150623174849-2415/285 on worker-20150520142452-nmg01-khan-spark0024.nmg01.baidu.com-31474 (nmg01-khan-spark0024.nmg01.baidu.com:31474) with 10 cores 15/06/23 17:50:57
INFO SparkDeploySchedulerBackend: Granted executor ID app-20150623174849-2415/285 on hostPort nmg01-khan-spark0024.nmg01.baidu.com:31474 with 10 cores, 50.0 GB RAM 15/06/23 17:50:57
INFO AppClient$ClientActor: Executor updated: app-20150623174849-2415/285 is now LOADING 15/06/23 17:50:57
INFO AppClient$ClientActor: Executor updated: app-20150623174849-2415/285 is now RUNNING 15/06/23 17:50:57
INFO AppClient$ClientActor: Executor updated: app-20150623174849-2415/285 is now EXITED (Command exited with code 1) 15/06/23 17:50:57
INFO SparkDeploySchedulerBackend: Executor app-20150623174849-2415/285 removed: Command exited with code 1 15/06/23 17:50:57
ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 285

集群计算结点出现坏节点或者Executor运行过程中因为OOM等原因挂掉时可能出现Executor Lost的情况。

发现后调度器会将已不存在的executor移除,上述打印信息为正常现象。

坏节点多次重试后会跳过,除非任务调度失败,否则不需要关心这条日志。 

 


喜欢 (2)