spark运行过程中打印如下信息:
15/06/23 17:50:57 ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 284 15/06/23 17:50:57
INFO AppClient$ClientActor: Executor added: app-20150623174849-2415/285 on worker-20150520142452-nmg01-khan-spark0024.nmg01.baidu.com-31474 (nmg01-khan-spark0024.nmg01.baidu.com:31474) with 10 cores 15/06/23 17:50:57
INFO SparkDeploySchedulerBackend: Granted executor ID app-20150623174849-2415/285 on hostPort nmg01-khan-spark0024.nmg01.baidu.com:31474 with 10 cores, 50.0 GB RAM 15/06/23 17:50:57
INFO AppClient$ClientActor: Executor updated: app-20150623174849-2415/285 is now LOADING 15/06/23 17:50:57
INFO AppClient$ClientActor: Executor updated: app-20150623174849-2415/285 is now RUNNING 15/06/23 17:50:57
INFO AppClient$ClientActor: Executor updated: app-20150623174849-2415/285 is now EXITED (Command exited with code 1) 15/06/23 17:50:57
INFO SparkDeploySchedulerBackend: Executor app-20150623174849-2415/285 removed: Command exited with code 1 15/06/23 17:50:57
ERROR SparkDeploySchedulerBackend: Asked to remove non-existent executor 285
当集群计算结点出现坏节点或者Executor运行过程中因为OOM等原因挂掉时可能出现Executor Lost的情况。
发现后调度器会将已不存在的executor移除,上述打印信息为正常现象。
坏节点多次重试后会跳过,除非任务调度失败,否则不需要关心这条日志。