How do you handle node failures during a massive, multi-week distributed training job without losing significant progress?
