Fine-grained Fault Tolerance in Distributed Training Toolkits using the Syndicated Actor Model

Rutger De Groen; Tony Garnock-Jones

doi:10.1145/3774902.3776630

Fine-grained Fault Tolerance in Distributed Training Toolkits using the Syndicated Actor Model

Rutger De Groen^*
, Tony Garnock-Jones

^*Corresponding author for this work

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

Abstract

Deep learning (DL) toolkits like TensorFlow and PyTorch provide distributed training strategies, but they have limited support for fault tolerance. They both provide the user with built-in checkpoint functionality to continue training from a restart; however, when a partial failure is detected, all ongoing training jobs are aborted and restarted, repeating work and wasting resources. In response, we explore use of the Syndicated Actor Model (SAM) [5, 10], a recent model of distributed computation, to coordinate activities in a distributed training environment. The SAM encourages expression of distributed systems in terms of the joint state of a shared activity instead of in terms of point-to-point message exchange. This change in perspective has the potential to improve task distribution, resilience to failure, and efficient use of resources in distributed training scenarios.

Original language	English
Title of host publication	MIND 2025 - Proceedings of the 1st International Workshop on Next-Gen Middleware for MLOps in Distributed Systems
Publisher	Association for Computing Machinery, Inc
Pages	1-6
Number of pages	6
ISBN (Electronic)	9798400723056
DOIs	https://doi.org/10.1145/3774902.3776630
Publication status	Published - 14 Dec 2025
Event	1st International Workshop on Next-Gen Middleware for MLOps in Distributed Systems, MIND 2025 - Nashville, usa, Nashville Duration: 15 Dec 2025 → 19 Dec 2025 https://mindmlops.github.io/

Conference

Conference	1st International Workshop on Next-Gen Middleware for MLOps in Distributed Systems, MIND 2025
Abbreviated title	MIND 2025
City	Nashville
Period	15/12/25 → 19/12/25
Other	Welcome to the 1st International workshop on Next-Gen Middleware for MLOps in Distributed Systems (MIND). MIND will be hosted in conjunction with the 26th ACM/IFIP International Middleware Conference conference, which will be held in Vanderbilt University, Nashville, TN, USA from 15th – 19th December 2025. This workshop aims to bring together researchers, practitioners, and industry stakeholders to explore middleware innovations that support MLOps in distributed systems. It will focus on practical solutions to real-world challenges in orchestrating end to end ML pipelines, from data collection to model deployment and continuous monitoring in dynamic, heterogeneous, and resource constrained environments.
Internet address	https://mindmlops.github.io/

Keywords

checkpointing
deep learning
fault tolerance
partial failure
PyTorch
syndicated actor model
TensorFlow

Access to Document

10.1145/3774902.3776630Licence: CC BY

Cite this

@inproceedings{fb7b0cee7de14c4d91d95294d0b7abe3,

title = "Fine-grained Fault Tolerance in Distributed Training Toolkits using the Syndicated Actor Model",

abstract = "Deep learning (DL) toolkits like TensorFlow and PyTorch provide distributed training strategies, but they have limited support for fault tolerance. They both provide the user with built-in checkpoint functionality to continue training from a restart; however, when a partial failure is detected, all ongoing training jobs are aborted and restarted, repeating work and wasting resources. In response, we explore use of the Syndicated Actor Model (SAM) [5, 10], a recent model of distributed computation, to coordinate activities in a distributed training environment. The SAM encourages expression of distributed systems in terms of the joint state of a shared activity instead of in terms of point-to-point message exchange. This change in perspective has the potential to improve task distribution, resilience to failure, and efficient use of resources in distributed training scenarios.",

keywords = "checkpointing, deep learning, fault tolerance, partial failure, PyTorch, syndicated actor model, TensorFlow",

author = "\{De Groen\}, Rutger and Tony Garnock-Jones",

note = "Publisher Copyright: {\textcopyright} 2025 Copyright held by the owner/author(s).; 1st International Workshop on Next-Gen Middleware for MLOps in Distributed Systems, MIND 2025, MIND 2025 ; Conference date: 15-12-2025 Through 19-12-2025",

year = "2025",

month = dec,

day = "14",

doi = "10.1145/3774902.3776630",

language = "English",

pages = "1--6",

booktitle = "MIND 2025 - Proceedings of the 1st International Workshop on Next-Gen Middleware for MLOps in Distributed Systems",

publisher = "Association for Computing Machinery, Inc",

address = "United States",

url = "https://mindmlops.github.io/",

}

De Groen, R & Garnock-Jones, T 2025, Fine-grained Fault Tolerance in Distributed Training Toolkits using the Syndicated Actor Model. in MIND 2025 - Proceedings of the 1st International Workshop on Next-Gen Middleware for MLOps in Distributed Systems. Association for Computing Machinery, Inc, pp. 1-6, 1st International Workshop on Next-Gen Middleware for MLOps in Distributed Systems, MIND 2025, Nashville, 15/12/25. https://doi.org/10.1145/3774902.3776630

Fine-grained Fault Tolerance in Distributed Training Toolkits using the Syndicated Actor Model. / De Groen, Rutger; Garnock-Jones, Tony.
MIND 2025 - Proceedings of the 1st International Workshop on Next-Gen Middleware for MLOps in Distributed Systems. Association for Computing Machinery, Inc, 2025. p. 1-6.

Research output: Chapter in Book/Report/Conference proceeding › Conference article in proceeding › Academic › peer-review

TY - GEN

T1 - Fine-grained Fault Tolerance in Distributed Training Toolkits using the Syndicated Actor Model

AU - De Groen, Rutger

AU - Garnock-Jones, Tony

PY - 2025/12/14

Y1 - 2025/12/14

N2 - Deep learning (DL) toolkits like TensorFlow and PyTorch provide distributed training strategies, but they have limited support for fault tolerance. They both provide the user with built-in checkpoint functionality to continue training from a restart; however, when a partial failure is detected, all ongoing training jobs are aborted and restarted, repeating work and wasting resources. In response, we explore use of the Syndicated Actor Model (SAM) [5, 10], a recent model of distributed computation, to coordinate activities in a distributed training environment. The SAM encourages expression of distributed systems in terms of the joint state of a shared activity instead of in terms of point-to-point message exchange. This change in perspective has the potential to improve task distribution, resilience to failure, and efficient use of resources in distributed training scenarios.

AB - Deep learning (DL) toolkits like TensorFlow and PyTorch provide distributed training strategies, but they have limited support for fault tolerance. They both provide the user with built-in checkpoint functionality to continue training from a restart; however, when a partial failure is detected, all ongoing training jobs are aborted and restarted, repeating work and wasting resources. In response, we explore use of the Syndicated Actor Model (SAM) [5, 10], a recent model of distributed computation, to coordinate activities in a distributed training environment. The SAM encourages expression of distributed systems in terms of the joint state of a shared activity instead of in terms of point-to-point message exchange. This change in perspective has the potential to improve task distribution, resilience to failure, and efficient use of resources in distributed training scenarios.

KW - checkpointing

KW - deep learning

KW - fault tolerance

KW - partial failure

KW - PyTorch

KW - syndicated actor model

KW - TensorFlow

U2 - 10.1145/3774902.3776630

DO - 10.1145/3774902.3776630

M3 - Conference article in proceeding

SP - 1

EP - 6

BT - MIND 2025 - Proceedings of the 1st International Workshop on Next-Gen Middleware for MLOps in Distributed Systems

PB - Association for Computing Machinery, Inc

T2 - 1st International Workshop on Next-Gen Middleware for MLOps in Distributed Systems, MIND 2025

Y2 - 15 December 2025 through 19 December 2025

ER -

Fine-grained Fault Tolerance in Distributed Training Toolkits using the Syndicated Actor Model

Abstract

Conference

Keywords

Access to Document

Fingerprint

Cite this