At the IT Press Tour, Clockwork.io cut network and power connections on a live training job. The company says LinkPass and TorchPass keep such jobs running; a $31 million round funds the rollout.
When a single GPU or network link among thousands fails, an AI training job usually stops completely. Clockwork.io wants software to absorb such faults. At the IT Press Tour, the company showed live how: engineers pulled a network cable and then a power cord on a running training job.
The cost of failure at scale
Large training jobs run in lockstep, so one failed link, an overheated GPU or a crashed node halts every participant. CEO Suresh Vasudevan cited a Meta study that puts the mean time to failure for 1,024-GPU jobs at 7.9 hours, and a ByteDance study in which 42.5 percent of jobs ran at least ten percent slower because of stragglers. After a crash, teams must locate the fault, which by hand can take one to three hours, restart the job, reload the last checkpoint and recompute the lost work.
From clock synchronization to fault tolerance
The company grew out of a Stanford project on clock synchronization led by Balaji Prabhakar and Yilong Geng, with VMware founder Mendel Rosenblum involved early on. According to the company, the software reaches an accuracy in the tens of nanoseconds without special hardware, which won financial customers such as Nasdaq and Wells Fargo. Since 2022 the same clocks have served network telemetry in GPU clusters; the FleetIQ platform became generally available in September 2025.
Three layers of protection
LinkPass, an NCCL plugin, reacts first. If a network card fails, it moves the traffic, including packets in flight, to the healthy cards of the same host and switches back once the link recovers. Detection takes hundreds of milliseconds and the whole switchover one to two seconds, whereas the NCCL transport timeout is about a minute and the PyTorch default ten minutes. In the demo, a pulled cable on a two-node job caused only a brief dip; the step counter kept climbing, and after reconnection the traffic spread across all four cards again. The demo ran live in the company’s lab, not from a recording. Vasudevan puts the share of failures caused by links at 15 to 20 percent. LinkedIn reports tens of thousands of GPU hours saved per month, and Oracle embeds LinkPass in its cloud. Dan Zheng, Chief Business Officer, added an operational benefit: switch maintenance used to require warning job owners so they could checkpoint first, whereas it can now run independently of the jobs.
TorchPass covers failing GPUs and nodes through live migration to a spare, which Vasudevan compared to vMotion. It quiesces the job at a consistent point, requests a replacement from the scheduler (Kubernetes, Slurm and variants) and transfers the state. For sudden failures it rebuilds the state from a data-parallel replica, so one must exist. When the team cut a node’s power, the job paused for about a minute at step 5,040 and resumed at the same step. Spares typically make up five to eight percent of a cluster and run low-priority work until needed. Beyond failures, migration can also move jobs off nodes for security patching or to remove stragglers, and can act on warnings such as rising GPU temperatures.
The third layer, announced as the industry’s first multi-node platform snapshots, captures CPU and GPU state without code changes. The company states they take under 20 seconds and cost about one percent at a 25-minute interval, but they need substantial storage, and restore times have not yet been benchmarked. Together AI plans to offer LinkPass and TorchPass as a service to its customers, with snapshots to follow.
Figures to treat with care
In a company benchmark, a Llama 4 MoE model on 64 GPUs needed about 400 minutes without faults. With hourly injected failures it took about 400 minutes using TorchPass and about 800 with checkpoint restart. SemiAnalysis’s ClusterMAX calculator, run on default values for a 4,096-GPU cluster, shows goodput loss falling from 13.7 to 2.8 percent in a gold tier and from 28.5 to 10.3 percent in a silver tier. These are vendor-supplied scenarios, not field measurements.
Observability and business model
FleetLens sends probes between all network card pairs every ten milliseconds and measures one-way delay with synchronized clocks. Triangulation then names the faulty link or rules out the network. FleetAudit applies this before handover; WhiteFiber uses it to validate clusters and says days of testing shrink to minutes. In a pilot at a large GPU cloud, switch counters averaged over 30 to 60 seconds looked normal, while the probes exposed one faulty leaf-to-spine link affecting a single job.
Pricing is per GPU per year. Neoclouds buy directly, while enterprises increasingly come through partners such as Together AI. NVIDIA and AMD GPUs, InfiniBand and RoCE are supported; TPUs and Trainium are not. Vasudevan sees the main rival in in-house checkpoint-restore setups and said the company credits customers if fewer than 90 percent of failures are recovered without recomputation. A $31 million round co-led by Premji Invest, Wing and Seligman Ventures brings total funding to $73 million. “Efficiency is the new capacity,” Zheng said.
Dr. Jakob Jung is Editor-in-Chief of Security Storage and Channel Germany. He has been working in IT journalism for more than 20 years. His career includes Computer Reseller News, Heise Resale, Informationweek, Techtarget (storage and data center) and ChannelBiz. He also freelances for numerous IT publications, including Computerwoche, Channelpartner, IT-Business, Storage-Insider and ZDnet. His main topics are channel, storage, security, data center, ERP and CRM.
Contact via Mail: jakob.jung@security-storage-und-channel-germany.de