Closing the Reliability Gap Through Failure & Recovery

Introduction

For physical AI to work in the real world, robots need to be reliable. Everyone is chasing deployments, but a policy that performs well under ideal conditions in a lab or in simulation is not necessarily ready for a real workflow. When it encounters an edge case, it needs to recover and continue the task. These interventions create data for robotics post-training, allowing us to further train a policy on failures it encounters during its own rollouts and human demonstrations of how to recover.

At Northstar, we see reliability as one of the biggest bottlenecks to deployment. We believe that means supporting learning across the entire lifecycle from initial pre-training demonstrations to post-training on failures and recoveries, and continued improvement after deployment.

Human intervention supports deployment before a policy achieves full autonomy. Rather than expecting 100% autonomy right away, we believe that human intervention is a way to build towards autonomy and collect the exact data needed to improve the policy after it has been deployed. We saw this parallel with autonomous vehicles as well. For robotics, a remote teleoperator monitors the robot, freezes it once the policy encounters a failure or is about to fail, the operator takes over and puts it back on track/recovers and the policy can continue on.

We see the operator’s role evolving as the policy improves. From collecting demonstrations in the lab to intervening during policy rollouts, capturing edge cases, and demonstrating recoveries. Once the policy is reliable enough for its intended workflow, deployment can begin with continued monitoring and intervention support. Using VR teleoperation, operators can continue collecting recovery data in the exact environment where the robot works.

This led us to the question we wanted to test: how does post-training on recovery demonstrations affect a policy’s success rate and what happens when you only train a policy on successful (“happy path”) demonstrations?

Evaluations:

We tested Physical Intelligence’s π₀.₅ model across 2 tasks: packing toiletries into the basket and stacking three cubes in an orange-blue-orange tower. We recorded 100 clean demonstrations, finetuned the base policies, then evaluated the policy after 100 demonstrations, taking note of success/failure rate, recovery rate, and what the most common failure modes were.

Afterwards, we collected an additional 50 clean demonstrations and an additional 50 recovery demonstrations. We posttrained two seprate finetuned policies on these demonstrations. These recovery demonstrations happened when we rolled out the fine-tuned policy and waited for a failure to naturally occur and we intervened once they happened and completed the task. We then had a human teleoperator intervene through remote teleoperation.

For the recovery-focused policy, we used a DAgger (Dataset Aggregation)-inspired approach to collect demonstrations from states encountered during actual policy rollouts.

Unlike conventional imitation learning, where a policy learns primarily from successful human demonstrations, DAgger addresses a key problem: once a robot makes a mistake, it may enter states that were never represented in its original training data. Without examples of how to respond in these situations, small errors can compound into complete task failures.

To address this, we deployed our fine-tuned π₀.₅ policy and allowed failures to occur naturally. When the robot encountered a failure, a human teleoperator remotely intervened, corrected the mistake, and demonstrated how to complete the remainder of the task.

We collected 50 of these recovery demonstrations and used them to post-train a separate recovery-focused policy. We then compared its performance against a policy trained with 50 additional clean demonstrations, evaluating task success, failure frequency, and recovery ability.

The goal was to showcase learning from failures and corrective interventions improves policy robustness more effectively than simply collecting more successful demonstrations.

How the interventions were collected

We ran the 100-clean policy on the robot and let it fail on its own. A remote operator watched each rollout in VR. As soon as the policy got stuck or made a mistake, the operator took over and completed the task. Every intervention episode starts with the policy driving and contains exactly one takeover.

Every episode logs the takeover time, so each one splits cleanly into the policy prefix and the operator segment. We train only on the operator-controlled segment, from the takeover to the end of the episode, which is the part where the task gets completed. The failed policy prefix is excluded, so the model never sees its own mistake labelled as an expert action, and each recovery segment starts in the failed state the policy left behind.

Reflection:

Across both tasks, adding recovery demonstrations improved task success more than adding the same number of clean demonstrations. These results reflect a single round of recovery collection and post-training. DAgger typically repeats this process as the updated policy encounters new states and failure modes. We’re interested in exploring how further rounds can build on these gains and improve reliability.

VR Teleoperation:

Collecting failure and recovery data is only one part of the challenge. To do this at scale, robotics teams need the infrastructure to evaluate policies, identify failures, remotely intervene, and collect corrective demonstrations across physical environments.

All of the data here was collected through a Meta Quest using Northstar ARC- our VR teleop stack, protocol, and intervention system. Get access to this here.

We believe that there are lots of applications for VR Teleoperation like data collection, remote intervention, and DAgger and post training, which we’re excited to continue working with our partners and customers to explore.

Through our custom network protocol we’ve optimized our one way latency from India to SF to ~120ms and Brazil to SF to ~80ms. See below

What’s next for Northstar?

We’re investigating the most data-efficient ways to improve robot reliability and success rates, while scaling the collection of failure and recovery data. Our goal is to help robots learn from the failures they encounter and recover in real-world workflows. If you’re working on robot learning or deployment, reach out to us.

Explore the data and models

Our model weights and 1,000 episodes—including teleoperation demonstrations, human interventions, and policy rollouts—are available on Hugging Face. Explore the release and build on our work!

Failure Recovery1,000 episodes · Model weightsExplore on Hugging Face ↗

References

1. π₀.₅ base model (LeRobot)Physical Intelligence · LeRobot model card
2. π₀.₅: a Vision-Language-Action Model with Open-World GeneralizationPhysical Intelligence · 2025
3. DAgger — A Reduction of Imitation Learning and Structured Prediction to No-Regret Online LearningRoss et al. · 2011
4. HG-DAgger: Interactive Imitation Learning with Human ExpertsKelly et al. · 2019
5. Policy Improvement using Human InterventionsANR Robot
6. HIL-SERL — Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement LearningLuo et al. · UC Berkeley
7. π*₀.₆: a VLA That Learns From ExperiencePhysical Intelligence · PDF
8. Oopsie DataRobot manipulation failures · Dataset project
9. CR-DAgger — Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human CorrectionsXu et al. · Stanford University