Remote control for robots on public streets

Autonomy alone is not enough

No autonomous robot is truly autonomous 100% of the time. Sidewalk delivery robots navigate one of the most unpredictable environments in robotics: public streets, construction zones, double-parked cars, pedestrians with strollers, and dogs on leashes. No matter how good the autonomy stack is, there are situations where the robot needs a human to take the controls.

The challenge goes beyond a simple remote control interface. This teleoperation system needed to work reliably over cellular networks in a dense urban environment, transition seamlessly between autonomous and human-controlled modes, and fail safely when connectivity degrades. A delivery robot stopped on a busy sidewalk because the teleop session dropped becomes a pedestrian hazard and a reputational risk.

The client had the autonomous driving stack. What they did not have was the infrastructure to support human intervention at scale: the networking layer that connects operator to robot over unreliable networks, the on-robot software agent that manages the handoff between autonomy and teleoperation, and the failover systems that ensure safe behavior when things go wrong.

‍

The full teleop stack, built from scratch

Networking and tunneling: reliable connections over unreliable infrastructure

Cellular networks in Los Angeles are inconsistent. Signal strength varies block by block. Handoffs between cell towers can cause momentary dropouts. A teleop system built on standard cloud connectivity would be fragile in exactly the conditions where it is needed most.

We designed and built a custom networking layer with secure tunneling that maintains persistent, low-latency connections between robots and operator stations. The system handles cell tower handoffs, manages bandwidth adaptation when signal quality degrades, and maintains connection state through brief interruptions rather than requiring a full session restart.

On-robot agent: the brain that manages the handoff

Each robot runs a lightweight software agent that manages the transition between autonomous operation and human control. When the autonomy system encounters a situation it cannot resolve, the agent initiates a teleop request, establishes the connection to an available operator, streams video and sensor data, and accepts remote commands, all while maintaining safety boundaries.

The agent is designed with failover as a first-class concern. If the teleop connection drops during a session, the agent does not leave the robot in an uncertain state. It executes a safe stop, maintains awareness of its environment, and attempts to reconnect. If reconnection fails within a defined window, the robot enters a safe mode and awaits physical intervention.

Operator interface: situational awareness at speed

The operator station needs to give a human enough situational awareness to make safe driving decisions in real time, from miles away, through a screen. We built an interface that streams live video from the robot's cameras, overlays sensor data and obstacle detection information, and provides intuitive controls for steering, speed, and mode transitions.

The interface is designed for operators who manage multiple robots in a shift. Session handoff is fast. The operator sees the robot's current context, the reason for the teleop request, and any relevant environmental data before taking control. When the situation is resolved, control transitions back to the autonomy stack with a single action.

‍

The hardest problems live at the boundary between robot and world

Building a teleoperation system is a networking problem, a robotics problem, a real-time systems problem, and a user interface problem simultaneously. It requires low-latency video streaming, secure tunneling, state machine design for safety-critical transitions, and an operator experience that supports fast, accurate decision-making under time pressure.

Very few engineering teams have depth across all of these domains. Robotics teams understand the autonomy stack but may not have the networking expertise for reliable urban cellular connectivity. Networking teams understand tunneling but do not know how to design a safe handoff between autonomous and human-controlled robot modes. Our team has built fleet management systems that span all of these concerns, and that cross-domain fluency is what makes a teleoperation system reliable rather than merely functional.

The system we built is deployed in one of the most challenging urban environments in the country. It handles the unpredictability of Los Angeles streets, Los Angeles cellular networks, and the edge cases that no autonomy system can fully anticipate. That it works reliably is a testament to the engineering rigor applied at every layer of the stack.

‍