SNAP: Semantic Navigation Among Pedestrians

From simulation to the real world

Following Semantic Terrain Constraints While Adapting to Obstacles and Crossing Pedestrians

Among Crossing Pedestrians With Semantic Terrain Constraints

Navigation among crossing pedestrians. Translucent floor overlays show low-risk (yellow) and high-risk (red) semantic areas, alongside the bird’s-eye-view map inset. Faces and identifying background details are blurred.

Following Semantic Terrain Constraints and Avoiding Static Obstacles

Navigation through a static scene with semantic terrain constraints. The robot travels around the low-risk area. The bird’s-eye-view inset shows the recorded map. Identifying background details are blurred.

Abstract

Navigating crowded urban environments requires robots to avoid pedestrians while adhering to social norms such as remaining on sidewalks. Existing approaches typically address either crowd avoidance or semantic path planning in isolation. We propose a reinforcement learning (RL) navigation policy that jointly reasons about semantic terrain and dynamic obstacles from a single semantically annotated bird’s-eye-view (BEV) map, without relying on explicit human-state estimation. Our policy outperforms established crowd-navigation and semantic-aware baselines and transfers zero-shot to a legged robot, demonstrating socially compliant navigation in highly dynamic real-world scenarios.

One observation, joint reasoning

Semantic Terrain Meets Crowd Navigation

01

A shared spatial representation

Terrain classes and nearby pedestrians occupy the same robot-centric BEV map.

02

No explicit pedestrian states

At inference, the policy reasons from the map without explicit pedestrian positions or velocities as separate state inputs.

03

Zero-shot transfer

A policy trained in simulation produces navigation commands at 10 Hz on a real legged robot.

Framework: a semantic BEV map and proprioception feed CNN and MLP encoders. The navigation policy outputs velocity commands tracked by a frozen locomotion policy. PPO training uses task rewards, penalties, regularization, and a privileged critic.

For real-world deployment, the BEV map combines a pre-defined semantic map with LiDAR-detected pedestrians.

Learning to share the space

Navigation in Simulation

Simulation rollout with semantic terrain, static obstacles, and moving pedestrians.

Simulation examples of the robot avoiding pedestrians and static obstacles, followed by a sequence passing an oncoming group in a corridor bounded by high-risk terrain. Insets show the robot’s BEV observation. — **Top:** Avoidance maneuvers across benchmark settings. **Bottom:** Passing an oncoming group in a corridor bounded by high-risk terrain; t₀ marks the closest approach. Insets show the 5 × 5 m BEV observation.

BibTeX

@article{wolfram2026snap,
  title={SNAP: Semantic Navigation Among Pedestrians},
  author={Jakob Wolfram and Pascal Roth and Fan Yang and Cesar Cadena and Marco Hutter},
  journal={arXiv preprint arXiv:2611.03498},
  year={2026},
}