Semantic SLAM

Semantic SLAM

Semantic SLAM (simultaneous localization and mapping) uses estimation and machine learning algorithms to create a map that has both metric (location, orientation) and semantic (class like car, window, chair, door) information about the objects in a scene.


Traditional robot mapping methods have created geometric maps in terms of 2D layouts or 2D point clouds, 2D- or 3D- occupancy grids. This is like seeing a classic geographic map without a single symbol or text on it. You can still navigate to places given their coordinates and find your way back home but you cannot verify your navigation with landmarks nor give directions that are based on those landmarks.

Semantic SLAM enables human-robot communication through words that describe objects or landmarks on the map. It facilitates reasoning about the scene for autonomous vehicles that have to make decisions based on “what” they see. Last, they provide even better metric maps by abstracting objects with their geometric shapes.

This technology achieves that goal by using machine learning for creating hypotheses about objects and key points on those objects. The selection of the correct hypotheses is made by a process called data association, which corresponds objects and key points across time-points and a database of saved places. The key is to use probabilistic techniques that will take the expectation (averaging) over all possible associations in order to estimate the poses of the objects and the robots.


Sean L. Bowman, Kostas Daniilidis, George J. Pappas. Robust and Efficient Semantic SLAM with Semantic Keypoints. [Download PDF]

Probabilistic Data Association for Semantic SLAM video [View]

Penn Researchers Win Best Paper at International Robotics Conference, Penn Engineering Today

Y-Prize Kickoff presentation (Jan. 19, 2021)

Y-Prize Tech Briefing (Jan. 26, 2021)

In the lab

In the lab