Deep Semantic Segmentation of 3D Scenes: A Survey

In this article, the authors propose a novel approach to online semantic 3D reconstruction, which fuses RGB-D observations into a globally consistent semantic map. The key component is a local spatio-temporal expert network that learns to select relevant information from 2D, 3D, and previous steps using an attention mechanism. This allows the network to efficiently integrate new observations into the existing scene representation, resulting in improved 3D reconstruction accuracy.
To understand how this works, imagine a group of experts in different fields, each with their own area of expertise. These experts are brought together to solve a complex problem by sharing their knowledge and collaborating on a local level. The same idea applies to the proposed system, where a local spatio-temporal expert network fuses new observations into a learned scene representation, leveraging the strengths of both 2D and 3D information.
The article explains that the proposed approach consists of two stages: a 2D encoder and a 3D encoder. The 2D encoder extracts semantic features from incoming 2D RGB-D images using a 2D convolutional network, while the 3D encoder fuses new observations into a learned scene representation using an attention mechanism. This allows the network to selectively focus on the most relevant information and improve the accuracy of the 3D reconstruction.
The authors also highlight the importance of the attention mechanism, which enables the network to efficiently integrate new observations into the existing scene representation. This is particularly useful in situations where the amount of available information is limited, such as when dealing with complex or dynamic scenes.
In summary, the proposed online semantic 3D reconstruction system offers a novel approach to fusing RGB-D observations into a globally consistent semantic map. By leveraging the strengths of both 2D and 3D information, the system improves the accuracy of 3D reconstruction while efficiently integrating new observations into the existing scene representation.