Overcoming Hallucination when Navigating within AR and VR Environments

Accurate 3D visualizations in surgical procedures: To our knowledge, this is the first successful application of point transformers directly on 3D Gaussian Splatting sets, overcoming the limitations of prior multi-scene training methods. This remarkable achievement is credited to Yutong Chen, Marko Mihajlovic, Xiyi Chen, Yiming Wang, Sergey Prokudin, and Siyu Tang from ETH Zurich, the University of Maryland, College Park, and ROCS at University Hospital Balgrist, University of Zurich.

Results on GSO-OOD. We compare SplatFormer, trained on Objaverse scenes, with Nerfbusters (Warburg* et al., 2023), 2DGS (Huang et al., 2024a) and MipNeRF360 (Barron et al., 2022).

While hallucination can be beneficial for creative applications, it may be undesirable in settings that demand accurate reconstructions, such as 3D visualization of surgical procedures (Hein et al., 2024), and unnecessary in typical daily capture scenarios.

More details you can find in the online publication.