3D City Modeling using Panoramic Images and Digital Map

Automatic Framework

• Published : Automatic 3D City Modeling Using a Digital Map and Panoramic Images from a Mobile Mapping System
• In this research, we proposed the automatic 3D city modeling framework which use wide-baseline panoramic image(e.g. Google Street View) and 2D digital map as a source data.
• Main contribution is identification of the component technologies which can be applied in each stage of the framework.
• Overall process and experimental result is shown below.

What I have learned

• What I've learned first is that unsupervised segmentation of the building (contour) is complex problem, even if we have a prior knowledge about the approximate position of the building from the digital map.
• There are several candidate methods I have found, vertical edge based, horizontal edge based, repetitive pattern based and semantic segmentation.
• The first two methods are very weak to the occlusion and repetitive pattern based method is not applicable to the building style of the Korea. Semantic segmentation is excessive approach for our problem.

• Since I'm relatively new to this field, I think I couldn't make a good contribution on image segmentation itself. 
• Instead, I think I have to focus more on the use of the digital map for my research.

What I have developed

• 3D cadastral model generator + panoramic sphere viewer using Unity3D
• Image-building relation calculator script for Unity3D
• Converter for panoramic image to perspective image for MATLAB
• Vertical & horizontal edge based segmentation library and VP estimator for MATLAB
• Image rectification library for MATLAB
• And many others...

3D cadastral model generated from digital map data

Duplicated view of the 3D cadastral model and panoramic sphere, with GPS location and direction data

Pano to perspective, given camera FOV, yaw and pitch