3D City Modeling using Panoramic Images and Digital Map, Using constraint

User-specified constraint concept

• In my opinion, two major advantage of digital map is, 1) precise footprint data and 2) simplified building shape data.
• These two characteristic may be incompatible, because simplification removes the data.
• But still, map is the most precise reference data we can have about the location and orientation of the building if you want data about large area. Even if the shape is projected to 2D and simplified.
• Also, in fact, simplification is necessary if you want to model large area, and the allowed degree of simplification is up to the application. For instance, if you reconstruct the pipe model of the ship for inspection, you cannot allow big simplification. So you may want to use LiDAR.

• One good approach I want to follow is "Building Architectural Models from Many Views Using Map Constraints", which also use digital map and (not panoramic) image for architecture modeling.
• In that research, they assumed 1D camera model and by matching it with digital map, found intrinsic and extrinsic parameters.

• I want to apply their approach in my research. But the detailed process is not clear yet.

Pre-processing, Tilt correction

• One thing I have to do first is the tilt correction. If tilt is not corrected, 1D camera assumption cause large error according to the scene geometry.
• Below figures illustrate the effect of the tilt correction, where the position and direction of the panoramic image has been manually corrected in first figure, and position,direction and tilt(pitch, roll) has been manually corrected.

Tilt not corrected. You can see the displacement between images and building models.

Tilt corrected. Amount of displacement has been reduced.

• For the first-step, I modified the MATLAB code for the pano to perspective, to rotate the entire panoramic image.
• Process is like below.

1. Each image points(pixel) can be mapped to the points on the unit sphere

2. Rotate the points on sphere with rotation matrix, from given angle

3. Points on sphere is re-projected to image points(pixel) with nearest neighbor

• The original version of my implementation was done by modifying the pano to perspective image code, which takes about three minute(196 seconds) to process for source image with 2700x5400 resolution.
• I changed the code consisted of nested for loop(column nested by row for loop) to the matrix calculation for speedup.
• As a result it takes about 20 seconds to process.

• This implementation will be used in unsupervised tilt correction.

3D City Modeling using Panoramic Images and Digital Map

Automatic Framework

• Published : Automatic 3D City Modeling Using a Digital Map and Panoramic Images from a Mobile Mapping System
• In this research, we proposed the automatic 3D city modeling framework which use wide-baseline panoramic image(e.g. Google Street View) and 2D digital map as a source data.
• Main contribution is identification of the component technologies which can be applied in each stage of the framework.
• Overall process and experimental result is shown below.

What I have learned

• What I've learned first is that unsupervised segmentation of the building (contour) is complex problem, even if we have a prior knowledge about the approximate position of the building from the digital map.
• There are several candidate methods I have found, vertical edge based, horizontal edge based, repetitive pattern based and semantic segmentation.
• The first two methods are very weak to the occlusion and repetitive pattern based method is not applicable to the building style of the Korea. Semantic segmentation is excessive approach for our problem.

• Since I'm relatively new to this field, I think I couldn't make a good contribution on image segmentation itself. 
• Instead, I think I have to focus more on the use of the digital map for my research.

What I have developed

• 3D cadastral model generator + panoramic sphere viewer using Unity3D
• Image-building relation calculator script for Unity3D
• Converter for panoramic image to perspective image for MATLAB
• Vertical & horizontal edge based segmentation library and VP estimator for MATLAB
• Image rectification library for MATLAB
• And many others...

3D cadastral model generated from digital map data

Duplicated view of the 3D cadastral model and panoramic sphere, with GPS location and direction data

Pano to perspective, given camera FOV, yaw and pitch