Diskhkme: 3D City Modeling using Panoramic Images and Digital Map, Using constraint

In my opinion, two major advantage of digital map is, 1) precise footprint data and 2) simplified building shape data.
These two characteristic may be incompatible, because simplification removes the data.
But still, map is the most precise reference data we can have about the location and orientation of the building if you want data about large area. Even if the shape is projected to 2D and simplified.
Also, in fact, simplification is necessary if you want to model large area, and the allowed degree of simplification is up to the application. For instance, if you reconstruct the pipe model of the ship for inspection, you cannot allow big simplification. So you may want to use LiDAR.

One good approach I want to follow is "Building Architectural Models from Many Views Using Map Constraints", which also use digital map and (not panoramic) image for architecture modeling.
In that research, they assumed 1D camera model and by matching it with digital map, found intrinsic and extrinsic parameters.

I want to apply their approach in my research. But the detailed process is not clear yet.

One thing I have to do first is the tilt correction. If tilt is not corrected, 1D camera assumption cause large error according to the scene geometry.
Below figures illustrate the effect of the tilt correction, where the position and direction of the panoramic image has been manually corrected in first figure, and position,direction and tilt(pitch, roll) has been manually corrected.

Tilt not corrected. You can see the displacement between images and building models.

Tilt corrected. Amount of displacement has been reduced.

For the first-step, I modified the MATLAB code for the pano to perspective, to rotate the entire panoramic image.
Process is like below.

1. Each image points(pixel) can be mapped to the points on the unit sphere

2. Rotate the points on sphere with rotation matrix, from given angle

3. Points on sphere is re-projected to image points(pixel) with nearest neighbor

The original version of my implementation was done by modifying the pano to perspective image code, which takes about three minute(196 seconds) to process for source image with 2700x5400 resolution.
I changed the code consisted of nested for loop(column nested by row for loop) to the matrix calculation for speedup.
As a result it takes about 20 seconds to process.

Diskhkme