Data visualization - Bibliography? (※5MB GIF included!!)

Data visualization of papers & keywords 

• Developed as a part of convergence research contest.
• Our team members decided to propose the concept for analyzing, visualizing and exhibit the research relationships in our institute.
• We started from extracting the keywords in papers published in recent 5 years from our institute.
• Because of the lack of time, I "conceptually" developed an application to show how the relationships can be visualized.

Data visualization, 2D or 3D? 

• The D3.js is now getting popular for most type of data visualization in 2D.
• 2D data visualization has advantages on concise expression and easy perception to users.
• On the other hand, the 3D visualization is needed when we want to provide "spatial" sort of experience to user.
• Usually the type of data we focused tends to be visualized in 2D, but we(actually I,) tried to visualize it in 3D for future.
• For instance, LBS or AR use.
• Of course the effective visualization approach should be discussed more in later.

Development of a demo application

• Like always, Unity3D is very useful when you need to develop an application in very short time. 
• The concept of my visualization is a planetarium, which each keyword is floating on the dome shaped sky. 
• Application includes relationship visualization functions and search functions. 
• Also the titles, authors and research fields can be seen if user selected certain keyword to get into the "Focus Mode". 
• Textures are simply made from MS PowerPoint. 
• Source database exists in Dropbox and read through WWW class at first time, but I changed it into TextAsset resource since the data is not going to be dynamically changing. 
• Again, the multi-channel visualization can be easily done though the multi-channel module I have.

The scene captured in 3rd person view

Selecting the keyword in 1st person view
Relationship is indicated with line 

Searching the keyword
Searching result is indicated with simple animation with different texture

 Focus Mode.
Specific information related with "Lithium-Ion Battery" can be seen in here

Multi-channel visualization

Discussion? 

• As I mentioned, the effectivess of visualization should be improved in so many ways.
• Location based service application using GPS data can be easily developed, but the performance should be improved when using it with smart device.
• There's not much "analysis" of the data in demo application. Data analysis method in bibliography fields should be applied.

Google Cardboard & 3DUI

Google Cardboard, HMD 

• I purchased two set of Google Cardboard about a week ago.
• The cardboard test app from Google was nice, but the app is getting slower and slower every time I run the app...weird.

• Anyway, I made the first simple test scene for it using Unity3D.
• The simple side-by-side scene configuration method is described in HERE
• I also combined camera manipulation method using gyro sensor data from smart device. HERE is nice and short introduction for it.

Test scene containing desk, sphere and capsule(occluded)

Rendered image in smart phone.

View from single lens

• Not bad. I guess I should remove lens distortion using shader.

3D City Modeling using Panoramic Images and Digital Map, Using constraint

Pre-processing, Tilt correction, Cont'd

• I developed the simple code for unsupervised tilt correction(perspective correction?) of panoramic image.
• First the "Line segment detector" in vpdetection library has been applied on each panoramic image
• By calculating the slope of line segments, define the subset of line segments that can be considered as vertical line segments.

Vertical line segments are indicated as red line

• I define the amount of tilt for each line segment as length multiplied by the difference of x coordinate position of start and end point.

Amount of tilt in original image. Mean : 1.6431

•  Like described in the previous post, each pixel of the image can be mapped to the point on the unit sphere, so calculate the mapped point of all start and end points.
• Then find lambda(rotation angle along x axis) and phi(rotation angle along  y axis) that is argmin for the sum of amount of tilt for vertical line segments.
• Apply rotation to the entire image by given lambda and phi.

Amount of tilt in rotated image. Mean : 1.0062

3D City Modeling using Panoramic Images and Digital Map, Using constraint

User-specified constraint concept

• In my opinion, two major advantage of digital map is, 1) precise footprint data and 2) simplified building shape data.
• These two characteristic may be incompatible, because simplification removes the data.
• But still, map is the most precise reference data we can have about the location and orientation of the building if you want data about large area. Even if the shape is projected to 2D and simplified.
• Also, in fact, simplification is necessary if you want to model large area, and the allowed degree of simplification is up to the application. For instance, if you reconstruct the pipe model of the ship for inspection, you cannot allow big simplification. So you may want to use LiDAR.

• One good approach I want to follow is "Building Architectural Models from Many Views Using Map Constraints", which also use digital map and (not panoramic) image for architecture modeling.
• In that research, they assumed 1D camera model and by matching it with digital map, found intrinsic and extrinsic parameters.

• I want to apply their approach in my research. But the detailed process is not clear yet.

Pre-processing, Tilt correction

• One thing I have to do first is the tilt correction. If tilt is not corrected, 1D camera assumption cause large error according to the scene geometry.
• Below figures illustrate the effect of the tilt correction, where the position and direction of the panoramic image has been manually corrected in first figure, and position,direction and tilt(pitch, roll) has been manually corrected.

Tilt not corrected. You can see the displacement between images and building models.

Tilt corrected. Amount of displacement has been reduced.

• For the first-step, I modified the MATLAB code for the pano to perspective, to rotate the entire panoramic image.
• Process is like below.

1. Each image points(pixel) can be mapped to the points on the unit sphere

2. Rotate the points on sphere with rotation matrix, from given angle

3. Points on sphere is re-projected to image points(pixel) with nearest neighbor

• The original version of my implementation was done by modifying the pano to perspective image code, which takes about three minute(196 seconds) to process for source image with 2700x5400 resolution.
• I changed the code consisted of nested for loop(column nested by row for loop) to the matrix calculation for speedup.
• As a result it takes about 20 seconds to process.

• This implementation will be used in unsupervised tilt correction.

3D City Modeling using Panoramic Images and Digital Map

Automatic Framework

• Published : Automatic 3D City Modeling Using a Digital Map and Panoramic Images from a Mobile Mapping System
• In this research, we proposed the automatic 3D city modeling framework which use wide-baseline panoramic image(e.g. Google Street View) and 2D digital map as a source data.
• Main contribution is identification of the component technologies which can be applied in each stage of the framework.
• Overall process and experimental result is shown below.

What I have learned

• What I've learned first is that unsupervised segmentation of the building (contour) is complex problem, even if we have a prior knowledge about the approximate position of the building from the digital map.
• There are several candidate methods I have found, vertical edge based, horizontal edge based, repetitive pattern based and semantic segmentation.
• The first two methods are very weak to the occlusion and repetitive pattern based method is not applicable to the building style of the Korea. Semantic segmentation is excessive approach for our problem.

• Since I'm relatively new to this field, I think I couldn't make a good contribution on image segmentation itself. 
• Instead, I think I have to focus more on the use of the digital map for my research.

What I have developed

• 3D cadastral model generator + panoramic sphere viewer using Unity3D
• Image-building relation calculator script for Unity3D
• Converter for panoramic image to perspective image for MATLAB
• Vertical & horizontal edge based segmentation library and VP estimator for MATLAB
• Image rectification library for MATLAB
• And many others...

3D cadastral model generated from digital map data

Duplicated view of the 3D cadastral model and panoramic sphere, with GPS location and direction data

Pano to perspective, given camera FOV, yaw and pitch

Mesh Comparison - Javascript & ASP.NET service

Web page for mesh comparison

Pure JS, client-side scripting

• For the interactive servicing of mesh comparison(or retrieval), I made the web page for it using X3DOM for visualization and JS for calculation.
• It was pretty successful, and you can see the result in the below link.
• You can test with your OBJ or OFF mesh data or,
• You can download sample model from here.
• http://ssong.kaist.ac.kr/upload_test.html
• (OBJ and OFF parser may have some limitation.)

Using ASP.NET

• Soon, I realize that the client-side scripting is not enough for maintaining DB for mesh data, and shape descriptors.
• I'm not familiar with the various web technologies and choose a server-side programming method that I can develop fast-ASP.NET
• First step is develop the calculation module with C# DLL, which is just a conversion task from JS.
• Second step is designing the layout, which I'm not good at, and connecting the module with elements in the page.
• Also, one functionality added is storing the uploaded mesh data into server machine for later development.
• For dynamic visualization of X3DOM, "HTMLGenericControl" of ASP.NET was used to change the attribute value in X3D node.(Is there any performance issue?)
• You can test with your OBJ of OFF mesh data, but you should keep in mind that the DATA YOU UPLOADED WILL BE STORED IN MY SERVER MACHINE.
• (OBJ and OFF parser may have some limitation.)
• http://143.248.54.5:1234/default.aspx

Further Issues

In development

• The postback harms the UX...

• Performance itself seems worse than JS

• Hopefully, I will solve the problems by modifying the structures.

• Model DB displaying, selection and retrieval.

In methodology

• I didn't clearly understand the Histogram --> PDF part yet.

• Several approaches on normalization, bin width selection and PDF comparing.

Large-scale visualization of a simulation data using Unity3D

Simulation Visualization Framework

• The 3D visualization can help the decision making process which usually involves diverse simulation data.
• We studied to develop efficient visualization framework, especially for the warfare simulation several years ago.
• The framework has been proposed consisted of 7 modules and interfaces.
• For efficient implementation, we used Unity3D which allows faster development of an application and cross-platform distribution.
• Detailed information can be seen in HERE. 

Simulation of a Capsized Ship, and Lifting Process

• There was a serious disaster in Korea. (related article)
• After few weeks, the ORIN Lab performed simulation to validate the reason of capsizing.
• And this data, should be visualized as soon as possible for publishing.
• So we reuse the framework above, and replacing only the "Data Processing" module to change the processable data from warfare simulation to capsizing simulation.
• As a result, a day was enough to implement the application.

Extending the Application to Large-scale Visualization

• Today, I converted the application described above, which is for standalone use, into large-scale visualization.
• This also takes fairly short time(a day), since I already developed "Multi-Channel" module for our facility.(small grey box in the first figure)
• So all I have to do is just plug the module in my Unity3D project and build it to install it in CAVE facility.
• Although the GUIs are not handled properly. It is not easy to remove it in client without modifying the script code...

Mesh Comparison - Day 4

DSME-(Histogram-based) Qualitative Mesh Comparison in between ship CAD systems.

%-------------------------------------------------------------------------------

Goal : Develop the module for 3D mesh comparison, in ACIS environment

Todo :

1. Study

• Qualitative shape (search, detection, matching, comparison, ...)
• MeshLab(VCG) or OpenMesh library
• ACIS & HOOPS

    2. Implementation

• Meshing(Solid, B-rep to facet data)
• Sampling on mesh surface
• Shape Distributions

%-------------------------------------------------------------------------------

Sampling on mesh surface

• Be aware, that weighted random number generation based on the binary search, can cause the memory problem.
• In our case, sampling from the model has 15,725 faces are crashed when we pick a random face with binary search.
• So instead of the binary search, we use C++ random library, which has weighted random number generation.(LINK)
• Interval = [1,number of faces], and weight is the area for each face.

Shape Distributions

• "Shape Distributions"
• From sampled points, pick N pair of points and calculate D2 distance(Euclidean).
• The distance values are stored in B number of bins, and the number of distance is calculated.
• Interval of each bin = (max_distance-min_distance)/B

(※The 3D model only works when you see this post separately with Day-3+, by clicking the title of this post)

You can see the difference of first and second model.
First model is more simplified one.

And follow is the corresponding histogram of these two.

Overall shape is similar but there's differences, the metric distance of these two histogram will be implemented later.

For instance, another completely different model and corresponding histogram shows different shape of histogram.

Mesh_Comparison - Day 3+

HTML5 Interactive

You can see the results in previous post interactively, by X3DOM.
Wheel click + drag : panning
Left button click + drag : rotating
Right button click + drag : zooming