Hybrid Approach for Automatic Building Model Generation by Integrating LiDAR and Image Data

Prévia do material em texto

Ayman F. Habib Photogrammetric & LiDAR Mapping 
Hybrid Approach for Automatic Building Model 
Generation by Integrating LiDAR and Image Data 
Chapter 6 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
2 
To avoid social and environmental problems 
arising from this rapid urbanization, federal 
and local governments must have access to 
accurate and current geo-spatial information 
in a timely manner and at a reasonable cost. 
• Global population is expected to rise to almost 
5 billion inhabitants by 2030. 
• 62% of the global population at that time will 
be dwelling in urban areas. 
Globally 
Canada metropolitan 
areas 
rural 
areas 
(1971 – 2001) 
Population growth in Canada 
Introduction 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Introduction 
 
 
 
 
 
 
 
 
 
 
Natural disaster 
planning & 
evaluation 
Urban planning 
Military 
simulation 
3D building models 
Telecommunication 
network 
planning 
3 
• 3D building models are among the key components of geo-
spatial information in urban areas. 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
• Current technology/tools (i.e., Google Earth, Microsoft’s Virtual Earth, etc.): 
– Good for navigation purposes 
– Level of accuracy is not high enough for design and engineering 
applications 
Examples of 3D building models in Calgary downtown (Source: Google Earth) 
Introduction 
4 
Need for automated tools that provide accurate and up-to-date 
3D building models 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Existing DBM Generation Methodologies 
Data sources 
Imagery 
LiDAR 
Multi-source 
Processing 
strategy 
Data-driven 
Model-driven 
Hybrid 
Amount of user 
interaction 
Manual 
Semi-
automatic 
Fully-
automatic 
5 
Proposed Research 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
6 
Reconstruction Strategy 
Model driven Data driven 
• Parametric models 
‒ Few models 
• Generic models 
– Can deal with any model type 
• Building database 
– Prior knowledge about the 
buildings’ shapes 
• Sensory data 
– Only source of information 
• Procedure 
– Generation of multiple building 
hypotheses 
– The model achieving the best fit is 
the selected one 
• Procedure 
– Extraction of building primitives 
– Reconstruction of surface topology 
– Construction of the building model 
• Can only handle simple models such 
as flat-roof and gable-roof buildings 
• In theory, it can handle any type of 
buildings. 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Model-Based Approach 
7 
l 
w 
h 
l w 
h 
l 
w 
h 
rh 
l 
w 
h 
rh 
Box Wedge 
Gable-roof Shed-roof 
Model base 
Model-image fitting 
Adjusted model parameters 
(Wang and Tseng, 2003) Initial model parameters 
Adjust the initial model 
parameters until the model fits 
well with the edges extracted 
from the corresponding images 
Edge detection 
Research objectives: 
• Automatic generation of the initial model parameters 
• Reduce the number of models while being able to 
deal with box, wedge, gable-roof, and shed-roof types 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Proposed Workflow 
8 
LiDAR data 
Data-driven approach 
Hypothesized building rooftops 
 
Initial model parameters 
Model-driven approach 
Refine the necessary model parameters 
Stochastic properties of the model parameters 
Multiple images Edge detection 
Model-image fitting 
LiDAR plane para. 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
9 
P
ro
p
o
se
d
 W
o
rk
fl
o
w
 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Data-Driven Stage (LiDAR) 
10 
Building 
Hypothesis 
Generation 
 
•Terrain/off terrain 
•Planar/non-planar 
point classification 
•Grouping 
•Area/height 
computation 
Segmentation 
Boundaries for 
Hypothesized 
Buildings 
 
•Neighbourhood & 
attribute comp. 
•Clustering 
(Habib et al., 2010 & Lari and Habib, 2011) 
LiDAR Point 
Cloud Processing 
•Local point density 
estimation 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
• Orthophoto over the test area 
Data-Driven Stage (LiDAR) 
11 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
• Original LiDAR data 
12 
Data-Driven Stage (LiDAR) 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
• An adaptive cylinder, which changes its orientation with the 
local planar surface, is used to decide whether the point 
belongs to a planar or rough surface. 
Point Classification & Local Point Density 
 
 
13 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
If the iterative plane fitting procedure does not converge within a pre-specified 
number of iterations, the point of interest is classified as a rough point. 
First Plane 
d 
Removed as outliers 
Derivation of the Adaptive Cylinder 
Point Classification & Local Point Density 
14 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
 
 
 
 
 
 
 
 
Planar Non-planar 
• To classify the point as being part of a planar surface, the following 
conditions should be satisfied: 
The point of interest should be inside the adaptive cylinder. 
The majority of the points in the spherical neighbourhood should 
be inside the adaptive cylinder. 
Point Classification as Being Part of a Planar Surface 
Point Classification & Local Point Density 
15 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
2
kr
n
LPD


n Number of the points that belong to the adaptive cylinder. 
 sphere)of pnts ink (number cylinder) adaptive of pnts inn (number 
Estimation of the Local Point Density 
Point Classification & Local Point Density 
16 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Planar Surfaces Rough Surfaces 
• Planar & rough surface classification 
Data-Driven Stage (LiDAR) 
17 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
• Point density map 
Data-Driven Stage (LiDAR) 
18 
0.07 1.63 pts/m2 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Each colour represents a group of planar points. The groups are derived using a 
region growing algorithm based on the estimated local point density. 
• Grouping 
Data-Driven Stage (LiDAR) 
19 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Segmentation 
Neighbourhood Definition 
Attribute Computation 
Aggregation/Clustering 
Segmentation 
Parameter-Domain Segmentation 
20 
Derived from the 
adaptive cylinder 
Octree-based peak 
detection 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
• Segmentation 
Data-Driven Stage (LiDAR) 
21 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Data-Driven Stage (LiDAR) 
• Boundary detection 
22 
Hybrid approach for outer/inner boundary detection 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Non-ground, Rough 
Ground, Rough 
Non-ground, Planar 
Ground, Planar 
Data-Driven Stage (LiDAR) 
• Ground/non-ground classification 
23 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
24 
Customized Parameters for Building Hypothesis Generation 
Building Hypothesis 
Planar / Non-Ground 
Group Height > Threshold (e.g., 3m) 
Group Area > Threshold (e.g., 10 m2) 
Building Hypothesis Generation 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
25 
Terrain/Off-terrain 
Classification 
Segmentation/ 
QC Process 
LiDAR Point Cloud 
Data-Driven Stage (LiDAR) 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
26 
Building Hypotheses 
Generation 
Boundary Detection Aerial Photo 
Irregular boundary due to low sampling rate 
 needs to be regularised 
Data-Driven Stage: Building Boundaries 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
27 
P
ro
p
o
se
d
 W
o
rk
fl
o
w
 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Model-Driven Stage (LiDAR & Imagery) 
28 
Boundary 
Regularization & 
Refinement 
 •Minimum Bounding 
Rectangle (MBR) – 
simple box model 
•Terrain/off 
terrain 
•Planar/non-planar 
point classification 
•Grouping 
•Area/height 
computation 
Segmentation 
•Neighbourhood 
& attribute comp. 
•Clustering 
(Habib et al.,2010 & Lari and Habib, 2011) 
LiDAR Point 
Cloud Processing 
•Local point 
density estimation 
We will start with a simple box model? 
•Recursive MBR – 
complex models 
•Model-image fitting 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Model Choice 
29 
Box model Rectangular model 
Can represent all the desired building models 
 
l 
w 
h 
l 
w 
• Building footprint might not be 
visible in the imagery. 
• Building height can be derived 
from LiDAR. 
• Building footprint can be derived 
from its rooftop & DTM. 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
30 
l 
w 
h 
Box 
l w 
h 
Wedge 
l 
w 
h 
rh 
l 
w 
h 
rh 
Gable-roof Shed-roof 
• The involved parameters change with the model type. 
Building models 
Model Choice 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
31 
Model Choice 
Box Wedge 
Gable-roof Shed-roof 
Rectangular model 
Required Model Parameters: (Xm0, Ym0, Zm0, ωm, φm, κm, l, w) 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Rectangle Model Parameters 
32 
𝓁 
Y 
Z 
 
X 
 
κ𝑚 
(𝑋𝑚0,𝑌𝑚0) 
W 
𝓌 
U V 
(ωm, φm) 
Zm0 
Required Model Parameters: (Xm0, Ym0, Zm0, ωm, φm κm, l, w) 
derived from LiDAR data 
derived from LiDAR data 
•Model parameters to be estimated: (Xm0, Ym0, κm, l, w) 
•Approximate values can be derived using the Minimum 
Bounding Rectangle (MBR). 
•Model parameters are refined through a LSA-based model-
image fitting. 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Initial Parameters for the Rectangle Primitive 
33 
• Minimum Bounding Rectangle – MBR (Freeman and 
Shapira, 1975) 
 The minimum area rectangle among the rectangles with an 
arbitrary orientation that contains the vertices of the boundaries 
Aerial image Segmented 
LiDAR patch 
Traced 
boundary 
Regularized 
boundary 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
34 
Aerial image LiDAR derived 
boundary 
Bounding rectangles 
with arbitrary orientations 
MBR 
Initial Parameters for the Rectangle Primitive 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Edge Detection and Filtering 
35 
edge linking & removal of 
short segments 
Canny edge 
detection 
Edge filtering 
(position and orientation) 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Model-Image Fitting 
36 
y V 
U 
θ 𝑣i 
𝑣i+1 
Edge pixel 
(𝑥𝑒𝑑, 𝑦𝑒𝑑) 𝑣i (𝑥i, 𝑦i) 
𝑣i+1 𝑥i+1, 𝑦i+1 
d 
x 
Adjusted line 
𝑣i(adjusted) 
Initial projected line 
 
model-
image 
fitting 
• Least-Squares Adjustment Implementation: 
1. Normal distance constraint 
 𝑑 = 𝑓𝑛 𝑥𝑖 , 𝑦𝑖 , 𝑥𝑖+1, 𝑦𝑖+1, 𝑥𝑒𝑑 , 𝑦𝑒𝑑 
2. Modified LSA using pseudo-conjugate points 
 𝑥𝑒𝑑 = 𝑥𝑖 + 𝐷𝑥 + 𝑒𝑥 & 𝑦𝑒𝑑 = 𝑦𝑖 + 𝐷𝑦 + 𝑒𝑦 
𝑥𝑖 , 𝑦𝑖 = 𝑓 𝑋𝑖 , 𝑌𝑖 , 𝑍𝑖 , 𝐼𝑂𝑃, 𝐸𝑂𝑃 
𝑋𝑖 , 𝑌𝑖 , 𝑍𝑖 = 𝑓𝑚 𝑚𝑜𝑑𝑒𝑙 𝑝𝑎𝑟𝑎𝑚𝑒𝑡𝑒𝑟𝑠 
𝑒. 𝑔. , 𝑓𝑜𝑟 𝑡ℎ𝑒 𝑟𝑒𝑓𝑒𝑟𝑒𝑛𝑐𝑒 𝑝𝑜𝑖𝑛𝑡: 𝑋𝑖 = 𝑋𝑜&𝑌𝑖 = 𝑌𝑜 
 
 
• The model-image fitting is done using all the 
rectangle sides and the available images. 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
37 
Simple Box / Gable Roof Building 
How to deal with complex building models? 
Recursive Minimum Bounding Rectangle 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
38 
1st level MBR 
Recursive MBR 
2nd level MBR 
• non-overlapping initial 
boundary with 1st level MBR 
• non-overlapping 1st level MBR 
with initial boundary 
Final MBR 
Subtract 
• Different levels of MBRs will be adjusted sequentially 
during model- image fitting 
 Some of the lower-level MBRs’ parameters are utilized for 
the higher-level MBRs 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
39 
LiDAR boundary 1st level MBR 
Non-overlapping 
boundary points 
Projection of non-
overlapping points 
2nd level MBR 
Non-overlapping 
boundary points 
Projection of non-
overlapping points 
3rd level MBRs 
Non-overlapping 
boundary points 
4th level MBR 
Projection of non-
overlapping points 
𝐹𝑖𝑛𝑎𝑙 𝑆ℎ𝑎𝑝𝑒 = −1 𝑛+1
𝑘
𝑛=1
𝑀𝐵𝑅𝑛 
• The planimetric coordinates of the boundary should 
be improved 
• Different MBR levels will be sequentially adjusted 
during model-based image fitting 
 Some of the lower-level MBRs’ parameters are utilized for 
the higher-level MBRs 
Sequential MBR Adjustment: Multiple Levels 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Dataset Description 
40 
 BCIT dataset 
 Photogrammetric data LiDAR data 
System MFDC Rollei P-65 Leica ALS50 
Average flying height 540m and 1,150m 540m and 1,150m 
GSD for image / 
Average point density 
for LiDAR 
5cm and 10cm 1.5 and 4.0 points/m2 
Strip 1
Strip 2
Strip 3
Strip 4
Strip 5
Strip 6
Strip 1
Strip 2
Strip 3
Strip 4
Strip 5
Strip 6
LiDAR strips configuration (left) 
configuration of aerial photos (right) 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
• Estimated model parameters & their standard 
deviations for a simple rectangular building 
Model Parameters - Results 
Modified LSA Normal distance constraints 
Unknown 
Parameter 
Initial 
value 
Final value 
after 
adjustment 
Standard 
deviation 
of 
adjusted 
parameters 
Final value 
after 
adjustment 
Standard 
deviation 
of 
adjusted 
parameters 
500238 500239.626 ± 0.008 500239.646 ± 0.008 
5455402 5455404.374 ± 0.007 5455404.317 ± 0.006 
10 11.571 ± 0.007 11.567 ± 0.006 
42 43.823 ± 0.012 43.882 ± 0.010 
-138 -142.133 ± 0.008 -142.044 ± 0.015 
41 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
42 
Model-Based Image Fitting: Number of Images 
• The proposed methodology works with single or 
multiple images. 
• Using more images captured from different views will 
reduce the impact of occlusions. 
• The impact of the number of utilized images on the 
quality of the derived models has been examined. 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
43 
• The quality of the models can be compared in a qualitative way 
by projecting the reconstructed points onto one of the images. 
• When visually checked, all of the projected lines coincide with 
building edge lines and all the cases show good results. 
manual 
reconstruction 
1 image 2 images 4 images 5 images 
Qualitative Analysis 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Quantitative Analysis 
44 
RMSE [m] 
manual 
reconstruction vs. 
X Y Z Total 
single image 0.03 0.08 0.23 0.24 
two images 0.03 0.09 0.23 0.25 
four images 0.05 0.08 0.23 0.24 
five images 0.05 0.07 0.23 0.24 
• RMSE between the reconstructed corner points using different 
number of images and the manual reconstruction are compared. 
• The accuracy of the results is not affected by the number of 
utilized images based on the comparison. 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
45 
Integrated Product 
(X, Y, Z): 1122.23 m, 3251.53 m, 72.03 m (±10 – ±30cm) 
(R, G, B): 23, 136, 69 
Accurate Enough for Engineering Applications 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
46 
• University of Calgary main campus area 
Applicable Partially applicable Non applicable Total 
# of buildings 37 10 4 51 
Percentage of 
buildings 
72.5 % 19.6 % 7.8 % 100 % 
Less than 10 % of the buildings cannot be modeled 
Future work: For partially possible buildings, what can we do? 
Experimental Results 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
47 
Realistic 3D Modeling / Visualization 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
Concluding Remarks 
• The presented approach takes advantage of overlapping image and 
LiDAR data that has undergone rigorous QA/QC and registration 
procedures. 
• The level of automation is increased by utilizing the LiDAR point cloud 
in a data-driven approach to generate the initial model parameters while 
using the imagery in a model-driven approach to refine the model 
parameters. 
– A general model that can deal with rectangular, gable, wedge, shed rooftops 
(a building can be a mix of these primitives)– The sequential MBR has been developed to deal with more complex 
building rooftops. 
• The stochastic properties of the model parameters are provided by the 
proposed approach. 
• The LiDAR DSM after its enhancement with a DBM leads to better 3D 
visualization of the integrated imagery and LiDAR data. 
 48 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
49 
Camera QA/QC LiDAR System QA/QC 
Co-registration 
Imagery 
(R, G, B) 
DSM 
(X, Y, Z) 
Preliminary 
Orthophoto 
DBM 
Photogrammetric Data LiDAR Data 
Realistic 3D Visualization 
Enhanced DSM 
Refined Orthophoto 
Building 
Hypothesis 
Extracted 
DTM 
OR 
• For effective integration of 
photogrammetric/LiDAR data, 
one must address these basic 
operations: 
QA/QC 
Registration 
Linkage of spectral/positional 
information 
Tight integration should be 
explored. 
Concluding Remarks 
Ayman F. Habib Photogrammetric & LiDAR Mapping 
50