Abstract
This study addresses the need of making reality-based 3D urban models more detailed. Our method combines the established workflows from photogrammetry and procedural modelling in order to exploit distinct advantages of both approaches. Our overall workflow uses photogrammetry for measuring geo-referenced satellite imagery to create 3D building models and textured roof geometry. The results are then used to create attributed building footprints, which can be applied in the procedural modelling part of the workflow. Thereby procedural building models and detailed façade structures, based on street-level photos, are created. The final step merges the textured roof geometry with the procedural façade geometry, resulting in an improved model compared with using each technique alone. The article details the individual workflow steps and exemplifies the approach by means of a concrete case study carried out in Singapore’s Punggol area, where we modelled a newly developed part of Singapore, consisting mainly of 3D high-rise towers.
1. Introduction
Today, the use of large-scale urban 3D models has become ubiquitous among many application domains; for example, exponents of the mapping industry are striving to augment their data sets with detailed 3D models reflecting the real world. In city design and planning and in scientific studies on city transformations, analysing stocks and flows, 3D models gain in relevance Citation(1). Geographical information systems (GIS) have traditionally operated in the 2D or 2.5D domain but are now rapidly moving towards real 3D, or sometimes 4D to denote the inclusion of time-dependent dynamic processes. Three-dimensional urban models also play an important role in the entertainment industry, in particular the film and gaming industries. The processes of creating the 3D models vary from domain to domain, ranging from completely manual content creation to semi-automatic techniques making use of computer-vision or photogrammetry and of generative modelling techniques. However, as the scale and quality requirements for generation of urban 3D environments is increasing, manual creation becomes very time-consuming. Therefore, computer-assisted, semi-automatic, or even fully automatic methods move to the centre of focus. However, one has to pay tribute to the fact that with higher resolution and level of detail the automation of reality-based model generation becomes less practical, and indeed, most of the truly high-resolution city models are nowadays still produced manually or at best semi-automatically.
Therefore, in this work, we aim at increasing the labour efficiency as well as the level of detail of created models by combining two established modelling methods into an applicable common workflow: We apply photogrammetry to create building models from aerial and satellite imagery and then use procedural modelling techniques to enhance the models. Thereby the procedural models make use of existing GIS layer information (such as zoning information or extracted attributes from the photogrammetric models) and in particular aim at increasing the level of detail of façade geometry and texturing. The resulting geometry combines the advantages of the two methods, in that it is (a) reality-based and (b) parametric in terms of level-of-detail and texture.
To summarise, the contributions of this study are the following: (a) we present a methodology how to combine photogrammetric modelling and procedural modelling approaches into a single workflow; (b) we show how the approach increases the level-of-detail and texture quality compared with using photogrammetry alone; and (c) we show how our methodology is applicable in practise in terms of a concrete case study based on satellite imagery and terrestrial façade photographs of new-town building towers in Singapore’s Punggol area.
The remainder of this study is organised as follows: In the next section, we will provide the background of our work and review the related work. In Section 3, we will provide the required basics of photogrammetry as well as procedural modelling (based on shape grammars) and present our approach to combine elements of both techniques into a single workflow. Section 4 provides a representative example illustrating our method. We will summarise and discuss the results in Section 5 and conclude with final remarks and an outlook in Section 6.
2 Background and related work
In this section, we first show how our approach fits into the general context of urban 3D modelling. We briefly present the state of the art of photogrammetric modelling and review existing approaches of geometric urban modelling, with a particular focus on the use of procedural modelling techniques.
Photogrammetry Citation(2), generally, is a technique that determines the geometrical properties of scenes and objects from pair or multiple images with different perspectives of view. It is normally used to generate commercially usable results for urban planning and city infrastructure management GIS platforms. The positioning accuracy of photogrammetric products can be guaranteed by means of bundle adjustment through a rigid sensors model, and the 3D building models derived from photogrammetric work can be viewed as realistic in both geographic position and texturing. Very high-resolution satellite imagery (e.g., QuickBird with 0.6 m or WorldView-2 with 0.5 m) constitutes a useful data source for generating large-scale 3D city models, especially when applied in conjunction with an universal geometrical model called rational polynomial coefficients (RPC) Citation(3 , Citation 4) instead of the traditionally rigid physical model, which, due to its complexity, needs an expert for handling. However, a typical photogrammetry generated model from top views will not include façade textures because only a very small amount of façade information can be seen in the top view imagery. Typically, reality-based 3D city models have been generated mainly from vertical aerial images; see overview in Ref. Citation(5). Nowadays, raw data from other sensors and platforms are increasingly used as well, like satellite images, oblique aerial images, UAV images and LiDAR data. A comparison of the different techniques can be found in Ref. Citation(6). A semi-automated photogrammetric technique is available with CyberCity-Modeler (CC-Modeler) Citation(7), which is still a leading approach in the field.
Similarly to the photogrammetry approach, one of the main goals of geometric urban modelling is to achieve a high level of detail in terms of building geometry. For an overview, refer to the literature Citation(8), which distinguishes between modelling of 3D urban structures and modelling of 2D layouts and structures. The latter case typically applies image synthesis methods in combination with interactive user input Citation(9) to create 2D urban layouts, including street networks, streets and parcels. Our technique of procedurally creating buildings and building details is based on the “computer graphics architecture” (CGA) shape grammar introduced in Ref. Citation(10 , Citation 11). The origins of CGA go back to the introduction of shape grammars in the seminal work by Stiny and Gips Citation(12–14) and to their adaptation to computer graphics applications presented by Wonka et al. Citation(15). Currently, CGA is implemented as part of the CityEngine software, originally developed by Procedural Inc., acquired by Esri in 2011, and currently being integrated into the ArcGIS suite Citation(16). In Ref. Citation(17), Mueller et al. presented a method for procedurally generating geometric façade details based on imagery using computer vision techniques. A semi-automatic version of this work is available in CityEngine through the so-called Façade Wizard, which is a tool that takes façade images as input and allows the user to quickly define detailed features in 3D, resulting in reusable shape grammar code. Note that the method presented here is not restricted to CGA, as it can similarly be applied to alternative shape grammars or grammar-based generative techniques. The main reason for using CGA in this work was its mature implementation within a publicly available application framework.
3 Workflow of combined photogrammetric and procedural modelling
In this section, we will give a detailed description of how we combine photogrammetric and procedural modelling techniques to create 3D building models. The general workflow is provided in Figure , comprising two separate pipelines, thereby leaving established workflows in either domain intact. The following sections will provide details for the individual photogrammetric processing steps and a brief overview of the CGA shape grammar. Then, we will show how CGA and CityEngine’s Façade Wizard are applied to generate rule-based building structures with high façade details.
3.1 Photogrammetric processing
As shown in Figure , the first step for 3D modelling in photogrammetry is to perform accurate geo-referencing of input imagery. We will in the following focus on satellite images, because those were used to generate the presented models. Though RPC parameters are provided, additional ground control points (GCPs) are needed to ensure accurate geo-referencing (bias-correction). GCP correction allows obtaining a geometrical accuracy in the sub-pixel domain. Here, we applied zero and first-order polynomial functions for geometry correction Citation(4); however, alternative, higher order correction models could be applied as well.
Figure 1 Overview of the overall workflow, indicating photogrammetric processing of high-resolution satellite images (top row) and procedural modelling (bottom row), resulting in the final 3D building models (last stage, bottom right).
After geo-referencing, a stereo model based on the corrected parameters is used for measurement. Both the digital terrain model (DTM) and building models were measured manually. The DTM was measured by means of profile measurement plus break-line measurement. For building measurement, we applied a semi-automatic approach using the in-house developed software CC-Modeler by Gruen and Wang Citation(7). In CC-Modeler, the operator manually measures the key points of buildings in stereo mode (corners and connection points) by a set of rules, and then, the software generates the topology of roofs and the building volume by projecting the roof geometry to the ground. Some measurement and modelling errors are unavoidable. Therefore, we applied post-processing using the Cyber-City Edit software to normalise parallel edges and remove hidden surfaces.
At the final step of building model editing, we applied texture mapping to the roofs and the terrain by means of re-projection of the geo-referenced images. The resulting models include reality-based building geometry with roof and terrain texturing.
3.2 Brief introduction to the CGA shape grammar
For better understanding the subsequent sections, we first provide a brief overview of the CGA shape grammar, introduced in Ref. Citation(11). CGA extends set grammars Citation(12) with concrete shape operations. The notation of rules and operations is borrowed from the L-system, introduced by Prusinkiewicz and Lindenmayer Citation(18) and so is the basic set of rules for transforming (translate, scale and rotate) shapes in space, with some adaptation for modelling of architecture. In contrast to the L-system, which belongs to the class of parallel grammars, CGA is a sequential grammar and therefore better suited to characterise architectural structures.
The basic idea of CGA is to start with an initial shape, either a 2D building footprint or a 3D building envelope, and to iteratively apply operations that modify the geometry in a top-down manner. The most important class of operations are splitting operations that subdivide shapes according to given rules. In addition, there are operations for component splits (i.e. to split a shape into its components of lesser dimension), repetition, transformation and for the specification of shape details such as colour, texturing and arbitrary metadata.
The following code exemplifies CGA notation and usage, and the result, a simple building centred in a given parcel, is shown in Figure :
Figure 2 Building model with roof generated by the CGA code provided in the text.
So far, we have shown how to generate an individual (and in this case simplistic) building starting from a 2D footprint. An important aspect of CGA is that the code can be further parameterised: In CGA, so-called attributes are arbitrary values defined at a specific geographic location. As commonly applied in GIS systems, the use of multiple attributes can be intuitively addressed by a layered approach as indicated in Figure : attributes can be of discrete nature, that is, they are directly assigned to elements in the layout layer. Otherwise, they might consist of samples from a raster image that are interpolated to obtain the value at the corresponding position. Alternatively, one may also define functional attribute layers, i.e., a = f(p), where f defines an arbitrary function with p as a 2D (or 3D) vector denoting the position.
Figure 3 Geo-located attributes are available to CGA using a layered approach. The CGA code can fetch layer attributes at a given location and use them during geometry generation.
With the help of attributes, the derivation process of a given initial shape gains access to parameters of its environment. Within CGA, attributes can be accessed like an ordinary variable:
This simple example uses the attribute height in order to extrude footprints accordingly into the y direction (note that y in CGA represents the vertical axis). An important additional aspect is that CGA attributes can be used to control level-of-detail (LOD), as illustrated in Figure .
Figure 4 Example illustrating level-of-detail (LOD) control using shape grammars: Initial footprint only (a); building model without details (b); sub-divided building model (c); and full detail including depth control (d). Shape grammars taken from CityEngine tutorial.
Referring to our workflow again, the above methodology is applied to use building-height and land-use attributes stored in an Esri shapefile in conjunction with CGA rules to generate the building models. The building models are then the basis for detailed façade generation, as shown in the next section.
3.3 Interactive creation of detailed façades
As indicated in the bottom-most row of Figure , an additional step in the workflow is the creation of detailed procedural façades. The Façade Wizard is an interactive tool within the CityEngine application, targeted at computer-assisted creation of textured 3D facades. With a single image of a building façade as input, the user can apply image rectification and basic CGA operations such as splits, repeats and depth adjustment, to obtain detailed façade structures quickly. The result is a CGA shape grammar code defining a hierarchical façade structure that can be reused with the overall procedural building code. When linking the façade code to the building models, additional parameterisation can be applied, in particular to define a specific level-of-detail (LOD) as shown in the CGA code below: LOD = 0 maps the original (rectified) façade image onto the side of the building volume without applying any further subdivision and depth. LOD = 1 applies splits and repeats, but no depth, while LOD = 3 applies full detail including depth information. In addition and if desired, only parts of the generated rules can be used. For instance, the user may chose to apply the complete façade rules to a surface starting from “Facade” or only apply information of a given excerpt further down in the hierarchy, for example, “Facade_1_1”:
To summarise, this section highlighted the three necessary components that allow the overall workflow to combine photogrammetry and procedural modelling: First, the photogrammetry workflow results in geo-referenced building footprints, 3D roof geometry models and detailed, textured roof geometries. Second, the footprints and building heights are used for the creation of CGA attribute layers, so the procedural building volumes can be created. And third, street level façade imagery is used to interactively create detailed façade structures and texturing. The resulting building geometry from photogrammetry and from procedural modelling are then fused (i.e. taking the procedural façade details and the photogrammetry roof details) into the final 3D building model. The next section will illustrate the methodology with a concrete example.
4 Case study: modelling of high-rise towers in Singapore’s Punggol area
Our case study in Singapore Punggol area starts with a stereo pair of ortho-ready imagery from the WorldView-2 satellite captured on 5 December 2010, covering an area of about 13.5 km2 (see a single image in Figure ). The data set contains a panchromatic band (with resolution of 0.5 m) and 4 multispectral bands (red, green, blue and near infrared with resolution of 2 m) separately. The data were fused by sub-pixel pan-sharpening using ERDAS Imagine. Four GCPs were acquired using Trimble GPS technology with accuracy of 2 cm in both horizontal directions and 5 cm in vertical direction. The geo-referencing, including bias correction was done using the ERDAS LPS module and the software Pixel grid from CASM, Beijing with a resulting accuracy of about 0.5 m in both horizontal and vertical direction.
Figure 5 WorldView-2 satellite image of Singapore’s Punggol case study area (covering ca. 13.5 km2).
The measurement of the key points for the 3D models was carried out using ERDAS Stereo Analyst and exported into encoded Esri shapefiles. The individual building models are generated using CC-Modeler and edited using CC-Edit. The final roof texture mapping was conducted using customized software by applying a direct geometrical re-projection. Figure shows the mapped area and the measured building models.
Figure 6 Case study area after photogrammetric measurements: 2D top-view (left) and 3D building volumes (right).
The geometric data obtained from photogrammetry were then used to create Esri shapefiles containing building footprints with attributed building heights. Both the textured photogrammetric roof geometry and the shapefiles were imported into CityEngine. Then, the Façade Wizard was applied to create detailed, textured 3D façade structures for the final buildings, as shown in Figure . In this case, the street level imagery was taken from manually shot photographs. The obtained data can then quickly be applied to a larger set of buildings and merged with the textured roof geometry, as shown in Figure .
Figure 7 Façade geometry generation using CityEngine’s Façade Wizard: Original street level photo taken (top left), sub-division and depth assignments (bottom row), resulting final façade structure (top middle), which can then be applied to the whole building structure (top right).
Figure 8 Final result showing multiple buildings in the case study area with textured roof geometry obtained using photogrammetry and procedurally modelled detailed façade structures.
5 Results and discussion
The methods described can be applied using commonly available software tools and all examples presented in this study were produced using the mentioned tools, thereby we show that the approach is applicable in practise and productively available to the end-user.
Our examples show that the methodology integrates well within existing frameworks of photogrammetry and procedural modelling and combines advantages from both sides to ultimately increase the productivity and the level of detail of the created 3D models. In particular, it uses the strengths of maintaining reality-based accuracy photorealistic texturing of photogrammetry, while employing the power of quickly creating complex building details using procedural modelling. The combined approach clearly surpasses manual techniques.
Currently, we see two main issues of the approach: Firstly, writing the rules requires scripting experience in yet another programming language (CGA). For people used to interactive modelling techniques, thinking in shape grammars can be challenging. However, our experience with students working with the software has shown that the efforts can be very rewarding once they understand how easy it can be to formulate a model in terms of shape grammar. Also, CGA is not a full-blown programming language and imposes some limitations on geometry generation, in particular for non-regular structures. The on-going combination of manual scripting with interactive rule-based modelling as seen in current CityEngine developments will certainly improve the situation.
Secondly, texture quality remains an issue, for example, in terms of resolution as well as geometric and colour consistency. Here, additional image pre- and post-processing must be considered in order to improve the texture quality of the final result.
6 Conclusion and future work
In this study, we presented a methodology that combines photogrammetry and procedural modelling techniques to create 3D building models. Altogether, our methodology aims at increasing the level of detail of 3D models, with the ultimate goal to make such models more usable for urban design tasks. While the present results show that we can achieve the aims using existing software tools, though there remain several detail issues to be resolved as pointed out in the previous section.
Beyond these issues, there are additional opportunities for future work: first, we would like to further extend the procedural modelling approach with additional data that are available from additional data sources. For instance, such data could be used to create floors, and even more detailed interiors. The resulting building geometry would then be available for more in-depth analysis or for agent-based pedestrian simulations, and the extended use of shape grammars could also be employed for GPU accelerated rendering of large-scale scenes Citation(19).
Finally, there is a class of simulation techniques that operate at urban scale and require concrete 3D building information. For example, Ratti et al. Citation(20) used digital surface models (DSMs) to simulate energy consumption and study it with respect to urban texture. Robinson et al. Citation(21) described a software system that uses 3D models for simulating solar radiance and radiation exchange. Here, our method is located in between urban modelling, urban simulation and other simulation techniques and could provide a novel means of connecting them, for instance through rule-based reporting or through generation of DSMs from the created 3D building models.
Notes on contributors
Stefan Müller Arisona is a principal investigator at ETH Zurich’s Future Cities Laboratory (FCL) in Singapore. His research focuses on the study and implementation of interactive design and modelling tools. Most recently, he published a book on Digital Urban Modeling and Simulation (Springer CCIS Series).
Chen Zhong is a PhD researcher at FCL, and she investigates spatiotemporal modelling methods for urban design.
Xianfeng Huang is a post-doctoral researcher at FCL, working on the advancement of reality-based 3D and 4D modelling.
Rongjun Qin is a PhD researcher at FCL, where he develops new techniques for tracking changes in cities over time.
Acknowledgements
This study was established at the Singapore-ETH Centre for Global Environmental Sustainability (SEC), co-funded by the Singapore National Research Foundation (NRF) and ETH Zurich.
Related Research Data
References
- Gruen , A. , Behnisch , M. and Kohler , N. 2009 . Perspectives in the reality-based generation, nD modelling, and operation of buildings and building stocks . Build. Res. Inform. , 37 ( 5–6 ) : 503 – 519 .
- Wolf , P.R. and DeWitt , B.A. 2000 . Elements of Photogrammetry with Applications in GIS , New York , NY : McGraw-Hill .
- Tao, C.V.; Hu, Y. 3D reconstruction methods based on the rational function model. Photogramm. Eng. Remote Sensing. 2002, 68 (7), 705–714.
- Fraser, C.S.; Hanley, H.B. Bias compensation in rational functions for IKONOS satellite imagery. Photogramm. Eng. Remote Sensing. 2003, 69 (1), 53–57.
- Gruen, A. Building extraction from aerial imagery. In Remote Sensing of Impervious Surfaces: Weng, Q., Ed.; Taylor & Francis Series in Remote Sensing Applications, 2008; pp 269–295.
- Baltsavias , E.P. and Gruen , A. 2003 . “ Resolution convergence – A comparison of aerial photos, LIDAR and IKONOS for monitoring cities ” . In Remotely Sensed Cities , Edited by: Mesev , V. 47 – 82 . London : Taylor & Francis .
- Gruen, A.; Wang, X. CC Modeler: A topology generator for 3-D city models. ISPRS Commission IV. Symposium on “GIS – Between vision and application”, Stuttgart/Germany, September 7–10, 1998; International Archives of Photogrammetry and Remote Sensing, Vol. 32, Part 4, pp 188–196; ISPRS J. Photogramm. Remote Sensing. 1998, 53, 286–295.
- Vanegas , C.A. , Aliaga , D.G. , Wonka , P. , Müller , P. , Waddell , P. and Watson , B. 2010 . Modelling the appearance and behaviour of urban spaces . Comput. Graph. Forum , 29 ( 1 ) : 25 – 42 .
- Aliaga , D.G. , Vanegas , C.A. and Beneš , B. 2008 . Interactive example-based urban layout synthesis . ACM Trans. Graph. , 27 ( 5 ) : 1 – 10 .
- Parish, Y.I.H.; Mueller, P. Procedural modelling of cities. In SIGGRAPH ‘01: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques, New York, NY, USA, 2001, pp 301–308.
- Mueller , P. , Wonka , P. , Haegler , Mueller , P. , Ulmer , A. and Van Gool , L. 2006 . Procedural modelling of buildings . ACM Trans. Graph. , 25 ( 3 ) : 614 – 623 .
- Stiny, G.N.; Gips, J. Shape grammars and the generative specification of painting and sculpture. In Information Processing ‘71: Freiman, C.V., Ed.; North Holland: Amsterdam, 1972, pp 1460–1465.
- Stiny, G.N. Pictorial and Formal Aspects of Shape and Shape Grammars and Aesthetic Systems. Ph.D. Dissertation, University of California, Los Angeles, 1975.
- Stiny , G.N. 1980 . Introduction to shape and shape grammars . Environ. Plan. B Plan. Des. , 7 ( 3 ) : 343 – 351 .
- Wonka , P. , Wimmer , M. , Sillion , F. and Ribarsky , W. 2003 . Instant architecture . ACM Trans. Graph. , 22 ( 3 ) : 669 – 677 .
- Esri. CityEngine, 2012. http://www.esri.com/software/cityengine
- Mueller , P. , Zeng , G. , Wonka , P. and Van Gool , L. 2007 . Image-based procedural modelling of facades . ACM Trans. Graph. , 26 ( 3 ) : 85
- Prusinkiewicz , P. and Lindenmayer , A. 1996 . The Algorithmic Beauty of Plants , New York , NY : Springer .
- Haegler , S. , Wonka , P. , Müller Arisona , S. , Van Gool , L. and Mueller , P. 2010 . Grammar-based encoding of facades . Comput. Graph. Forum , 29 ( 4 ) : 1479 – 1487 .
- Ratti , C. , Baker , N. and Steemers , K. 2005 . Energy consumption and urban texture . Energ. Build. , 37 ( 7 ) : 62 – 776 .
- Robinson , D. , Campbell , N. , Gaiser , W. , Kabele , K. , Le-Mouel , A. , Morel , N. , Page , J. , Stankovic , S. and Stone , A. 2007 . SUN tool – A new modelling paradigm for simulating and optimising urban sustainability . Solar Energ. , 81 ( 9 ) : 1198 – 1211 .