Use of gaming technology to bring bridge inspection to the office

Abstract

This paper proposes a novel method for bridge inspection that essentially digitises bridges using Light Detection and Ranging (LIDAR) so that they can be later inspected in a virtual reality (VR) environment. The work uses conventional terrestrial LIDAR together with affordable VR hardware and freely available software development kits originally intended for authoring computer games. The resulting VR app is evaluated for a case study involving a typical masonry bridge, comparing the proposed technique with traditional inspection methods. The new approach promises to be highly effective in terms of interpretation of results, accessibility to critical areas and safety of inspectors. The work represents an important step towards the creation of digital twins of important assets in the built environment. Recent bridge collapse incidents have affected local economies, traffic congestion, and in some extreme cases led to a loss of life. The work is timely as law making agencies are paying greater attention to structural rehabilitation. This paper will be of particular interest to bridge engineers, construction professionals and law makers and could lead to future revisions of bridge inspection processes and standards.

Keywords:

• Bridge
• inspection
• virtual reality
• LIDAR
• digital twin
• evaluation
• maintenance
• non-destructive methods

1. Introduction

State highway bridges are part of the critical transportation infrastructure and can be considered as the backbone of a nation’s economy (Merkle & Myers, 2006). In recent history, the construction industry experienced an exponential growth in bridge construction after the advent of new materials, technology and skills. In the USA, more than 70% of the bridges were built before 1935 (Golabi, Thompson & Hyman, 1993), whilst in the UK, most bridges were constructed between the 1950s and 1960s (Flaig & Lark, 2000). Approximately 70% of bridges in New South Wales, Australia were built before 1985 (Rashidi & Gibson, 2011). Due to increasing traffic loads, climatic deterioration of the structure (as a consequence of continuous environmental exposure) and natural disasters, bridges suffer damage and require rehabilitative measures for safe operation.

According to a recent survey conducted in the US, 55,710 bridges have been declared as structurally deficient and need immediate attention (Holcombe, 2017). Moreover, recent bridge collapse incidents in Italy, South Africa, the USA and India has forced law making agencies to pay more attention to structural rehabilitation. Failure to effectively maintain these structures can have a catastrophic effect on a nation’s economy, day-to-day activities and in the worst case, may lead to loss of life. Originally, bridge inspections were carried out as a result of an obvious incompetency or as a consequence of warnings received from sources mainly outside the bridge network system (Branco & Brito, 2004). However, ‘to keep good bridges good and prevent more bridges as being classified as deficient, timely inspection becomes a necessity’ (Rehm, 2013).

Bridge inspection techniques are broadly summarised into four categories, namely, Safety Inspection (Routine Surveillance), General Inspection, Principal Inspection and Special Inspection. Each technique varies in terms of its frequency, duration and accuracy. Safety Inspections are performed for a cursory check of highway structures that are subjected to obvious deficiencies or signs of deterioration such as collision damage to bridge supports, insecure expansion joint plates and support restraints, or spalling concrete. These inspections are undertaken at frequencies which ensure timely identification of safety related defects which might lead to accidents or high maintenance costs in the future.

Approximately 24 months after a bridge has been built, a General Inspection is carried out. This mainly focuses on a bridge’s most exposed areas and involves direct visual inspection, without the need for special access equipment or traffic management. After this, in six yearly intervals, Principal Inspection is performed. Its purpose is to provide a qualitative assessment of all the inspectable parts of a highway structure by physical examination (detailed visual inspection) within touching distance. These inspections are comprehensive and provide more detailed information than a General Inspection. If a functional flaw or defect is detected which is outside the remit of the General/Principle Inspection regime, a Special Inspection is performed. This comprises of a close visual inspection, in situ destructive, static and dynamic load tests and laboratory tests. Detailed structural analyses are carried out for dead loads, and superimposed dead loads and live loads. The strength of the structural members is determined and the allowance for future loading is also evaluated (Inspection manual for highway structures, 2007).

1.1. Motivation

Since the beginning of the US federal bridge inspection program in 1968, visual inspection has been the primary technique for assessing the serviceability and performance of these structures (Robert, 2007). This approach is time consuming and requires certain tools to access different elements of a bridge; which may be unsafe for the inspector or cause traffic disruption to the user. In certain cases, these tools may not be feasible specially if the geometry or location of the bridge is complicated (Bolourian, Soltania, Albahria, & Hammad, 2017). Drawbacks of visual inspection include interpretability of results, the static nature of inspection, difficulties regarding accessibility to critical areas and ensuring the safety of inspectors (Agdas, Rice, Martinez, & Lasa., 2015). Therefore, new automated inspection systems are required to provide a safer working environment as well as more efficient and accurate methods for bridge inspection (Liu et al., 2014).

The motivation of this research is to investigate whether virtual reality (VR) could be used to address the limitations and difficulties faced by engineers when carrying out visual inspection. A 3D digital inspection environment is proposed in which Light Detection and Ranging (LIDAR) images are captured and processed for display in a custom app written by the authors. This is written using the computer games authoring platform, Unity and is displayed on a Samsung Gear VR headset which uses an android smart phone for both processing and display. A typical masonry arch railway bridge is used as an example to demonstrate a promising improvement in Principal Inspection. In the paper, the hardware and software required to perform the inspection are described. Sufficient detail is provided for those readers who may wish to replicate the work. Results of the case study are presented, highlighting the advantages and disadvantages of the new approach, which will improve with the continuing development of technology over the passage of time.

2. Background

This section gives an overview of the historical development of VR systems and their application. The development of 3D scanning and its reported use in bridge inspection is also discussed. The conventional procedure for performing Principle Inspection is highlighted, with a focus on its limitations. Emphasis is placed on how VR can be used in bridge inspection to resolve these limitations.

2.1. Development of virtual reality systems

From a philosophical point of view, the desire to experience presence in an immersive simulated environment has always existed. Since the renaissance, artists and painters have used realistic imaginary to create optical illusions of 2D objects in three dimensions, submersing the viewer in an environment where they could feel part of the scene (Shields, 2003). In the 1900s, early cinema took this artistry one step further. In 1915, Edwin Porter and William Waddell demonstrated a 3D cinema experience using red-green anaglyph technology. The first colour stereoscopic 3D movie, Bwana Devil, was directed by Arch Oboler in 1952 and 3D movies became popular in the 1950s and 1960s (Jockenhovel, 2011).

The next step was the development of technology for telepresence ‘the use of technology to operate machinery by remote control or to create the effect of being at a different or imaginary location’ (Collins English Dictionary, 2018). In 1956, Morton Heilig, a cinematographer, built a machine called the Sensorama – the earliest VR technology that provided multimodal experiences, including 3D views, smells, sounds and vibration. Although Sensorama embodied the sense of telepresence, it was still not possible for the user to interact with the immersive environment until the development of the first head-mounted display, nicknamed ‘the Sword of Damocles’ (Sutherland, 1968). The helmet-like digital device could track the physical movements of the user and generate digitally composed surroundings from the user’s gaze. In 1970's, real-time interaction advanced further with the ability to render a Virtual Environment (VE) in an artificial reality laboratory (Krueger, 1991).

Two further important advances included the Large Expanse Extra Perspective optical system (Howlett, 1983) which improved the depth of view and the development of technology to facilitate physical interaction with VEs, using ‘Datagloves’ (Zimmerman, 1982) and (Rheingold, 1991). In the late 1980s, the term ‘virtual reality’ was coined (Ellis, 1991). It is now formally defined as ‘a computer-generated simulation of a three-dimensional image or environment that can be interacted with in a seemingly real or physical way by a person using special electronic equipment’ (McMillan, Flood, & Glaeser, 2017).

From the 1990s onwards, the World Wide Web provided tremendous potential for VR technological development by giving access to the information and virtual communities of experts (Lau, Lau, & Kan, 2013). Due to improvements to computer processing speeds and miniaturisation of electronics, companies such as Samsung have been able to develop mass market VR products such as the Samsung Gear VR system. Today, the market size for VR products and services is estimated to be more than $7billion. Important developments in the history of VR are summarised in Figure 1.

Figure 1. Major developments in virtual reality technology.

2.2. Applications of virtual reality

VR technology has been applied to a broad range of disciplines. In aviation, VR has been used to develop virtual flight simulators (Hays, Jacobs, Prince, & Salas, 1992). In the military, it has been used as a recruitment tool (Zyda, 2005) and to create immersive combat environments for training soldiers (Lele, 2013). Scientists have used VR to visualise research outputs in fields such as cosmology (Bryson, 1993), magnetohydrodynamics (Margetts, 2002) and aerodynamics (Seu, Hewitt, & Margetts, 2018), significantly improving scientific insight and interpretation. VR has also been used in journalism to increase the impact on the audience of a news story (Goldman, 2018) as well as healthcare and medicine, for example, the treatment of post-traumatic stress disorder (Leadem, 2016). In the retail sector, businesses such as Walmart are increasingly using VR to train their employees in a virtual classroom before they start a new role (Feloni, 2017).

Applications in the engineering sector are also broad, ranging from training for nuclear safety (Shuster, 2010) to decision-making in the construction industry for construction management (Sampaio, Ferreira, Rosário, & Martins, 2010), site design (Harrison, Sletner, Preble, Harrison, & Anderson, 2015), planning (Comunale, 2017) and inspection (Omer, Hewitt, Mosleh, Margetts, & Parwaiz, 2018). It is also used for evaluating the processes of assembly and maintenance in manufactured products (Shiratuddin & Zulkifli, 2001). In the field of transportation, VR is being used to understand traffic densities and flow patterns, enabling engineers to design optimal layouts for highways, signalling and crossings (Jin, Yan, & Ito, 2007). VR is used to forecast possible damage due to floods, earthquake or geological disasters (Eon, 2018). Architects use VR to effectively communicate their ideas to their clients, showing them features such as building elevation, terrain and interior decoration.

In the construction industry, VR is used to render 4D models (3D + time), assisting engineers and managers in many phases of construction projects, from initial design through to final construction. Without this technology, many problems may arise due to the inability of stakeholders to visualise their projects before they are built. As a consequence, work may have to be stopped or plans altered leading to time and cost inflation. VR is used to provide the client and construction team with an immersive experience of the project before it is built. Potential conflicts in the design and changes in the architectural plans can be modified before there is an issue. Construction engineers can ‘walk inside’ the VE and view all the utilities systems installed enabling them to identify any issues that may be easily overlooked when viewed on a 2D screen. VR also puts clients in a better position to confirm if their idea resembles the work proposed to be carried out on site (Comunale, 2017).

In summary, the literature shows that VR is becoming an essential technology that has a promising future in every discipline. There is evidence that VR is seeing growing use in this sector. A recent study carried out by ARC confirmed that 65% of civil engineering firms agreed that VR improved visualisation of projects and has potential in design and construction practice (Brown, 2017). The main application areas of VR are summarised in Table 1. This review confirms that the work presented herein is the first time that VR has been evaluated for bridge inspection.

Table 1. Applications of virtual reality.

Use of virtual reality	Description	Reference
Military	Training and soldier assessment	Lele (2013)
Army	Recruitment of soldiers	Zyda (2005)
Aviation Authorities	Flight Simulations	Hays et al. (1992)
Journalist	Share news in an effective way	Goldman (2018)
Medical Science	Treatment of PSTD	Leadem (2016)
Business	Communication and training of employees	Feloni (2017)
Scientists	Visualisation of Cosmology	Bryson (1993)
Manufacturing Industry	Optimisation of design	Shiratuddin et al. (2001)
Construction Industry	To improve analysis and design management	Comunale (2017)
Transportation Industry	Simulation of driving	Jin et al. (2007)
Structural Engineering	Design shape and inspection of bridges	Omer et al. (2018)

2.3. 3D Scanning

The first reported attempts at 3D scanning were carried out in 1953 by the US military who began experimenting with an optical measuring device using light and high-speed shutters (Newton, 2017). Around the same period of time, the Ferranti Company of Scotland released a dual axis electronic probing machine called the Coordinate Measuring Machine that was entirely mechanical and relied on physical contact (i.e. did not use light).

Nowadays, there are a variety of light-based contactless scanning systems available on the market. Cheaper devices use conventional or pulsed lamps with a high-speed shutter. More expensive systems, such as LIDAR, use laser light and have been developed for more accurate range measurement. The first reported use of LIDAR was in the early 1960s, initially for atmospheric studies by the US Military (Molenby, McMananmon, & Kobayashi, 2016). In 1969, a laser rangefinder was used to determine the distance between earth and moon using a target mounted near the spacecraft, Apollo 11 (Woods, 2011).

A basic LIDAR system consists of a transmitter, a receiver and a signal-processing unit (Liu, Chen, & Hasuer, 2012). A pulsed or continuous light is emitted from the transmitter to the target object and a part of it is reflected back to the receiver. By multiplying the speed of light with its travel time, the two-way distance between the scanner and the object can be calculated. Until the 1980s, LIDAR systems used only a single line of data to scan objects and produced a point cloud as a cross-sectional profile. Thus, the system had limitations for covering wide areas. Early in the 1980s, the ‘stripe’ method of optical scanning was developed which was faster and more accurate than ‘point’ or ‘area’ methods.

After 1985, scanners used white light, lasers and shadowing to capture a given surface (Ebrahim, 2014). In 1992, the French company MENSI (now a Trimble company) introduced the S-series 3D laser scanners, primarily for use in industrial applications such as nuclear power plants. In 1996, 3D scanners perfected the design with ModelMaker, a combination of manual digitisation and stripe laser scanning. It was able to accurately produce complex models and digitise those models with texture and colour.

These early systems were incredibly slow and imperfect but compared to manual surveying and measurement techniques routinely employed at the time, it was a huge step forward. The recent explosion in computing power has seen improvements in resolution and speed for a reasonable budget, facilitating the uptake of 3D scanning in the mechanical, automotive and civil engineering sectors. The technological development of 3D scanners over time is summarised using the timeline in Figure 2.

Figure 2. Historical development of 3D scanners.

2.4. Use of LIDAR in bridge engineering

The latest imaging technology is being continually adopted in the field of bridge engineering. For example, image analysis algorithms have been used to automatically identify defects in engineering structures (Adhikari, Moselhi, & Bagchi, 2014; Hampel & Maas, 2009; Hutchinson & Chen, 2006). Typically, these techniques rely on photography and are influenced by lighting conditions. In some cases, they require additional supplementary information (Laefer, Truong-Hong, Carr, & Singh, 2014). Terrestrial LIDAR scanning is nowadays a standard technique used in bridge health monitoring (Gisresources, 2016). Compared with photography, the imaging is more accurate and can be used to quantify 3D information such as mass loss (Teza, Galgaro, & Moro, 2009) as well as the depth of defects (Byrne, Pakrashi, Schoefs, & Ghosh, 2014; Mizoguchi et al., 2013).

LIDAR has also been used to measure the vertical deflection of bridges under loading conditions (Lichti & Harvey, 2002) and acquire the geometry of historical arch bridges (Riveiro, Arias, Armesto, Rial, & Solla, 2008). In some cases, the 3D scans are combined with close range photogrammetry (Riveiro, Jorge, Varela, & Jauregui, 2013), enabling the detection of degenerated surfaces (biological crust, cracking, spalling or volume loss) based on the analysis of point-based coordinates and the RGB values of point clouds (Hong, Laefer, & Debra, 2014). Whilst these are important milestones, prior to the research presented herein, no other authors in the field of structural engineering have reported the use of interactive VR environments to display or examine 3D scans and photogrammetry data.

2.5. Conventional procedure to carry out principal inspection

A detailed review of the Principal Inspection technique was recently carried out by the authors (Omer et al., 2018), so only a brief overview is provided here. Primarily, the aim of Principal Inspection is to observe and record areas of stress concentration using rough sketches, photographic evidence and drawings. The general procedure adopted before conducting an inspection is to study all relevant structural drawings available, any past inspection records or repair work on the bridge. Whilst conducting an inspection, any difference between the intended use of a structure when it was first constructed and how it is used now must be recorded.

The National Bridge Inspection Standards – NBIS (Thompson, Shepard, & AASHTO, 2000) divides bridges into four main structural units: superstructure, substructure, deck slab and culverts. These units are further divided into elements depending on the details of inspection required. Grades ranging from 1 to 9 are attributed to each of the four main parts of the bridge. The grades describe the level of deterioration, for example, grade 9 is given for a bridge without any damage and defects, whilst grade 0 indicates a bridge that has deteriorated to such an extent that it can no longer be repaired. Once all the elements are graded, an overall grade for the bridge is determined and this is checked against national standards to determine the severity index of attention (Federal Highway Administration, 2004). In Principal Inspection, the precision and accuracy of the assessment can vary according to the professional qualifications, motivation and inspection equipment of those who actually carry out these inspections.

2.6. Proposed principal inspection technique using virtual reality

The research presented herein proposes the combined use of VR and LIDAR as a novel tool for the inspection of bridges and demonstrates its use for a typical masonry bridge. Terrestrial LIDAR is used to digitise the geometry of a selected masonry bridge. This results in a point cloud where each point has x, y, and z coordinates together with an RGB intensity value. The point cloud is rendered to produce surfaces, using post-processing software, which is then ported to a VR environment which depicts a real-world scenario. The vision is that engineers could perform Principal Inspection in VR, overcoming the drawbacks discussed earlier that are associated with conventional techniques, thus, improving the overall standard of Principal Inspection. It is intended that the proposed methodology will contribute to a revision of the industrial standards of Principal Inspection. Instead of the office going to bridges, the bridges come to the office.

3. Method

This section outlines the details of the experimental setup. First, the vendor specifications for each type hardware and software used are described. Next, the location and details of the masonry bridge under study are presented. Finally, the pre-processing and post-processing stages of the newly proposed inspection technique are explained.

3.1. Hardware

The hardware used included a Leica Scan Station P40 laser scanner, a Samsung Galaxy S6 smart phone, a Samsung Oculus Gear VR headset with a remote control and a HP Pavilion laptop running the windows operating system. The Leica Scan Station P40 is used to capture a 3D scan of the bridge. It weighs 12.25 kg and has dimensions of 238 mm × 358 mm × 395 mm. The data are captured at a rate of 1,000,000 points per second (Leica Geosystems, 2018). The most important technical details of the Leica Scan Station P40 are listed in Table 2.

Table 2. Details of Leica Scan Station P40.

S. No.	Feature	Value
1.	Range accuracy	1.2 mm
2.	Angular accuracy	8° vertical and horizontal
3.	Position accuracy	3mm
4.	Resolution	10 mm
5.	Wavelength	150 nm invisible
6.	Laser	Class 1
7.	Beam divergence	< 0.23 mrad
8.	Range	270 m
9.	Field-of-view	360° horizontal; 270° vertical

The authors have written their own custom VR application for Principal Inspection. The VR app is run on a Samsung Galaxy S6 smart phone with a storage capacity of 32 GB; a 5.1 inch screen with a resolution of 1440 × 2560 pixels; 3 GB of random access memory and a built-in microUSB 2.0 port for charging and connecting to the Samsung GEAR VR headset. For online networking and connecting to the Oculus store, the device has a WLAN Wi-Fi connection. The smart phone used for this research used version 7.0 of the Android operating system (Samsung, 2015).

A Samsung Gear VR Headset was used to view the VR app. The headset reproduces media content at a refresh rate of 60 Hz or above. The display has an aspect ratio of 16:9, a field of view of 101° and a lens resolution of 2560 × 1440 pixels. Head movements and position data are captured by the headset using a built-in accelerometer, gyro sensor and proximity sensor. User interaction with the VR environment can be controlled using a remote control connected by bluetooth. The headset is lightweight (345 g excluding the weight of the smart phone) and has physical dimensions of 207 mm × 102 mm × 98 mm (Samsung Oculus, 2017).

Writing the VR app and pre-processing the imaging data are carried out using an HP Pavilion Core i5 workstation running the 64-bit Windows 10 operating system. The system has 4 intel(R) core (TM) i5-6200 CPU @ 2.30 GHz processors. To ensure smooth operation of software, the system has an internal memory (RAM) of 12 GB. An AMD Radeon (TM) R5 M430 graphics card is attached in addition to the Intel(R) HD Graphics 502 chip. The computer has an overall storage capacity of 500 TB which was more than enough to store the processed data. All the hardware devices used in the project are shown in Figure 3.

Figure 3. Hardware device used in proposed technique of inspection.

3.2. Software

Several different software packages were used for processing the data and building the VR app. Leica’s Cyclone REGISTER 360 version 1.4.3 was used to register, edit and downsize the raw scan data. Unwanted areas of the scan were cropped and the resulting point cloud was exported in .pts format for further processing. Cloud Compare v2.91.1 was used to convert the point cloud from .pts to .ply format. The VR app for Principal Inspection was developed using Unity 2017.2, a GUI based software development platform which uses a simplified C# coding language. The software is typically used to develop video games for web plugins, desktop platforms, consoles and mobile devices. It can import 2D/3D models from CAD packages and other drafting software. Another important capability relevant to writing VR apps is that Unity supports adding VR features such as teleportation and navigation.

3.3. Developing a Unity application

In order to view architectural models and point clouds in the VR headset, a number of steps are required. Architectural models can be created using AutoCAD 2017 and exported as a .fbx file; a proprietary format for 3D models. To import a point cloud, a script is used to read the point cloud (saved in .ply file format) and then each point is plotted within Unity. Once the architectural model and/or point cloud is loaded into Unity’s scene, the initial position of the camera is set manually. Extra coding is required to enable the user to navigate in the scene using a button on the bluetooth controller. Once the scene creation has been completed, specific settings are chosen in Unity to be able to export the scene within an app for viewing on the VR headset. For example, Unity needs to know that it is building an app for the Android operating system so that it is compatible with the Samsung Galaxy S6 smart phone. Furthermore, additional plugins such as the Android SDK Manager and Java Runtime need to be installed in Unity.

A variety of different texture compression methods are available that can be used to reduce the memory requirements of the app. In this work, the Adaptable Scalable Texture Compression (ASTC) texture compression method has been used. This preserves the quality of the model, which is an important requirement if the VR app is to be used to examine features such as cracks. Unlike other compression methods, ASTC uses variable block sizes to store the data instead of a single block size, ensuring that the quality of the graphics is preserved (Microsoft, 2018).

The software development involves choosing appropriate ‘player settings’ in Unity for features such as ‘resolution scale mode’ and ‘presentation’. The frame buffer alpha value is not fixed to a specific value as this will either turn all objects to completely opaque or completely transparent. The ‘resolution scale mode’ is kept to its default value of ‘disabled’, which minimises memory requirements. Another important option ‘build type’ is set to ‘always’ which ensures all the rendering takes place whilst the app is being loaded. In terms of user experience, this ensures that there is an uninterrupted display of the scene the moment application executes. The ‘aspect ratio’ is set to ‘wide screen’ as the Samsung Gear VR can only display applications in wide screen mode. Similarly, the setting ‘orientation’ must be set to ‘landscape’ for the same reason. The option ‘32-bit buffer display’ is used to improve the quality of the graphics. The ‘splash image’, which is a welcome screen displayed before the execution of the app, is disabled to reduce memory requirements. ‘Colour space’ is set to ‘gamma’ as ‘linear’ is not supported by smart phones.

Some other key settings in Unity include: setting the API to ‘auto graphic’; enabling ‘multithread rendering’ (to speed up building the app); setting the minimum API level to Android 5.0; selecting ‘Oculus’ from the ‘XR’ options (which ensures Unity builds a VR app for the Samsung GEAR VR headset) and giving the app a name, setting ‘model package name’ to ‘.com.Manchester.bridgeinspection’.

The final step is to sign up the application with the Oculus store, which requires the provision of a file with device information that is added to Unity’s asset panel (where all the scenes are saved). After completing all the steps, the scene is debugged and if no errors are found, the file is directly ported to the smart phone by pressing the export button in the player settings menu.

3.4. Case study of a masonry bridge

A typical masonry bridge was selected to evaluate the proposed VR technique for Principal Inspection. The bridge is located in Manchester, UK and has global coordinates (according to the Universal Transverse Mercator (UTM) system) of 550841.02 m E, 5925414.12 m N in zone 30U. The bridge is a typical railway masonry arch bridge, made of red bricks and over 100 years old (Figure 4). Due to its extensive use and age, the bridge has already been subjected to external strengthening and is still subject to regular maintenance by the local authorities. A map showing the location of the bridge, together with a photograph is shown in Figure 5.

Figure 4. Elevation and sectional view of the masonry bridge.

Figure 5. Railway network bridge Manchester, United Kingdom.

3.5. Proposed workflow of inspection

In this section, the proposed workflow is written in the style of a set of procedures, which highlights that much of the process is similar to Principal Inspection. The first and foremost step is to schedule an inspection based on the availability of resources, traffic management and weather conditions. The next step is to prepare a method statement which summarises the details of all the activities to be performed on site, specifically any safety procedures and details of traffic management. Upon arrival on site, a careful check should be carried out to confirm the identity of the structure.

Once the structure and all precautionary measures have been confirmed, the LIDAR is attached to the tripod stand and levelled at the specified location of interest. The height of the LIDAR is measured and its position on the ground is recorded. The angle of the scan is set to between 0° and 360°. The desired scan resolution and sensitivity is set to ‘12 mm’ and ‘low’, respectively. A point cloud and its corresponding image are selected as the output option from the scanner. To fully capture all the defects, the LIDAR is placed at three different locations and three data sets are collected. To help combine the data sets later, markers are placed on site in positions where they are visible in all three scans. The data are stored in a pen drive for later post-processing.

The point cloud is saved in the proprietary .bin format and the associated photographic images are captured in .jpg format. The two datasets are co-registered using the Cyclone REGISTER 360 software version 1.4.3. In the software, general settings such as ‘units’; ‘number of decimal places’; ‘link’ settings (to customise the error threshold) and ‘cloud-to-cloud’ settings are set to default values. Using the height of the LIDAR and its recorded location, using markers on the ground as reference points, the point clouds are combined and superimposed with the photographic images. Unwanted points and noise are manually deleted from the scan. The final cleaned up dataset is then saved in the .pts format for further processing.

The file conversion from .pts to .ply (needed to import the data into the VR app) is carried out using Cloud Compare which reads the x, y and z coordinate values as well as the RGB intensity data for each point from the .pts file. The point cloud is enhanced to brighten the view and is then exported in the .ply format. In the final step, the point cloud file is imported into Unity. Interaction features such as navigation and teleportation are added to the scene and the appropriate app settings are applied. Once all the settings are finalised, the file is exported to the Samsung Galaxy S6 smart phone. The bridge can now be inspected in the Samsung Gear VR headset. Results and conclusions from the inspection are drawn and evaluated using the element condition rating system as defined in the CSS Guidance Documents. These results are then reported to the authorities concerned for further action. The entire workflow procedure is summarised in the flow diagram shown in Figure 6.

Figure 6. Flow diagram of visual inspection procedure.

4. Results

Different views of the bridge displayed in the virtual world are shown in Figure 7. The bridge is essentially a digital twin. When wearing the headset and immersed in the VR experience, the inspector is completely disconnected from the real world and can focus on carrying out an accurate inspection, safely and with no distractions. Table 3 lists the most important tasks carried out during Principal Inspection and assesses the possibility of using the proposed technique. As can be seen in Figures 8 and 9, most of the features are visible with the naked eye. These are highlighted by white circles and white rectangles in Figures 8 and 9, respectively. Some require further post-processing.

Figure 7. Multiple views of the bridge replicated in a 3D VR environment.

Figure 8. Visible features in proposed inspection technique.

Figure 9. Visible features with current inspection technique.

Table 3. Assessment of feature visible in proposed technique.

S. No.	Visual inspection parameters	Possibility of assessment in virtual world	Further processing in MATLAB
1.	Geometric measurements	Yes	Yes
2.	Cracks	Yes	Yes
3.	Differential movements	Yes	Yes
4.	Open joints	Yes	Yes
5.	Degree of wetness	Yes	No
6.	Mortar loss	Yes	No
7.	Leaching	Yes	No
8.	Chemical attack	Yes	No
9.	Eroded surfaces	Yes	No
10.	Presence of vegetation	Yes	No

Features typical of an inspection, such as identifying the presence of water and assessing its effects on the fabric of the bridge can be clearly identified, particularly through the decolourisation of the masonry bricks. Furthermore, areas of leaching are also easily noted by recognising variations in the texture of the bridge. The presence of mortar and areas of mortar loss are also clearly visible in the virtual world, as are eroded surfaces and open joints in the bridge. These are important as they affect the strength of the structure. Vegetation, including lichen and other visible colonies of micro-organisms (which might be indicative of internal deterioration) are also captured and visible in the app. Notably, all of these features are easily spotted when inspecting the bridge using the proposed VR technology.

Based on the case study performed, the feasibility of the proposed technique in terms of implementation, time and cost is evaluated by a subjective comparison of bridge inspection in VR with bridge inspection using conventional tools. The comparison in Table 4 uses adjectives to highlight the relative merits of each approach. Using VR, important elements of inspection such as navigation, access to critical areas, interpretation of results and lighting problems are greatly improved. For example, as the bridge has a clear height of 13 m, using the traditional inspection technique to inspect one of the critical areas such as the connection between the arch ring and piers, requires the use of special cranes or scaffolding. In VR, the zero gravity navigation feature enables easy access to any part of the bridge, regardless of accessibility in the real world.

Table 4. Critical comparison of Principle Inspection using VR.

Criteria	Principal Inspection using VR	Principal Inspection
Defect detection	Repeatable	Subject to fatigue
Time (disruption to user)	Low	Significant
Total cost per bridge	Cheap	Expensive
Accessibility to critical areas	Easy	Difficult
Consistency in findings	High	Low
Safety of inspectors	Excellent	Needs improving
Industrial implementation	Low technology readiness	High technology readiness

From the time and cost point of view, the fact that this bridge is a main railway network bridge, connecting Manchester with other cities, becomes significant. A thorough inspection of such a long bridge may take weeks due to surrounding obstacles such as buildings and slow movement of the inspector(s). In VR, any obstacles are removed at the post-processing stage, allowing faster and easier navigation. In the traditional inspection, the arch was subject to poor lighting conditions, due to shadows from surrounding buildings, making it difficult to detect defects. In VR, visibility is enhanced by artificial lightning. In traditional inspection, additional lightning can be deployed, but this would have a further impact on time, cost and logistics.

Interpretation of results is also improved as the findings can be shared with other bridge inspectors; even those who have never visited the real site. In traditional inspection, only a few engineers visit a site. Weather, lighting and other factors such as fatigue may affect their judgment on the day of inspection. The VR bridge can be conveniently revisited again and again. On the basis of the case study presented, it is demonstrated that the VR inspection technique offers several advantages over traditional inspection. Since the technique is newly proposed, industrial implementation is yet to be recognised and future advances in LIDAR, computing and VR will improve its effectiveness.

5. Discussion

The accuracy and reliability of documentation generated during the inspection of a bridge plays a key role in the allocation of resources for construction and maintenance. Whilst Principle Inspection is one of the most important inspection techniques, it currently relies purely on subjective assessments made by the inspectors. This could lead to an inaccurate assessment of the structure and colossal losses. It is the responsibility of the research community to discover novel and objective ways in which inspection could be improved.

The presence of cracks plays a vital role in the performance of a structure. For masonry bridges, cracks may occur at abutment joints, piers, arches, spandrel beams and arch keys. Specific details about the cracks, such as their position, orientation and length are of particular importance. Furthermore, differential movements in the bridge structure also need to be noted. These movements manifest themselves in the form of distortion of the overall shape, misalignment of bricks, tilting, bulging and ring separation. Other features such as open joints, degree of wetness, mortar loss, signs of leaching or chemical attack, eroded surfaces and presence of vegetation must also be noted (McKibbins, Melbourne, Sawar, & Gallard, 2006).

Once a scan has been obtained, these features could be detected and described automatically. For example, image processing techniques could be used to estimate the thickness of the mortar or detect cracks; a tedious job if done manually. In conventional Principal Inspection, cracks are sketched and measured manually. This methodology lacks objectivity and reliability in terms of quantitative analysis (Mohan & Poobal, 2017).

In the literature, there are many papers that describe various types of image processing technique that can be used to automatically detect and quantify crack parameters based on LIDAR images (Hutchinson & Chen, 2006; Tong, Guo, Ling, & Yin, 2011). Crack detection can be carried out using directional filters (Mohajeri & Manning 1991); analysis of variations in brightness (Miyamoto, Konno, & Bruhwiler, 2007) and texture analysis (Chanda et al., 2014). Features such as degree of wetness and leaching, which are dependent on colour variation, can also be quantified using MATLAB image processing algorithms. Other features of inspection such as geometric measurement of the bridge and differential movements are also recorded manually and lack accuracy in current inspection procedures. Automated methods are already developed and in practice to measure these parameters (Hong et al., 2014).

The limitations of conventional Principal Inspection which still remain a problem include: accessibility to critical areas; interpretation of results; time and safety or visibility issues when working on an irregular terrain or in dark areas. In some cases, the inspection process is also affected by weather conditions or traffic conditions if inspection takes couple of weeks (for landmark bridges). With the proposed VR technique, not only are all the features of inspection visible but they can become more prominent when using colour enhancing algorithms adopted during the post-processing stages. In the real world, lighting might be a concern if a bridge is inspected in a remote area where electricity is absent or power lines are not properly laid-out. This problem is solved using artificial lighting effects in the virtual world to ensure all areas become equally visible; as shown in Figure 10. Moreover, problems such as traffic disruption are also eliminated as these factors are controlled in the virtual world. The method improves the quality of inspection as shown in Figure 11, by allowing seamless navigation to all parts of the virtual world. Inspection also becomes less time consuming and travelling cost reduces as the inspector does not need to travel to the bridge. The data collection could be contracted out to a third-party service.

Figure 10. Artificial lighting in the VR environment keeps the weather always sunny. Defects become more prominent and saves lighting costs, especially in areas where inspection takes place in darker areas such as tunnels, under water bridges or bridges.

Figure 11. Undisrupted zero gravity navigation in the VR application makes movement flexible, fast and effortless.

Using the proposed VR technique, accessibility to critical areas and the safety of the inspector is ensured. The inspector can travel anywhere in the virtual world to closely inspect areas of interest, particularly joint connections or the arch ring as shown in Figure 12 without any safety concerns. Normally, these places are difficult to access and require special cranes to lift an engineer or a scaffolding arrangement. Engineers are not trained to work in such a manner and this can lead to accidents or life taking injuries if safety procedures are over looked.

Figure 12. Critical areas become accessible such as the arch ring, the most critical element of masonry bridges required for load transfer, without scaffolding or any crane lifting.

More importantly, the key findings from inspections carried out in VR are likely to be consistent and repeatable; overcoming one of the main problems of conventional Principal Inspection. Multiple engineers can perform the inspection of a bridge at the same time. Areas of interest can be viewed from different angles and discussed amongst engineers using VR multi-networking capability to accurately assign a condition rating based on a collaborative decision (see Figure 13). Moreover, data transmission via satellite also allows bridge data capture for the remote assessment of bridges located in difficult to access regions of the world; especially important in disaster relief efforts.

Figure 13. Interpretation of results becomes accurate as defects of interest can be viewed from multiple angles and can be discussed amongst inspectors using VR networking facility.

Despite of all the promising features, there are also some drawbacks associated with proposed inspection technique. Cheap VR hardware has limited memory size and processing speed. The image data sets need to be down sampled or compressed for smooth execution of the application. This limitation means that it is not currently possible to process large data sets and maintain good quality graphics simultaneously. In the future, smart algorithms or techniques such as machine learning could be used to address these limitations. Other hardware could be used, for example, the HTC Vive, a powerful VR device that has enough memory to process large data sets. However, this is a considerably more expensive option than the Samsung GEAR VR. 3D scanning devices are also expensive and require training to operate. Furthermore, they are currently limited to a minimum feature size of around 2–3 mm, which needs to be improved to be able to better capture some details. That said, the advance of technology is towards more powerful and cheaper hardware and software.

6. Conclusions

Technologically, the main contribution of this paper is the development of a VR application for bridge inspection using affordable devices running the android operating system such as smart phones. The application supports the import of a LIDAR scan of a bridge that can be used for inspection and as an accurate historical record of the state of a bridge at a moment in time. The application includes head tracking via the headset and navigation controls through a hand-held controller linked to the headset using bluetooth communication protocols.

The five key advantages to using VR over traditional inspection are as follows:

• Inspection becomes less time consuming with lower travel costs as the inspector does not need to travel to the bridge; data collection could be contracted out to a third-party.

• As bridges are very large, even in a VR environment, time for navigation and inspection is reduced by allowing the inspector to be teleported to locations of interest.

• The inspector can travel anywhere in the virtual world to inspect normally inaccessible areas of interest without the use of cranes or climbing gear, improving safety.

• Artificial lighting effects in the virtual world ensure all areas become equally visible. In the real world, lighting might be a concern where electricity is absent.

• The key findings from inspections carried out in VR are likely to be consistent and repeatable; overcoming one of the main problems of conventional Principal Inspection.

From a practical point of view, this research represents a step towards addressing the limitations and difficulties faced by engineers when undertaking Principal Inspection. For the first time, VR is being used as an inspection technique, unlocking new possibilities as well as raising the standard of bridge evaluation. Instead of the office going to the bridge, the bridge is coming to the office.

Acknowledgements

The authors would like to thank the School of Engineering at the University of Manchester for investing in the hardware and software required for this project.

Additional information

Funding

This work was funded by a University of Manchester School of Engineering research scholarship and EPSRC through grant number EP/N026136/1.