Requirements and challenges for infusion of SHM systems within Digital Twin platforms

Figures & data

Figure 1. Integration of needs for SHM systems embedded within a DT platform.

Figure 2. Organization of the review. Identification of sources of keywords.

Figure 3. Techniques for the generation of the virtual asset.

Figure 4. From available 2D documentation to an IFC-based BIM model.

Table 1. References for point cloud processing.

Cleansing	Registration	Segmentation	Identification
Rashidi and Brilakis (2016)	Rusu et al. (2009)	Wang et al. (2019)	Pu and Vosselman (2009)
	Theiler et al. (2014)	Chacón et al. (2021)	Perez-Perez et al. (2021)
			Pierdicca et al. (2020)
			Kim and Kim (2020)
			Kim and Kim 2021)

Figure 5. From available point clouds to an IFC-based BIM model.

Table 2. References for computer vision methods.

Structure from motion	Multi-view stereo	Visual Simultaneous localization and Mapping (VSLAM)	Featureless methods
Agüera-Vega et al. (2018)	Furukawa and Hernandez (2013)	Fuentes-Pacheco et al. (2015)	Hammoudi and Dornaika (2011)

Table 3. References for sensor-based SHM deployments.

Sensor	MEMS	Fiber optics	Accelerometers	Self-sensing and smart materials	Corrosion sensors	Acoustic sensors	Sensing sheets	Smart Phones
Description	Miniaturized mechanical and electro-mechanical elements. Two types of MEMS switch technology: capacitive and ohmic. Easy connection to a wireless sensor network	Light propagates in an optical fiber, characteristic parameters such as light intensity, phase, polarization state, or wavelength change.	Measurement of acceleration based on piezoelectric materials, in MEMS and fiber optics.	A self-sensing material exhibits a measurable property change in response to external stimuli.	Sensors based on different measuring parameters directly related to the corrosion of embedded reinforcement	The dynamic perturbation caused by sudden growth in defects releases elastic waves converted to electrical signals	Sensing sheets contain arrangements of thin-film sensors, electronics and circuits deposited and integrated on a flexible substrate	Smartphones can provide crowd-sensing data including embedded sensors as accelerometer, gyroscope, cameras, microphones, proximity sensors and GPS.
Data type and output	Raw or pre-processed Analog-Digital	Requires pre-processing provided by developers	Raw analog	Raw analog. Low TRL	Requires pre-processing provided by developers	Requires pre-processing provided by developers- Low signal to noise ratio	Requires pre-processing provided by developers	Sensitivity and noise of the sensors is highly dependent on the manufacturer
SHM applications	Monitoring of temperature, pressure, acceleration, inertial forces, chemical species, magnetic fields, radiation,	Temperature and strain (locally or distributed), sound (through the so-called Distributed Acoustic Sensing (DAS)	Detection of local and global damage. In bridges, a vehicle instrumented with accelerometers acts as a dynamic measurement device for monitoring and provides information on the structural response of the bridge.	Self-sensing concrete, Smart concrete to sense very small structural flaws. textile-reinforced concrete (TRC).	Optic sensors to measure pH and chloride concentration, increment of strain, temperature or change in the refractive index of the fibers due to corrosion process.	Piezoelectric and fiber optics (DAS) to measure elastic waves due to damage growth. Very suitable for fatigue damage detection	Can cover large areas of structures. The sensors that can be incorporated in the sheet are strain, pressure, temperature and humidity sensors. Two-dimensional (2D) quasi-distributed strain sensor	Can be used as reliable sensors to collect acceleration data and apply them into indirect health monitoring techniques to assess structural condition
References	Villacorta et al. (2021)	Barrias, Casas, and Villalba (2016), Bado and Casas (2021) Wu et al. (2018)	Malekjafarian et al. (2015) Mei et al. (2019).	Alonso and Puentes (2020) Alwis, Bremer, and Roth (2021).	Taheri (2019) Abbas et al. (2015) Zhou et al (2017	Wu et al. (2018). Olaszek et al. (2016) Tonelli et al. (2020)	Yao et al. (2014)	Mei et al. (2019), Morgenthal et al. (2018) Shirzad-Ghaleroudkhani et al. (2020) Chacón et al. (2018)

Table 4. References for CV-based SHM deployments.

CV-SHM	Local level	Global level
Fields	Crack detection in concrete, steel, pavement.	Structural response.
	Spalling, delamination, rust, bolt looseness.	Displacement monitoring
		Condition assessment
References	Hiasa et al. (2018).	Feng et al. (2015)
	Watase et al. (2015)	Chu et al. (2021).
	Ali et al. (2022)	Omar and Nehdi (2017)
	Shin et al. (2020)	Dong and Catbas (2021)
		Isailović et al. (2020)
		Abdeljaber et al. (2018)
		Azimi and Pekcan (2020)

Table 5. References for remote sensing techniques.

Sensor	TLS	GB-SAR	SB-SAR
Description	Identifies points in the 3D space using the travel distance of light pulses emitted at different sensor poses (the sensor rotates in the vertical and horizontal axes during measurements)	This technique is based on the same operational principles of the satellite SAR interferometry, but the synthetic aperture of the radar (SAR) is obtained by an antenna moving on a rail instead of a satellite moving around an orbit.	The feasibility of using space borne high resolution synthetic aperture radar (SAR) data to sense bridge deformations at periodic time intervals without the need to install any equipment has been studied recently
Data type and output	Point cloud	Raw or pre-processed analog-digital	Requires pre-processing provided by developers
SHM applications	Geometric inspection	Deformation and vibration assessments	Deformation assessment
References	Artese and Zinno (2020)	Huang et al. (2020)	Ponzo et al. (2021)
	Cha et al. (2019)	Xing et al. (2020)	D’Amico et al. (2020)
	Cabaleiro et al. (2016)	Pieraccini et al. (2019)	Qin et al. (2019)
	Mistretta et al. (2019)		Xiong et al (2021)
	Bueno et al. (2018).		Schlögl et al. (2021)

Table 6. Summary of Ashvin demo sites, digital files and sensing techniques.

Demo site	Country	Stage	Virtual Asset	Measurements
#1. Railways Bridges	Spain	Construction-Maintenance	IFC-based virtual asset developed from drawings	Sensors (strain gauges, accelerometers. Temperature, LVDTs) Cameras (Drones for photogrammetry) Remote sensors (GB-InSar) for tracking seasonal behavior.
#2. Building renovation	Poland	Design-Maintenance	BIM developed from point clouds	Sensors (air quality, thermocouples in walls)
#3. Airport runway	Croatia	Maintenance	CAD provided by owners	CV-based evolution of cracking in pavements
#4. Logistics hall	Germany	Construction	CAD provided by owners	CV-based analysis of construction processes
#5. Office Building	Sweden	Construction	BIM provided by construction company (partner)	Sensors (IMU, accelerometers, weather conditions) for tracking equipment. Remote sensors (TLS) for tracking construction evolution.
#6. Office Building	Spain	Construction	BIM provided by third party	Sensors (strain gauges, accelerometers. Temperature, LVDTs) Remote sensors (TLS) for tracking construction evolution.
#7. Road bridge	Spain	Maintenance	IFC-based developed from drawings	Sensors (strain gauges, accelerometers. Thermocouples) Remote sensors (TLS) for tracking seasonal behavior. Drones for traffic analysis
#8. Footbridges	Germany	Design	BIM models provided by design office (partner)	No measurements at design stage
#9. Stadium roof	Germany	Maintenance	Provided by third party	Remote sensors (TLS) for tracking seasonal behavior. Environmental sensors.
#10. Quay wall	The Netherlands	Construction	From drawings	Fiber Optics Extensometer: Smartec SOFO (fiber optic based)

Figure 6. A minimal pipeline from strain gauges to an IFC-based DT.

Figure 7. A minimal pipeline from satellite imagery to an IFC-based DT.

Rashidi, A., & Brilakis, I. (2016). Point cloud data cleaning and refining for 3d as-built modeling of built infrastructure. Construction Research Congress. May 31–June 2, 2016 | San Juan, Puerto Rico. doi:10.1061/9780784479827.093

 Google Scholar

Rusu, R. B., Blodow, N., & Beetz, M. (2009). Fast point feature histograms (FPFH) for 3D registration. In IEEE International Conference on Robotics and Automation, 3212–3217. doi:10.1109/ROBOT.2009.5152473

 Google Scholar

Wang, Y., Sun, Y., Liu, Z., Sarma, S. E., Bronstein, M. M., & Solomon, J. M. (2019). Dynamic graph CNN for learning on point clouds. ACM Transactions on Graphics, 38(5), 1–12. doi:10.1145/3326362

 Web of Science ®Google Scholar

Pu, S., & Vosselman, G. (2009). Knowledge based reconstruction of building models from terrestrial laser scanning data. ISPRS Journal of Photogrammetry and Remote Sensing, 64(6), 575–584. doi:10.1016/j.isprsjprs.2009.04.001

 Web of Science ®Google Scholar

Theiler, P. W., Wegner, J. D., & Schindler, K. (2014). Keypoint-based 4-points congruent sets–automated marker-less registration of laser scans. ISPRS Journal of Photogrammetry and Remote Sensing, 96, 149–163. doi:10.1016/j.isprsjprs.2014.06.015

 Web of Science ®Google Scholar

Chacón, R., Puig-Polo, C., & Real, E. (2021). TLS measurements of initial imperfections of steel frames for structural analysis within BIM-enabled platforms. Automation in Construction, 125, 103618. doi:10.1016/j.autcon.2021.103618

 Web of Science ®Google Scholar

Perez-Perez, Y., Golparvar-Fard, M., & El-Rayes, K. (2021). Scan2BIM-NET: Deep learning method for segmentation of point clouds for scan-to-BIM. Journal of Construction Engineering and Management, 147(9), 1–14. doi:10.1061/(ASCE)CO.1943-7862.0002132

 Web of Science ®Google Scholar

Pierdicca, R., Paolanti, M., Matrone, F., Martini, M., Morbidoni, C., Malinverni, E. S., … Lingua, A. (2020). Point cloud semantic segmentation using a deep learning framework for cultural heritage. Remote Sensing, 12(6), 1005. doi:10.3390/rs12061005

 Web of Science ®Google Scholar

Kim, H., & Kim, C. (2020). Deep-learning-based classification of point clouds for bridge inspection. Remote Sensing, 12(22), 3757. doi:10.3390/rs12223757

 Web of Science ®Google Scholar

Kim, H., & Kim, C. (2021). 3D as-built modeling from incomplete point clouds using connectivity relations. Automation in Construction, 130, 103855. doi:10.1016/j.autcon.2021.103855

 Web of Science ®Google Scholar

Agüera-Vega, F., Carvajal-Ramírez, F., Martínez-Carricondo, P., Sánchez-Hermosilla López, J., Mesas-Carrascosa, F. J., García-Ferrer, A., & Pérez-Porras, F. J. (2018). Reconstruction of extreme topography from UAV structure from motion photogrammetry. Measurement, 121, 127–138. doi:10.1016/j.measurement.2018.02.062

 Web of Science ®Google Scholar

Furukawa, Y., & Hernández, C. (2013). Multi-View Stereo: A Tutorial. Foundations and Trends in Computer Graphics and Vision, 9(1–2):1–148. doi:10.1561/0600000052

 Google Scholar

Fuentes-Pacheco, J., Ascencio, J., & Rendon-Mancha, J. (2015). Visual simultaneous localization and mapping: A survey. Artificial Intelligence Review, 43(1), 55–81. doi:10.1007/s10462-012-9365-8

 Web of Science ®Google Scholar

Hammoudi, K., & Dornaika, F. (2011). A featureless approach to 3D polyhedral building modeling from aerial images. Sensors (Basel, Switzerland), 11(1), 228–259. doi:10.3390/s110100228

 PubMed Web of Science ®Google Scholar

Villacorta, J., Del-Val, L., Martínez, R., Balmori, J. A., Magdaleno, A., López, G., … Basterra, L. A. (2021). Design and validation of a scalable, reconfigurable and low-cost structural health monitoring system. Sensors, 21(2), 648. doi:10.3390/s21020648

 PubMed Web of Science ®Google Scholar

Barrias, A., Casas, J. R., & Villalba, S. (2016). A review of distributed optical fiber sensors for civil engineering applications. Sensors, 16(5), 748. doi:10.3390/s16050748

 PubMed Web of Science ®Google Scholar

Bado, M., & Casas, J. R. (2021). A review of recent distributed optical fiber sensors applications for civil engineering structural health monitoring. Sensors, 21(5), 1818. doi:10.3390/s21051818

 PubMed Web of Science ®Google Scholar

Wu, Q., Okabe, Y., & Yu, F. (2018). Ultrasonic structural health monitoring using fiber bragg grating. Sensors, 18(10), 3395. doi:10.3390/s18103395

 PubMed Web of Science ®Google Scholar

Malekjafarian, A., McGetrick, P. J., & OBrien, E. J. (2015). A review of indirect bridge monitoring using passing vehicles. Shock and Vibration, 2015, 1–16. doi:10.1155/2015/286139

 Web of Science ®Google Scholar

Mei, Q., Gül, M., & Boay, M. (2019). Indirect health monitoring of bridges using Mel-frequency cepstral coefficients and principal component analysis. Mechanical Systems and Signal Processing, 119, 523–546. doi:10.1016/j.ymssp.2018.10.006

 Web of Science ®Google Scholar

Alonso, M. C., & Puentes, J. (2020). Self-compacted concrete with self-protection and self-sensing functionality for energy infrastructures. Materials, 13, 1106. doi:10.3390/ma13051106

 PubMed Web of Science ®Google Scholar

Alwis, L. S. M., Bremer, K., & Roth, B. (2021). Fiber optic sensors embedded in textile-reinforced concrete for smart structural health monitoring: A review. Sensors, 21(15), 4948. doi:10.3390/s21154948

 PubMed Web of Science ®Google Scholar

Taheri, S. (2019). A review on five key sensors for monitoring of concrete structures. Construction and Building Materials, 204, 492–509. doi:10.1016/j.conbuildmat.2019.01.172

 Web of Science ®Google Scholar

Abbas, Y., Have, B., Hoekstra, G., Douma, A., de Bruijn, D., Olthuis, W., & Van den Berg, A. (2015). Connecting to concrete: Wireless monitoring of chloride ions in concrete structures. Procedia Engineering, 120, 965–968. doi:10.1016/j.proeng.2015.08.827

 Google Scholar

Zhou, S., Sheng, W., Deng, F., Wu, X., & Fu, Z. (2017). A novel passive wireless sensing method for concrete chloride ion concentration monitoring. Sensors, 17(12), 2871. doi:10.3390/s17122871

 PubMed Web of Science ®Google Scholar

Olaszek, P., Casas, J. R., & Świt, G. (2016). On-site assessment of bridges supported by acoustic emission. Proceedings of the Institution of Civil Engineers - Bridge Engineering, 169(2), 81–92. doi:10.1680/jbren.15.00007

 Google Scholar

Tonelli, D., Luchetta, M., Rossi, F., Migliorino, P., & Zonta, D. (2020). Structural health monitoring based on acoustic emissions: Validation on a prestressed concrete bridge tested to failure. Sensors, 20(24), 7272. doi:10.3390/s20247272

 PubMed Web of Science ®Google Scholar

Yao, Y., Ellen, S. T., & Glisic, B. (2014). Crack detection and characterization techniques—An overview. Structural Control and Health Monitoring, 21(12), 1387–1413. doi:10.1002/stc.1655

 Web of Science ®Google Scholar

Morgenthal, G., Rau, S., Taraben, J., & Abbas, T. (2018). Determination of stay-cable forces using highly mobile vibration measurement devices. Journal of Bridge Engineering, 23(2), 04017136. doi:10.1061/(ASCE)BE.1943-5592.0001166

 Web of Science ®Google Scholar

Shirzad-Ghaleroudkhani, N., Mei, Q., & Gül, M. (2020). Frequency identification of bridges using smartphones on vehicles with variable features. Journal of Bridge Engineering, 25(7), 04020041. doi:10.1061/(ASCE)BE.1943-5592.0001565

 Web of Science ®Google Scholar

Chacón, R., Posada, H., Toledo, A., & Gouveia, M. (2018). Development of IoT applications in civil engineering classrooms using mobile devices. Computer Applications in Engineering Education, 26(5), 1769–1781. doi:10.1002/cae.21985

 Web of Science ®Google Scholar

Hiasa, S., Birgul, R., Matsumoto, M., & Catbas, N. (2018). Experimental and numerical studies for suitable infrared thermography implementation on concrete bridge decks. Measurement, 121, 144–159. doi:10.1016/j.measurement.2018.02.019

 Web of Science ®Google Scholar

Feng, D., Feng, M. Q., Ozer, E., & Fukuda, Y. (2015). A vision-based sensor for noncontact structural displacement measurement. Sensors (Basel, Switzerland), 15(7), 16557–16575. doi:10.3390/s150716557

 PubMed Web of Science ®Google Scholar

Watase, A., Birgul, R., Hiasa, S., Matsumoto, M., Mitani, K., & Catbas, F. N. (2015). Practical identification of favorable time windows for infrared thermography for concrete bridge evaluation. Construction and Building Materials, 101, 1016–1030. doi:10.1016/j.conbuildmat.2015.10.156

 Web of Science ®Google Scholar

Chu, C., Ghrib, F., & Cheng, S. (2021). Cable tension monitoring through feature-based video image processing. Journal of Civil Structural Health Monitoring, 11(1), 69–84. doi:10.1007/s13349-020-00438-9

 Web of Science ®Google Scholar

Ali, R., Chuah, J. H., Abu Talip, M., Mokhtar, N., & Shoaib, M. (2022). Structural crack detection using deep convolutional neural networks. Automation in Construction, 133, 103989. doi:10.1016/j.autcon.2021.103989

 Web of Science ®Google Scholar

Omar, T., & Nehdi, M. L. (2017). Remote sensing of concrete bridge decks using unmanned aerial vehicle infrared thermography. Automation in Construction, 83, 360–371. doi:10.1016/j.autcon.2017.06.024

 Web of Science ®Google Scholar

Shin, H. K., Ahn, Y. H., Lee, S. H., & Kim, H. Y. (2020). Automatic concrete damage recognition using multi-level attention convolutional neural network. Materials, 13(23), 5549. doi:10.3390/ma13235549

 PubMed Web of Science ®Google Scholar

Dong, C. Z., & Catbas, F. (2021). A review of computer vision–based structural health monitoring at local and global levels. Structural Health Monitoring, 20(2), 692–743. doi:10.1177/14759217209355

 Web of Science ®Google Scholar

Isailović, D., Stojanovic, V., Trapp, M., Richter, R., Hajdin, R., & Döllner, J. (2020). Bridge damage: Detection, IFC-based semantic enrichment and visualization. Automation in Construction, 112, 103088. doi:10.1016/j.autcon.2020.103088

 Web of Science ®Google Scholar

Abdeljaber, O., Avci, O., Kiranyaz, M. S., Boashash, B., Sodano, H., & Inman, D. J. (2018). 1-D CNNs for structural damage detection: Verification on a structural health monitoring benchmark data. Neurocomputing, 275, 1308–1317. doi:10.1016/j.neucom.2017.09.069

 Web of Science ®Google Scholar

Azimi, M., & Pekcan, G. (2020). Structural health monitoring using extremely compressed data through deep learning. Computer-Aided Civil and Infrastructure Engineering, 35(6), 597–614. doi:10.1111/mice.12517

 Web of Science ®Google Scholar

Artese, S., & Zinno, R. (2020). TLS for dynamic measurement of the elastic line of bridges. Applied Sciences, 10(3), 1182. doi:10.3390/app10031182

 Google Scholar

Huang, Q., Wang, Y., Luzi, G., Crosetto, M., Monserrat, O., Jiang, J., … Ding, Y. (2020). Ground-based radar interferometry for monitoring the dynamic performance of a multitrack steel truss high-speed railway bridge. Remote Sensing, 12(16), 2594. doi:10.3390/rs12162594

 Web of Science ®Google Scholar

Ponzo, F. C., Iacovino, C., Ditommaso, R., Bonano, M., Lanari, R., Soldovieri, F., … Rompato, M. (2021). Transport infrastructure SHM using integrated SAR data and on-site vibrational acquisitions:“Ponte Della Musica–Armando Trovajoli” case study. Applied Sciences, 11(14), 6504. doi:10.3390/app11146504

 Google Scholar

Cha, G., Park, S., & Oh, T. (2019). A terrestrial lidar-based detection of shape deformation for maintenance of bridge structures. Journal of Construction Engineering and Management, 145(12), 04019075. doi:10.1061/(ASCE)CO.1943-7862.0001701

 Web of Science ®Google Scholar

Xing, C., Wang, P., & Dong, W. (2020). Research on the bridge monitoring method of ground-based radar. Arabian Journal of Geosciences, 13(23), 1–10. doi:10.1007/s12517-020-06283-w

 Web of Science ®Google Scholar

D’Amico, F., Gagliardi, V., Bianchini Ciampoli, L., & Tosti, F. (2020). Integration of InSAR and GPR techniques for monitoring transition areas in railway bridges. NDTE and E International, 115, 102291. doi:10.1016/j.ndteint.2020.102291

 Web of Science ®Google Scholar

Cabaleiro, M., Riveiro, B., Arias, P., & Caamaño, J. C. (2016). Algorithm for the analysis of deformations and stresses due to torsion in a metal beam from LIDAR data. Structural Control and Health Monitoring, 23(7), 1032–1046. doi:10.1002/stc.1824

 Web of Science ®Google Scholar

Pieraccini, M., Miccinesi, L., Abdorazzagh Nejad, A., Nejad, N., & Fard, A. (2019). Experimental dynamic impact factor assessment of railway bridges through a radar interferometer. Remote Sensing, 11(19), 2207. doi:10.3390/rs11192207

 Web of Science ®Google Scholar

Qin, X., Ding, X., Liao, M., Zhang, L., & Wang, C. (2019). A bridge-tailored multi-temporal DInSAR approach for remote exploration of deformation characteristics and mechanisms of complexly structured bridges. ISPRS Journal of Photogrammetry and Remote Sensing, 156, 27–50. doi:10.1016/j.isprsjprs.2019.08.003

 Web of Science ®Google Scholar

Mistretta, F., Sanna, G., Stochino, F., & Vacca, G. (2019). Structure from motion point clouds for structural monitoring. Remote Sensing, 11(16), 1940. doi:10.3390/rs11161940

 Web of Science ®Google Scholar

Xiong, S., Wang, C., Qin, X., Zhang, B., & Li, Q. (2021). Time-series analysis on persistent scatter-interferometric synthetic aperture radar (PS-InSAR) derived displacements of the Hong Kong–Zhuhai–Macao Bridge (HZMB) from Sentinel-1A observations. Remote Sensing, 13(4), 546. doi:10.3390/rs13040546

 Web of Science ®Google Scholar

Bueno, M., Bosché, F., González-Jorge, H., Martínez-Sánchez, J., & Arias, P. (2018). 4-Plane congruent sets for automatic registration of as-is 3D point clouds with 3D BIM models. Automation in Construction, 89, 120–134. doi:10.1016/j.autcon.2018.01.014

 Web of Science ®Google Scholar

Schlögl, M., Widhalm, B., & Avian, M. (2021). Comprehensive time-series analysis of bridge deformation using differential satellite radar interferometry based on Sentinel-1. ISPRS Journal of Photogrammetry and Remote Sensing, 172, 132–146. doi:10.1016/j.isprsjprs.2020.12.001

 Web of Science ®Google Scholar