Picture This: A Deep Learning Model for Operational Real Estate Emissions

Abstract

We present a deep learning model estimating carbon dioxide equivalent (CO₂e) emissions in the real estate sector. The model, which utilizes convolutional neural networks (CNNs) and image classification techniques, is designed to estimate CO₂e emissions based on publicly available images of buildings and their corresponding emissions. Our findings show that the model has the ability to provide reasonably accurate estimations of CO₂e emissions using images as the sole input. Notably, incorporating primary energy sources as additional input further improves the accuracy up to 75%. The creation of such a model is particularly important in the fight against climate change, as it allows for transparency and fast identification of buildings, contributing significantly to CO₂e emissions in the building sector. Currently, information on emission intensity in the real estate sector is scarce, with only a few countries collecting and providing the required data. Our model can help reduce this gap and provide valuable insights into the carbon footprint of the real estate sector.

Keywords:

• Deep learning
• decarbonization
• CO2 footprint
• image classification

Introduction

The release of carbon dioxide (CO₂) from the building stock constitutes a significant portion of worldwide greenhouse gas (GHG) emissions (UNEP, 2022). In the European Union, buildings are accountable for approximately 40% of final energy consumption. (European Commission, 2019). The reduction of greenhouse gas emissions in real estate holds significant potential for mitigating global warming and achieving the imperative objective of keeping the global temperature increase below 2°C (IPCC, 2022). Notably, approximately 85% of the building stock in the European Union predates 2001 (European Commission, 2020), a period characterized by less strict energy performance requirements or more relaxed standards than those in place currently (Joint Research Centre (European Commission), 2019). Based on the report presented by the European Commission (European Commission, 2020), it becomes clear that the stipulated climate targets aiming at reducing energy consumption by a minimum of 60% shall remain unfulfilled despite the proposed weighted renovation quotas of 1 and 0.2% for deep renovations. There is an apparent discrepancy between actions required and actions taken. This discrepancy, also referred to as decarbonization gap, has increased in the EU in recent years with the only exception being 2021 due to the COVID-19 pandemic (Buildings Performance Institute Europe (BPIE) 2022).

While reducing the CO₂e emissions of the real estate sector is one of the pillars to limit global warming to below 2°C (Directorate-General for Energy, 2012; IPCC, 2022), complete nation-wide emission data on the asset level is scarce. To tackle this problem, the EU introduced the Energy Performance of Buildings Directive (EPBD) recast in 2010, making it mandatory to provide potential buyers and tenants of a property with an energy performance certificate (EPC) (European Parliament, 2010). After its introduction, EPCs turned into one of the most important sources of information on energy performance of buildings (Arcipowska et al., 2014). However, there is criticism on how EPCs are calculated (Coyne & Denny, 2021; Majcen et al., 2013; Pasichnyi et al., 2019) and also that EPCs are not publicly available in many countries (Arcipowska et al., 2014; United Nations Environment Programme, 2021). The European Union’s response to the aforementioned critique is encapsulated within the revised EPBD. Nevertheless, the timely development of relevant databases, serving as a dependable foundation for forthcoming sustainable real estate policies or renovation strategies, remains an open question. Further creation of EPCs is, at the moment, only mandatory in the EU when selling an object or letting it out to a new tenant. Even with the creation of publicly available EPC databases, the energy performance of owner-occupied real estate and long-term rented objects will remain intransparent at the current state.

To reduce this information barrier, a new approach requiring little resource to estimate GHG emissions of buildings has to be developed. As the determination of energy performance is based on many visual features such as the roof, windows, building age, walls, building size, and materials used (BRE, 2022), a deep learning model could possibly provide an approximation of the energy performance and CO₂e emissions. In other fields, deep learning has proven to be a possibility to produce cost effective and efficient tools, using only little data as input (Jenkins & Burton, 2008; Liermann et al., 2019). Nonetheless, despite its already existing utilization within the real estate sector, there is still a considerable scope for continued exploration and investigation into the potential applications of this subject matter (Koch et al., 2019).

In this paper, we provide a to our knowledge novel approach to estimating CO₂e emissions of residential buildings using a pre-trained Convolutional Neural Network (CNN) model. We train our model using EPC data from the United Kingdom and corresponding Google Street View (GSV) images. The remainder of the paper is structured as follows: In Section “Literature Background and Context,” we provide an overview over relevant literature for our study. In Section “Methodology,” we describe the methodology used to built our model and the database; in Section “Results,” we describe the results of our model, and, in Section “Discussion and Limitations,” discuss limitations and further research potential.

Literature Background and Context

With burgeoning computing capacity, Artificial Intelligence (AI) and especially Machine Learning (ML) systems, due to their potential to automate time consuming processes, have become increasingly popular for practical appliances such as computer vision, speech recognition, natural language processing, robot control, and others (Jordan & Mitchell, 2015). With rising interest, real estate is adapting the new methodologies in various ways.

Deep Learning, a subset of Machine Learning, particularly Convolutional Neural Networks (CNNs), has demonstrated significant utility in the realm of image analysis models and computer vision for real estate applications. Its profound efficacy lies in its ability to accurately forecast a diverse range of variables. Koch et al. (2019) provide a literature overview of the image analysis done in real estate research with CNNs. The authors classify the models into distinct categories, based on the input data employed, which comprise models predicated on aerial and satellite images, exterior front-view images, floor plans, and interior images. Due to the high accessibility of data, many papers utilize satellite images as their training and validation input. Earlier papers in this field on real estate related image analysis predominantly work on land-use segmentation or classification (Fröhlich et al., 2013; Marmanis et al., 2015; Yang & Newsam, 2010) and building footprint detection (Cohen et al., 2016; Ok, 2013; Raikar & Hanji, 2016). Accelerating progress in image classification through improvement of deep learning algorithms, especially the introduction of CNNs, and increasing strength of graphic processor units (Gu et al., 2018) has opened the possibility to take on more complex tasks and better results.

Jean et al. (2016), for example, predict the economic well-being of five different African countries using a CNN, achieving an accuracy of up to 75% explaining the variability of both asset wealth and consumption expenditure. The researchers construct their model by leveraging publicly accessible data, thereby endowing the model with the potential for scalability to different geographical regions. Through the utilization of satellite image data, the presented model allows for the extraction of valuable information, thus contributing to the amelioration of a data transparency gap. We share a similar motivation of extracting information from publicly available data with a CNN, to build a model that could reduce a data gap. Koch et al. (2021) present a CNN to predict where university graduates live in a city at a micro level, using different density classes. The presented model is able to derive the information using only satellite images after training. They find that their model has trouble differentiating neighbouring classes and thus allow a standard deviation of 1, to determine their overall model fit. The accuracy of the model increases by 37.8% when compared to true class prediction accuracy.

Exterior front-view images provide an alternative basis for image analysis in real estate, presenting features not included in satellite images. With the increase in their availability through web based services such as GSV or Apple’s Look Around, new areas of application have opened up (Koch et al., 2019). Paired with the increase in the efficiency of CNNs, several use-cases with high accuracy have been presented. Schmitz and Mayer (2016) use a CNN for the semantic interpretation of facade images. They achieve an accuracy of 85% differentiating facades, doors, windows and other building features. Furthermore, they conclude that even with relatively small training data sets CNNs can be trained sufficiently, if the available data is augmented and an already existing pre-trained model is fine-tuned. Obeso et al. (2017) train their model to distinguish street-view images of buildings in Mexico into four different architectural classes (pre-Hispanic, colonial, modern and other). The model is not only capable of identifying buildings correctly, but also of deriving distinct building features from street-view images linked to a particular architectural style. They suggest that CNNs are able to identify architectural characteristics of a building and assign it to the fitting class with relatively high accuracy (88.01%). Further applications utilizing exterior street-view images to classify buildings include the prediction of building age (Zeppelzauer et al., 2018) and building condition (Koch et al., 2018). Considering that both the age and condition of a building contain valuable insights into the emissions of CO₂e from that building, it is reasonable to contemplate the possibility of developing a comprehensive model that can extract pertinent building characteristics for emission prediction through the analysis of street-view images using a CNN.

Despotovic et al. (2019) build on the previous models and simultaneously predict the construction age and the heating demand, assuming that both can be predicted solely based on visual exterior features of a building. The presented model is able to predict both variables with similar precision resulting in an accuracy of 62% differentiating between five demand classes, proving that exterior images provide enough information to approximately determine their energy demand.

Utilizing CNNs in real estate image classification tasks, has been successful in the past, with all the presented models achieving accuracies significantly above the random probability threshold. Even complex matters, such as architectural style or heating demand, usually determined by professionals, can be approximated using CNNs, as the model are able to extract enough of the required information from the input images. The outcome of this literature review indicates that image analysis utilizing CNNs holds promise for mitigating the information gap associated with the estimation of CO₂e emissions attributed to the building stock.

Methodology

Over the past few years, the use of CNNs as a visual-based algorithm has become widely accepted as the standard in the scientific community (Shi et al., 2018; Wu et al., 2019; Zhou et al., 2018). CNNs are multilayer neural networks, taking their name from the convolutional layers which are part of their architecture (Albawi et al., 2017). The models are able to learn and recognize visual patterns from the input database, making them very effective for image classification tasks (Sarker, 2021). Various models have emerged since the advent of the AlexNet model in 2012, originating from different annotated datasets, which vary in architecture, primarily in the number of layers in a network (depth), the number of kernels per layer (width), and the dataset used (Krizhevsky et al., 2017; Wu et al., 2019). Investigations have shown that the depth of the model is particularly important for the success of neural networks, especially looking at accuracy or identifying complex features. However, training networks with a substantial depth is challenging, requiring high computational power and offering limited help after a certain stage (Srivastava et al., 2015). In order to achieve optimization, a variety of architectures have been developed that utilize identity shortcut connections between layers. This allows for improved information flow between different parts of the network, resulting in more efficient and accurate computations. This has led to the general concept of Residual Neural Networks (short: “ResNets”) (He et al., 2016; Targ et al., 2016), of which the approach implemented by He et al. (2016) constitutes the fundamental architecture of our model. In our baseline model, we employ a 34-layer ResNet model, which processes the image input from Google Street View.

The foundation for the training and validation data consists of two components, EPCs and GSV. While European energy certificates are subject to the EPBD, there are still differences between the individual member states (Semple & Jenkins, 2020). Thus, in order to obtain reliable information regarding the CO₂e emissions of buildings, an extensive and quality-controlled database is necessary. After conducting a thorough comparison of European energy performance certificate databases, it was determined that the Energy Performance of Buildings Register database of the Department for Levelling Up, Housing & Communities England & Wales is particularly suitable as a data source for the required emissions data. Particularly, UK EPCs incorporate three important factors: (1) the database is freely available and, therefore, particularly suitable for research purposes, (2) the CO₂e emission data does not require any conversion since the data is already displayed in the EPCs, and (3) EPCs are only issued by accredited assessors, which should ensure uniform quality. Additionally, there is already extensive literature on the used UK EPCs which can help to consider any potential information asymmetries and specific points in our research as well as identify weaknesses of the EPCs (e.g. as shown by Crawley et al. (2019); Fuerst et al. (2015); Taylor et al. (2019)).

Operational emissions (in CO₂e) of buildings are calculated based on two components. Firstly, the existing energy source, and secondly, the energy intensity of the building. The calculation is then derived by multiplying the conversion coefficient of specific energy sources with the energy intensity of the building. Consequently, in practice, it is possible for older, less modernized buildings, for example, to emit low CO₂e values due to their utilization of renewable energies, resulting in a low emission factor despite their high energy intensity. Thus, the trained model faces a high degree of complexity as the overall energy consumption of the house alone is insufficient to deduce the emissions. Instead, a detailed examination of externally visible characteristics, typical for each energy source, must take place. In this context, indicators such as existing chimneys or renewable energy systems can provide insight into the energy source. If none of these characteristics are externally visible, the model may estimate the energy source based on the observable condition or year of construction. The current Energy Efficiency Report (Office for National Statistics, 2022) suggests that the energy source correlates with the respective construction years, revealing trends specific to each period. Consequently, a subsequent heating system modernization may go unidentified by the model, potentially leading to inaccuracies.

The calculation and procedures for issuing EPCs for residential buildings in the United Kingdom are generally based on the Standard Assessment Procedure (SAP), which can be applied in a reduced form as Reduced Data SAP (RdSAP) for older buildings constructed before 2008 (BRE, 2022). The latest version of the SAP, the SAP 10.2, has been introduced in England in June 2022 and in Wales in November 2022. The SAP calculates (only) the operational CO₂e emissions based on the emission factors specified in Table 12 of the SAP annex, which are given in emissions of kg CO₂e per kWh. These emission factors are based on the energy source present in the building. In addition to the conventional Energy Efficiency Rating of A–G, this construction of the so-called Environmental Impact Rating represents a measure of notional CO₂e emissions. As is the case with the Energy Efficiency Rating: the higher the rating, the lower the expected emissions, with A equating to the lowest emissions and G to the highest emissions (Department for Business, Energy & Industrial Strategy, 2022). For the presentation in our training and validation split, the relative indication of CO₂e emissions per m² per year is used.

In December 2022, the England EPC-database held a total of 24,015,615 records. However, considering the ongoing process of EPC updates and their limited validity period, a portion of the available data is outdated. Therefore, it is unsuitable for the purpose of our dataset. If for example an address has two EPCs in the database, we eliminate the EPCs not depicted in the GSV image. This is accomplished by cross-referencing the date when the GSV image was captured with the date of issuance for the EPC. If the EPC is issued after the image was taken, we eliminate the data point. Further we filter addresses, which have no GSV image availability and are therefore unsuitable for our purpose. In order to maintain balance within our database, we restrict the number of data points for each class to match the lowest available data count among all classes. For a CNN model, a balanced and sizeable quantity of images in each class constitutes an indispensable prerequisite for achieving model precision (Luo et al., 2018). Predominantly, buildings of interest in the database are within the range of 0–140 CO₂e per m² per year. EPCs that exceed or fall below this range exist but are lacking in terms of the required frequency and timeliness. The classification of our CO₂e emissions categories is based on the established Environmental Impact Rating, which designates seven classes (Department for Business, Energy & Industrial Strategy, 2022). As depicted in Table 1, each classification corresponds to a unique range of 20 kg CO₂e per m² per year.

Table 1. CO₂e emission classifications.

Classification	1	2	3	4	5	6	7
CO2e/m^2/ year	0–20	>20–40	>40–60	>60–80	>80–100	>100–120	>120–140

After all expired, duplicate, and outdated EPCs, as well as EPCs for buildings outside the time periods available on Google Street View, were excluded from the database, a random sample of 3,000 images was drawn for each classification. Especially the availability of high emission class data after the filtering process limits us to this number. This results in a dataset of 21,000 EPCs distributed throughout England. The descriptive statistics for the entire dataset are shown in Table 2.

Table 2. Descriptive statistics of the EPC sample.

	CO2e/m^2/year	Energy efficiency score	Total floor area in m^2
Mean	69.51	53.43	84.74
Std	38.43	21.39	46.38
Min	0.40	1.00	5.00
Q25%	36.00	39.00	59.00
Q50%	67.00	54.00	78.00
Q75%	103.00	70.00	100.00
Max	139.00	112.00	1,115.00

N = 21,000.

The descriptive statistics demonstrate a balanced sample across the classifications. The Energy Efficiency Score, which indicates the calculated energy efficiency class, differs from our classification, as it focuses on energy intensity rather than CO₂e emissions (Department for Business, Energy & Industrial Strategy, 2022). The analysis of energy sources’ descriptive statistics (refer to Table A1 in the Appendix) demonstrates a notable prevalence of gas utilization in low-emission categories, where more than 90% of cases are attributed to gas heating in the initial three classes. In the higher classes, this observation shifts towards electricity as the primary energy source. As previously explained, CO₂e consumption of buildings is fundamentally calculated based on the primary energy source and energy intensity. In this context, a positive correlation coefficient between energy intensity and emission values can be assumed. Therefore, for the validity of the sample, it appears crucial to have sufficient variability between energy sources and energy intensity. Regarding the seven CO₂e classifications, a differentiated picture emerges (Figure A1 in the Appendix). For energy-efficient buildings, there appears to be a strong correlation between energy consumption and CO₂e emissions. However, this strong correlation gradually weakens, until in category 120–140 CO₂e/m²/p.a., there is only a correlation coefficient of approximately 0.3.

The Google Street View images were retrieved using the address information provided in the EPC database. The dataset comprises numerous images in which buildings are obscured by obstacles such as cars, vegetation, or other buildings, or are not visible at all. To train the model effectively and reduce the error rate, these images must be removed from the dataset. Examples of high and low information content images are provided in Figure 1. Images with high information content are characterized by their substantial explanatory value for the model, as they provide a comprehensive view of the building, thus serving as a solid foundation for feature extraction. Conversely, images where buildings are obscured by other objects or captured from unfavorable angles offer limited explanatory value for the model.

Figure 1. High information content (top) and low information content (bottom) images.

Following the manual filtration of images with low information content, a total of 17,922 images remained, resulting in the removal of approximately 14.66% of all images from the dataset. Across all classifications, the dataset remains balanced, as illustrated in Figure 2. Achieving balance between classes is of paramount importance for machine learning models, as imbalanced training data has the potential to significantly impair the performance of convolutional neural networks (Johnson & Khoshgoftaar, 2019).

Figure 2. Count of images in each classification.

As previously mentioned, we use the pretrained ResNet34 model developed by He et al. (2016) as our baseline model. ResNet34 is an image classification model with 34 weighted layers. The utilized model undergoes pretraining on the ImageNet dataset, where it initially learns from a large dataset for a different task. The resulting parameters from this pretraining are then fine-tuned and adjusted for our specific use case. Pretraining can significantly increase accuracy and is found to be particullary important for use cases with relatively small datasets (Li, Singh et al., 2019). To validate our results and identify the appropriate architecture and depth for our use case, we test the performance of several other models based on the ResNet architecture.

We have selected the following comparative models: a deeper and more complex ResNet101 (He et al., 2016), a NoisyStudent EfficientNet of size B3, which is a less deep and complex CNN that achieves higher efficiency (Tan & Le, 2019), a ResNext101 model that incorporates a cardinality building block into the ResNet architecture, reducing the number of hyperparameters required (Xie et al., 2017), and finally, a modified SE ResNet101 that assigns weights to different channels based on their importance for prediction (Hu et al., 2019). The chosen models either exhibit greater complexity compared to our baseline model (ResNet101, ResNext101, SE ResNet101) or are less complex (NoisyStudent EfficientNet). We employ this comparison to assess the suitability of our architecture for our specific use case. Deeper models can potentially offer improved performance depending on the task, but also carry the risk of overfitting the data (Ying, 2019). For our training the images are input as 3-channel RGB with a (resized) size of 224 × 224 pixels. The 80%/20% training and validation split is performed using a deterministic learner and a stratified k-fold split to ensure comparability of the splits across all model comparisons. In our baseline model, we use simple image augmentation techniques such as random flipping, rotation, zoom warp, and lighting transform. Additionally, we normalize the images. Hyperparameter tuning of the model is performed using the Adam optimization algorithm (Kingma & Ba, 2017) which optimizes the parameters during training. One of the most important hyperparameters is the learning rate, which determines the step size at which the model adjusts its internal parameters during the training process and varies based on the specific use case (Smith, 2017). We control the learning rate using the Smith (2017) two-stage transfer learning protocol in which the first half of the epochs updates the head of the pre-trained model with a learning rate of 1e-02 and then fine-tunes the entire model with the knowledge gained in the second half. In stage 2, we gradually increase the learning rate, then gradually decrease it, starting from 1e-5 and ending at 1e-4. This allows the neural network to learn more detailed information from new data and, thereby, improve performance. (Ng et al., 2015) To evaluate the results, we employ the conventional accuracy equation shown in Equation 1(1) $$\text{Accuracy}=\frac{\text{Number of correctly classified buildings}}{\text{Total number of buildings in the sample}}\times 100$$(1) , where the accuracy is defined as the ratio of the number of correctly classified buildings to the total number of buildings in the sample, multiplied by 100 and the F1-Score (Equation 2(2) $$\text{F1-Score}=2\times \frac{\text{Precision}\times \text{Recall}}{\text{Precision}+\text{Recall}}$$(2) ), the harmonic mean of precision and recall. The F1-Score is a popular metric when looking at models with unevenly distributed classification classes, as it accounts for the different class sizes. While our classification classes are evenly balanced we still provide F1-Scores for a better comparability with other models. Furthermore, Training and Validation Loss are compared between the models. (1) $$\text{Accuracy}=\frac{\text{Number of correctly classified buildings}}{\text{Total number of buildings in the sample}}\times 100$$(1) (2) $$\text{F1-Score}=2\times \frac{\text{Precision}\times \text{Recall}}{\text{Precision}+\text{Recall}}$$(2)

Augmentation and Energy Source

A possible measure to tackle overfitting of CNN models is (further) augmenting the input data or increasing the input sample size (Perez & Wang, 2017). We randomly choose three of the following methods for the augmentation for every photo of the input sample: image rotation (45°, 90°, 135°), image transposition (left to right, top to bottom), image resizing (divide or multiply height or width by factor 2), and image cropping (reduce size by factor 2). With the overarching objective of enhancing the transparency of the market, we also opted to add additional input to our training data.

As the calculated CO₂e emissions are significantly influenced by the primary energy source used (additional elaboration pertaining to the CO₂e calculation is provided in Section “Methodology”), we decided to incorporate an additional RGB-layer into the image matching process, allowing for the inclusion of both the CO₂e emissions and the primary energy source of the building. The necessary information regarding the primary energy source is readily obtainable from the corresponding EPCs. The same methodology utilized in the base model was applied to the new model, which includes heavy image augmentation and the incorporation of primary energy source data.

Grad-CAM Approach

In addition to the numerical metrics, we employ the Gradient-weighted Class Activation Mapping (Grad-CAM) method developed by Selvaraju et al. (2017) to achieve a visual validation of the model’s performance. This technique employs a heatmap overlaid on the image to highlight regions of particular importance to the model’s predictions. The Grad-CAM method leverages the model’s gradients, which represent the direction for adjusting the CNN weights during iterative training to minimize error. By utilizing these gradients, relevant activation regions within the image are identified, providing insights into the weighting of activations corresponding to specific CO₂e classes. This approach significantly improves interpretability and instills confidence. However, it is important to note that the evaluation of the model still relies on established performance metrics such as accuracy, F1-Score, training, and validation loss, which serve as the primary indicators of its overall performance.

Results

In the following, we first present the results of our baseline model. We then compare the results of the baseline model with alternative CNN models. Finally, we evaluate a modified version of the base model.

Baseline Model

We find that our baseline model achieves an accuracy of 43.38%, already outperforming random probability of 14.29% (Figure 3). In Figure 4 it is apparent that around epoch 34 the models starts overfitting the training data, causing the validation loss to increase, while the training loss continues to decline.

Figure 3. Baseline accuracy.

Figure 4. Baseline train/validation loss.

After this epoch, the algorithm detects characteristics in the training set that are helpful in predicting the training data but, when used in the validation phase, are not improving accuracy due to the inability to generalize the characteristics to the overall population (Mutasa et al., 2020).

Model Comparison

To test the assumption that deeper models might overfit and to benchmark our baseline model, we utilize four additional state-of-the-art models. As expected, the results suggest that the more complex models have a lower training loss whereas the validation loss increased, in some cases significantly. Our training set does not seem to be suited to train the deeper and more complex models encouraging overfitting (Table 3). While the computing time using the deeper models significantly increases, none of the tested models outperform our baseline model in terms of accuracy. We suspect that due to the relatively small sample size of the data set, the deeper models are not able to generalize the features learned in the training data (Zhang et al., 2021). The only model outperforming the baseline model in computing time is the less complex EfficientNet model. However, it is also achieving a significantly lower accuracy.

Table 3. Benchmarking the baseline model.

Model	Train loss	Validation loss	Accuracy	Time
ResNet34 (baseline)	0.8379	1.5500	0.4382	00:44
ResNet101	0.7255	1.6568	0.4201	01:50
NoisyStudent EfficientNet-B3	0.7720	1.8317	0.3429	00:26
ResNeXt101	0.5172	1.8328	0.4329	05:06
SE ResNet101	0.1930	2.3859	0.4245	05:50

Augmentation and Energy Source

The results of the image augmentation indicate an increase in accuracy of approximately 4 percentage points, with the new model achieving an accuracy of approximately 47.59% (Figure 5). Although the new model exhibits superior performance in comparison to the baseline model, the modest increase in accuracy is somewhat unexpected. Upon scrutinizing the dataset, it was observed that a mere 33.4% of the buildings featured a primary energy source other than gas. If we now assume that the energy source can also be partly determined by features of the house, such as appearance, roof, or chimneys it might become clear why the addition of the primary energy source only had little effect on the prediction results, while being a significant factor in influencing the CO₂e emission intensity. This fact contradicts our initial assumption that the model might only estimate energy intensity without drawing conclusions about the energy source or conversion coefficient. In this case, one would expect a higher increase in accuracy since, in practice, one of the two variables used to calculate CO₂e values is effectively included in the dataset, thereby greatly reducing the complexity of the model. It is important to note that the results do not provide a direct conclusion regarding whether the baseline version of the model derives the energy source directly (through building characteristics) or indirectly (via, for example, typical trends based on construction year). To test if the addition of the primary energy source altered the model comparison, we run a 50 layer (SE ResNet) to once again benchmark the results of the ResNet34 model. Similar to the previous outcome, the SE ResNet50 model is achieving a lower accuracy at 46.89%, requiring a computing time roughly 6 times higher than the ResNet model.

Grad-CAM Approach

Although it is apparent that our model significantly outperforms random probability, it remains unclear which specific visual features the model employs to approximate CO₂e emissions. To further validate our model, we use the Grad-CAM approach (Selvaraju et al., 2017). As shown in Figure 6, the model seems to identify the relevant buildings in every emission class. With the main focus on the property, important features in determining the CO₂e emission intensity such as the windows, the walls, and the roof among other components might be used for the estimation. Other parts of the image not related to the emission intensity like the sky, the street, or the neighboring buildings are identified as negligible by the model. Solely the front yard of some images seem to be helping the model with the prediction, which is not surprising as the look and size of the yards might help identify the building age of a property and, therefore, its energy efficiency (Aksoezen et al., 2015). In the overall context, however, the front yard does not appear to have any prevailing significance.

Figure 5. Accuracy of ResNet34 model with supplemented primary energy sources.

Figure 6. Random heat maps produced by Grad-CAM for different emission classes.

Accuracy

Given that our model is demonstrating superior performance compared to random probability and is visually validated through the Grad-CAM approach, it raises the question as to which factors are hindering its ability to accurately predict the emission class. Analyzing the confusion matrix in Figure 7 we conclude that the model is able to predict every class above random probability, with the lowest CO₂e emission class being predicted exceptionally well with a F1-Score of 80% (as seen in Table 4). With an accuracy rate of close to 50%, the model demonstrates reasonable precision across most of the other emission classes, 20–40 kg CO₂e/m² with 37% and 80–100 kg CO₂e/m² with 27% being the least accurate. Furthermore, the actual classes often seem to be confused with neighboring classes. The class with the lowest level of predictive accuracy, 80–100 CO₂e/m², is frequently misidentified with its two neighboring classes, namely 60–80 CO₂e/m² and 80–100 kg CO₂e/m².

Figure 7. Confusion matrix.

Table 4. Classification report.

Class	Precision	Recall	F1-Score	Support
CO2_00_20	0.81	0.79	0.80	518
CO2_100_120	0.40	0.39	0.40	499
CO2_120_140	0.46	0.49	0.47	489
CO2_20_40	0.44	0.32	0.37	518
CO2_40_60	0.44	0.61	0.51	525
CO2_60_80	0.42	0.49	0.45	523
CO2_80_100	0.33	0.23	0.27	513
Accuracy			0.48	3585
Macro average	0.47	0.47	0.47	3585
Weighted average	0.47	0.48	0.47	3585

A similar trend can be observed across all other emission classes, except for the lowest emission class, which was identified with high accuracy from the outset. The model appears to encounter challenges in correctly classifying images falling within the mid-range of the emission spectrum into the appropriate category. Given the fixed boundaries of the predetermined emission classes, it is possible for a house with an emission of, for instance, 41 kg CO₂e/m², to be erroneously classified as belonging to the lower class, owing to the emission value being in close proximity to the upper boundary of the lower class. We test this assumption by plotting a histogram of the deviation of the model over all classes (Figure 8). The histogram reveals that the majority of incorrect predictions are off by a single emission class, with only a minimal number of predictions diverging by more than one class. When accounting for this standard deviation of one emission class, the model demonstrates an accuracy of approximately 75.34%. This noteworthy improvement in accuracy highlights the model’s superior performance compared to random probability and underscores its exceptional ability to achieve accurate predictions when inaccuracies stemming from the fixed emission class boundaries are disregarded.

Figure 8. Deviation Histogram.

Discussion and Limitations

To the best of our knowledge, the discussed application of image classification regarding CO₂e emission in real estate represents a novel approach that yields promising results and may play a pivotal role in fostering global transparency in the ecology of real estate. While the simplicity and swiftness of the evaluated model are counterbalanced by certain limitations, the former predominantly prevails. Subsequently, we discuss the advantages and restrictions to our model, based on a practical application in a suburban street in Manchester, UK. Utilizing the EPC data of the houses and the corresponding Google Street View images, it becomes apparent that, as shown in Figure 9, there is significant heterogeneity among the emissions of individual residential buildings in the street.

Figure 9. Illustration of sample EPCs in Manchester, UK. Green = low CO₂e emissions; Orange = average CO₂e emissions; Red = high CO₂e emissions. Satellite image acquired from Google Maps.

If the primary objective is to achieve a maximum reduction in emissions, it appears that the buildings located in the middle and lower section of the street would benefit most from retrofits, as they are currently emitting high levels of CO₂e. Based on front view images, the presented CNN model could potentially identify and designate these areas in a city or on a street without the need for EPC data.

As shown in Figure 10, a large portion of the buildings are correctly classified by the model. In this example, the emission-intensive street section in the middle-lower area of the image is also correctly predicted. Although this example cannot be equated with an extensive test dataset, the results visualize the potential of the model. When sufficient computing power is available, the model can also be applied at the city or national level. At the same time, the weaknesses and limitations of the model are also apparent, requiring further research. Analyzing the buildings that were not correctly classified reveals the following implications:

Figure 10. Predictions based on the ResNet34 model [σ = 1]. Satellite image acquired from Google Maps.

The image quality can significantly affect the ability to identify the correct class. Upon examining the faulty classifications, images taken from unfavorable angles or obstructed by obstacles such as cars or greenery in the field of view appear to be a weakness (e.g., as shown in the incorrectly classified object in Figure 11). This result is consistent with Koch et al. (2019), who are suggesting that pre-processing may be necessary to reduce noise for further use. Automated procedures (e.g., through prior object detection based on annotated images) for quality assurance of input data might thus further increase accuracy. Furthermore, certain aspects remain unclear, such as the extent to which the influence of land characteristics not directly tied to the main structure, such as the front yard, impacts the accuracy of the assessments. Further research, potentially in conjunction with object detection techniques, could offer solutions to this matter. (Zhao et al., 2022)

Figure 11. Wrong angle image.

Google Street View data is not always up-to-date, limiting the possibilities for obtaining information on the current building stock at the macro level or for specifically identifying regular changes in the building stock, such as renovation rates. In the example from Figure 10, the images were taken in August 2021 and not updated until the time of publication. The periods of capture might also differ even within one city. Therefore, a comparison of the periods between the training and validation datasets is essential. Figure 12 provides an example of an image of a plot of land that was taken in 2008 but is now developed and responsible for emissions as a building.

Figure 12. Empty plot of land.

In addition to technical limitations due to data quality, the data foundation is subject to limitations arising from the interplay between the CO₂e values indicated in the energy certificate, the energy source, and the energy intensity. As previously explained, the CO₂e emission values correlate strongly and positively with energy intensity, particularly in the more emission-efficient classes. However, this correlation weakens in the less energy-efficient classes, while the F1-Scores remain relatively constant. Despite the overall high accuracy, it remains uncertain to what extent the model may be biased, potentially generalizing the energy source based on external construction quality, rather than identifying it based on specific details such as the presence of a chimney. Further research could explore the combination of two dedicated models. One model could be trained to identify energy intensity—as demonstrated, for example, by Despotovic et al. (2019)—while another model could identify the energy source based on various external details, using techniques such as object detection, to provide additional insights.

One additional factor that potentially limits the data foundation of the model is the quality and meaningfulness of the obtained Energy Performance Certificates. In scientific discourse, the meaningfulness of EPCs has been subject to intense discussion. At its core, the debate centers around the difference between consumption and demand certificates (also referred to as the “Performance Gap”) (among others, Coyne and Denny (2021); Cozza et al. (2020); Herrando et al. (2016)), the quality of data acquisition methods (among others, Hardy and Glew (2019); Hårsman et al. (2016); Li, Kubicki et al. (2019); Pasichnyi et al. (2019)), the overall meaningfulness with regard to retrofitting (among others, Christensen et al. (2014); Cozza et al. (2020); Gouveia and Palma (2019)), and potential price premiums (among others, Fuerst and McAllister (2011); Olaussen et al. (2017)).

In the context of our research, the Performance Gap and potential quality constraints in the data acquisition methodology are particularly relevant. Concerning the Performance Gap, it can be observed that actual consumption values are not incorporated into the model, leading to a divergence between the estimated and actual CO₂e emissions. However, incorporating consumption values would not be advantageous for the model, as individual consumption information is not contained in the front-view images, in contrast to the building’s physical properties. As Cozza et al. (2021) pointed out, in addition to imprecise input data and assumptions, individual occupant behavior and varying climate data are factors contributing to the Performance Gap. These individual characteristics of an inhabited property that are difficult to capture through a front-view image play a crucial role in the calculation of EPCs. However, they are not methodologically relevant to the CNN model. On the other hand, quality constraints in the input data and assumptions, similar to inaccurate or incorrectly oriented Google Street View images, can have a direct impact on the accuracy. In this context, quality assurance of the EPCs, as suggested by Hardy and Glew (2019) through machine learning, can further improve the quality of the model.

In conclusion, the input data, as previously elaborated, is one of the most important factors for a successful CNN model. Although EPCs are not perfect, they currently offer the only comprehensive means of analyzing the building’s energy-related properties and thus serve as a good foundation for feature extraction in ResNet models.

Concluding Remarks

We presented a novel approach for estimating CO₂e emissions for residential buildings. In our baseline model we achieve an accuracy of 43.38%, using a ResNet34 architecture. After including image augmentation and primary energy sources to the training data, the accuracy of the model increased to 47.59%. Disregarding the rigid borders of the emission classes and allowing a standard deviation of 1 for the prediction results, our accuracy significantly increased to 75.34%.

Evaluating the achieved results, the model can help to increase transparency regarding CO₂e emissions in the real estate market and, thereby, to contribute to the containment of global warming. Potential applications include the support of ESG due diligences, policy making, and identifying need for retrofit. However, there remains a scope of potential for improvement in the model. Especially looking at the database, an increase in quality and sample size of the input data could be beneficial for prediction accuracy. Moreover, the model could be extended by adding additional input information not restricted to visuals, as already demonstrated with the addition of primary energy sources. With increasing progress of computing power, more sophisticated models and images collected by autonomous vehicles, even a model tracking daily progress could be conceivable (Gebru et al., 2017). Owing to the model’s capability to enhance emission transparency, it could be correlated with other research domains in the real estate industry, such as price/rent prediction, which often overlooks emission data due to its unavailability. Overall, the presented model delivered promising results and might increase transparency of operational emissions in the residential housing market.

Appendix A

Figure A1. Scatter plots for energy consumption and CO₂e emissions.

Table A1. Main energy sources per classification.

CO2e/m^2/year	Electric	Gas	Other
0–20	0.02	0.97	0.02
>20–40	0.02	0.96	0.02
>40–60	0.02	0.97	0.01
>60–80	0.12	0.87	0.01
>80–100	0.42	0.55	0.03
>100–120	0.75	0.15	0.10
>120–140	0.85	0.08	0.07

Note: The shares are shown aggregated at the level of the primary energy source. Electric includes e.g. heat pumps and storage heaters, gas includes e.g. gas-fired central heating or gas floor heating, Other includes e.g. coal and oil heating. N = 21,000.