The Effectiveness of Big Data-Driven Predictive Policing: Systematic Review

Abstract

In this study, we aimed to investigate the effectiveness of big data-driven predictive policing, one of the latest forms of technologybased policing, and also the risks of data concentration on police forces or algorithmic bias. In order to properly weigh the benefits and risks, we first conducted a systematic review of the effectiveness of big data-driven predictive policing, based on the terminology of the research topic, and finally extracted 161 articles. We classified them into four categories based on the strength of the evidence and examined the evidence of effectiveness in each category. We concluded that while it is encouraging to see a range of studies, given the significant concerns about big data-driven predictive policing, evidence of effectiveness that can be used by policymakers still needs to be supported by more research, as only 6 out of 161 were evidencestrong in our study categorisation.

Keywords:

• predictive policing
• effectiveness
• systematic review
• big data
• hot spot
• place-based

1. Introduction

With the development of new technologies, a new type of policing has emerged: “predictive policing,” where police organisations seek to use data technologies to optimise the allocation of police resources to prevent crime. This has now begun to be coupled with so-called “big data,” as large datasets from multiple sources are used in predictive models in attempts to increase their power and efficacy. Even in the UK, for example, several police forces have been involved testing an analytics platform called NDAS (National Data Analytics Solution) to prioritise resources and find insight (including geospatial and network perspectives) against violent crime from multiple data sources like crime reports, people information across crime& intelligence, custody information (West Midlands Police, 2022) not to mention the controversial Predpol in the US. We are thus seeing the development of what can be termed “Big Data-Driven Predictive Policing (BDPP),” which aims to maximise the benefits of data-driven technologies for better policing. On the surface, such developments look promising, as they are based on a more accurate prediction of crime, which in turn will make police responses more effective and efficient. However, this turn in policing is facing challenges to its legitimacy due to the risks BDPP poses.

These risks can be divided into two categories: macro and micro. At the macro level, the emergence of big data in policing has raised concerns about the “surveillance society” and the security and transparency issues associated with the collection and retention of big data by police and other state actors (Brayne, 2017; McGuire, 2021; Shapiro, 2017; Zuboff, 2019). Recent studies point to the challenges democratic societies face in dealing with the expansion of law enforcement agency powers and capabilities, combined with data technologies and often increasing bureaucratic barriers to civil society oversight. In other words, at the macro-level there are concerns that the use of big data in law enforcement poses new challenges for the legitimacy of policing in democratic societies.

At the micro-, or practical level, there are some well-known problems with the algorithms that underpin predictive policing. If the data is mishandled during processing, or if it reflects the biases often present in current policing activities and practices, then algorithm-guided policing is also likely to be biased (Brantingham, 2017; Richardson et al., 2019). This can be seen as pointing to the inherent limitation that predictive policing does not always produce rational and legitimate outcomes, but is essentially just a way of reflecting existing data with the additional application of predictive technology. Despite the seemingly scientific, reliable and rational outcome that the term BDPP implies, there is always the possibility that using data generated in problematic ways and reflecting problematic policing—that, for example, over-represents ethnic and racial minorities in recorded crime statistics—will merely serve to justify and reinforce existing patterns of policing.

Furthermore, the trend towards algorithm-based policing can lead to distrust not only between police and the public, as described above, but also within police organisations. (Egbert & Krasmann, 2020; Ratcliffe et al., 2020; Sandhu & Fussey, 2021). More specifically, gaining the trust of police officers who want or who are asked to use predictive technology is also a challenge. This is not only because police cultures (Crank, 2014; Goldsmith, 1990) often resist the idea of giving up existing emphases on intuition and discretion, but also because police officers are insiders who know more about BDPP than the general public, and this makes it difficult to sell the effectiveness of BDPP to them if it is not thoroughly tested.

Why then is BDPP an emerging facet of police practice, despite the issues raised above? When we look at how governments and police who want to use this technology think and talk about it, effectiveness seems to be the most important concern. Among the democracies where the use of big data and predictive policing has been established, both the UK and the US have claimed effectiveness as a basis for the legitimacy of implementation in the law enforcement sector (UK House of Parliament, 2014; US Executive Office of the President, 2014; NYC, 2015; NPCC & APCC, 2016;). Various studies have explored the effectiveness of predictive policing (Carter et al., 2021; Kadar & Pletikosa, 2018; Mohler et al., 2015) and have found results favourable to these arguments (e.g., increased prediction rates, reduced crime rates). However, some studies have found mixed results depending on the crime or intervention methods (Hunt et al., 2014; Ratcliffe et al., 2021; Vomfell et al., 2018).

This brief review of the debate on BDPPs to date highlights both concerns and expectations. Three aspects of concern are, firstly, the development of BDPP as a potential threat to democracy in terms of the concentration of data and power in the hands of LEAs (if data analysis is misused), second, the potential for data-driven policing to be used as a tool for justification through data of unreasonable and inequitable realities, and third tensions internal to the police organisation using the new technology.

The expectation of BDPP is that it is more effective than conventional policing. At the current point in time, addressing the question of effectiveness appears vital if police organisations are to start using this technology at scale. If it is not effective, not only would there be no point in doing so, but use would seem likely to pose significant risks in terms of the concerns noted above—to, for example, the legitimacy of police.

This paper uses a systematic review methodology to examine the strength of evidence for the effectiveness of BDPP. Of course, even if the effectiveness of the BDPP is verified, this does not mean it use is unproblematic, or that it should be adopted, but merely that there is a basis for a debate about its potential widespread use if concerns, like those outlined above, can be addressed.

2. Background: BDPP and Claims of Its Effectiveness

2.1. The Terminology of BDPP

In order to define the scope of a systematic review, it is necessary to examine the definition of the research topic: in this case, BDPP. Considering the first part of the term, computer scientists have characterised big data as data with large “volume,” “velocity” and “variety” (Laney, 2001). When it comes to administrative and law enforcement perspectives, the adoption of this term varies depending on the context, but generally touches on the characteristics of data as “large and complex” (UK House of Parliament, 2014), “rapidly changing, large in size and breadth” (US Bureau of Census, 2021), or “vast, fast, disparate, and digital” (Brayne, 2017).

Turning to the second half of BDPP, one of the widely used, older, definitions of predictive policing is “Any policing strategy or tactic that develops and uses information and advanced analysis to inform forward-thinking crime prevention” (Uchida, 2009, p. 1). Recently, newly used definitions are “Application of analytical techniques-particularly quantitative techniques-to identify targets for police intervention and prevent crime or solve past crimes by making statistical predictions” (Perry, 2013, pp. 1-2) or “Harness[ing] the power of information, geospatial technologies and evidence-based intervention models to reduce crime and improve public safety” (NIJ, 2014). Brantingham (2017, p. 473) also referred to predictive policing as a three-parted process wherein “(1) data of one or more types are ingested; (2) algorithmic methods use ingested data to forecast the occurrence of crime in some domain of interest; (3) police use forecasts to inform strategic and tactical decisions in the field”. Definitions of predictive policing therefore include “source (information/data),” “analysis (advanced analysis/algorithmic solutions)” and “purpose (optimised allocation of police resources/better crime control).” This three-part conceptualisation echoes several previous attempts to define predictive policing, as Table 1 shows.

Table 1. Conceptualizing predictive policing.

Component	Content	In
Source	“big data”	Sandhu and Fussey (2021); Benbouzid (2019)
	“data from different sources”	Pearsall (2010)
	“power of information”	NIJ (2014)
	“a broad variety of sorts of data”	Meijer and Wessels (2019)
	“data of one or more type are ingested”	Brantingham (2017)
Analysis	“analytical techniques particularly quantitative techniques”	Perry (2013)
	“algorithmic technologies”	Sandhu and Fussey (2021)
	“advanced analytics”	Beck and McCue (2009)
	“mathematical algorithms”	Egbert and Krasmann (2020)
	“algorithmic methods”	Brantingham (2017)
Purpose	“identify likely targets for police intervention and prevent crime or solve past crimes”	Perry (2013)
	“generate near-future predictions about the people and places deemed likely to be involved in or experience crime”	Sandhu and Fussey (2021)
	“crime reduction; more efficient police agencies; and modern, innovative policing”	Beck and MacCue (2009)
	“anticipate, prevent and respond more effectively to future crime”	Pearsall (2010)
	“to reduce crime and improve public safety”	NIJ (2014)
	“that law enforcers act before criminal activities take place to prevent crime from happening”	Meijer and Wessels (2019)
	“inform strategic and tactical decisions in the field”	Brantingham (2017)

Reflecting on the above definition of big data and the various definitions of “sources,” it would seem that the commonly agreed characteristics of the “big data” in BDPP are “large quantities of data” and “multiple sources of data.” Prediction methods, the “analysis” component, can be summarised as “algorithmic analysis to predict” some outcome, event or situation. If we think about what algorithm means in general, it is the process between an input and an output. In the current context, it is the chain of computational processes that produces the output (prediction of a criminal event) from the input data. Examples of algorithmic applications to predictive policing are machine learning or deep learning techniques (Castro et al., 2020; Duan et al., 2017; Shukla et al., 2021). Moreover, AI for crime prediction can include automatic and semi-automatic models, with automatic models processing input data, analysing it and drawing conclusions without human intervention, while semi-automatic models are used for more complex and unpredictable tasks where humans and machines work together to analyse and make decisions. However, in light of recent advances in AI, we should be cautious about entrusting more human decisions to AI when we do not logically understand all of its decision-making principles, and at the same time it has been suggested that a major challenge is to establish an oversight network for AI that makes algorithms more transparent and allows for human intervention when necessary (UK Government Office for Science, 2023). In addition, there is a difference between static and dynamic data used as a source of AI, where static data is based on a fixed dataset and dynamic data is processed based on a constantly updated dataset. Applying this to crime prediction, it is of course reasonable to assume that the use of dynamic data will produce better results, but even if static data is used, if the dataset is updated at regular intervals (yearly or monthly), it can be considered dynamic in the long term.

When it comes to the third component, “purpose,” definitions vary because the strategic goals of police organisations adopting big data analytics can vary widely. However, one general purpose of police is to protect people from crime; another is to maintain the established order, and these purposes rarely change. Predictive policing aims to provide a more efficient and better way of achieving these overall goals. As such, its purpose could be construed as twofold. The first purpose simply refers to efforts to enforce and maintain the established social order and to “keep people safe from crime.” The second purpose is to find a more efficient way of achieving these goals within the policing framework (e.g., more accurate prediction of crime, allowing better allocation of police resources).

In summary, the concept of “BDPP” in this study can be said to be “policing behaviour or practice that is consistent with the strategic goal(s) of police organisations and that is based on, or initiated by, predictive algorithmic analysis of data from multiple, large datasets.”

2.2. Varieties of Big Data-Driven Predictive Policing

Uchida (2009) suggests different ways of using predictive policing, such as “hot spots,” “data mining,” “social network analysis,” etc. Perry (2013) also discusses different types of predictive policing (e.g., predicting place and time, predicting victims, etc.). Among these different types, a useful categorisation standard might focus on the target of the prediction. Here, two types of targets are widely discussed: “person-based (targeting a person’s future behaviour)” and “place-based (targeting the future occurrence of crime in a particular area)” (Degeling & Berendt, 2018; Ferguson, 2016; Fitzpatrick et al., 2019).

Person-based applications of predictive policing efforts tend to involve predicting the future behaviour of offenders, based on previous police records and offender characteristics. Specifically, person-based prediction as a method has been widely studied under the topics of “recidivism” (Cottle et al., 2001; Quinsey et al., 1995; Zeng et al., 2017) and so called “terrorist studies” (Gao et al., 2019; McKendrick, 2019, p. 11; Soliman & Abou-EI-Enien, 2019). In the case of place-based predictive policing, strategies with the intuitive title of “hot spot” policing have constituted an initial type of place-based predictive policing (Mohler et al., 2015; Kaufmann et al., 2019; Ratcliffe, 2015). There is an established consensus on the effectiveness of hot spot policing in reducing crime (Braga et al., 2012). Just as hot spot policing aims to effectively concentrate or deploy police forces according to analysis of the spatial and temporal distribution of crime, proponents of place-based predictive policing are keen to allocate police resources based on the more sophisticated predictive analysis that modern technology enables (Hunt et al., 2014; Ratcliffe et al., 2021).

When comparing person-based and place-based predictions, a number of important issues emerge. First, previous studies of person-based prediction have focused not only on the police, but also on the prison or probation phase of the offender’s journey through the criminal justice system. Person-based predictive “policing” is only a subset of this wider set of technologies and practices. By contrast, place-based types of prediction are mostly the work of the police or private security providers.

Second, as Ferguson (2016) noted, place-based predictive policing is often less “consequential” than person-based predictive policing. The latter leads the investigator directly to a particular set of individuals suggested by the algorithm (or is solely applied to such a group), and can be used to make life changing decisions about them (e.g., to focus deterrence efforts upon them, invoke a prevention order, etc.). Deploying patrol cars or other resources to a location, as is the case in most place-based approaches, has a much less certain, and in all likelihood less onerous, impact on individual people. As such, it is perhaps not surprising that place-based predictive policing is already being commercialised and implemented (as can be seen in PredPol etc.) and has a broader police-specific research base, with more studies on the subject than is the case with person-based predictive policing.

Third, and relatedly, person-based predictive policing is more ethically controversial. In the context of recidivism research, an exemplary case of person-based prediction, there are a number of studies (Dressel & Farid, 2018; Hart et al., 2007; Tonry, 2014) that have questioned its usefulness and legitimacy, raising concerns about whether it produces biased outcomes based, for example, on the social background of those targeted. Such concerns are likely to be exacerbated in policing contexts where those targeted are, by definition, innocent of the crimes they are “predicted” to commit in the future.

In short, place-based predictive strategies seem, ethically and for other reasons, more feasible in policing contexts. The consequences of negative or perverse outcomes are less severe (there may be biases or errors in the selection of places to patrol, but this is less “consequential” than monitoring an individual on the basis of some assumed future behaviour). Therefore, for the purposes of this review, the discussion will be limited to place-based predictive policing.

2.3. The Effectiveness of BDPP

Governments’ claims that the use of BDPP is rightful and legitimate - as found, for example, in policy documents - are often clearly based on two types of efficacy: the administrative efficacy of police resource allocation, followed by the expected effective control of crime through optimised resource allocation. For instance, in the UK, policymakers have aimed “to gain insights into criminal activity and to use resources more efficiently” (UK House of Parliament, 2014). Similarly, in the US, BDPP is claimed to help “allocate scarce resources more efficiently to prevent crime” (US Executive Office of the President, 2014). Experts in EU member countries also explicitly note the purpose of predictive policing in a similar vein. The operator of predictive policing in the Dutch police (Crime Anticipation System in Amsterdam) said that their programme aims to “intelligently allocate manpower where and when it matters most, using data mining methods” (Willems, 2014) in order to reduce crime. A member of the French gendarmerie also identified predictive policing as a way to “optimise the allocation of resources during some specific periods” (Perrot, 2017).

The obvious question is whether these claims of effectiveness are supported by evidence. To date, studies have examined the effectiveness of BDPP in two categories. The first is the development or validation of algorithm-based predictive models as the basis for effective police resource allocation (Catlett et al., 2019; Ellison et al., 2021; Ruiz & Sawant, 2019). The second is the examination of the effectiveness of these models when applied in the field (Meijer & Wessels, 2019; Mohler et al., 2015; Ratcliffe et al., 2021). To date, however, there has been no attempt to systematically review this literature, and it remains unclear whether BDPP can indeed be considered effective.

This study will systematically review the literature on the effectiveness of BDPP in a way that reflects the distinctions drawn by precedent research. Studies will be categorised into four types in this review, as shown in Table 2. The standard for categorisation is the type of evidence provided by a study. Firstly, type 1 and type 2 studies involve real-world applications. The application of predictive algorithms to “real life” crime fighting is highly complex due to the many variables (i.e., police culture, social context and the activities of other actors) that affect this activity. As such, “real-world” research should be treated differently from other types of studies. Within this, type 1 studies have tested for displacement effects, while type 2 studies have not. Much previous research (Bowers & Johnson, 2003; Brantingham & Brantingham, 2000; Reppetto, 1976) has been concerned with the potential unexpected or unintended outcome of simply shifting crime from target area A to another area B. Studies that consider potential displacement effects provide a stronger test of the effectiveness of the predictive policing programme.

Table 2. Categorized effectiveness verification.

	Tested in the real-world application	Tested displacement effect	Constructed direct prediction model
Type 1	O	O
Type 2		X
Type 3	X		O
Type 4			X

Types 3 and 4 in Table 2 refer to studies without real-world application. These tend to be retrospective studies that test the accuracy of prediction algorithms based on past crime data. Type 3 studies are retrospective studies that have tested their prediction algorithm to prove its accuracy. These studies present a specific prediction model with its own quantitative prediction results (e.g., the prediction accuracy rate). Meanwhile, type 4 studies, also retrospective, are indirect and do not involve direct application of the algorithm to past data in terms of place-based prediction. So they did not present the direct prediction model based on the algorithm. Instead, they simply specify possible predictor variables of crime or distinguished hot spots from cold spots by using multiple datasets: they use multiple datasets to produce background material that might be useful for crime prevention through predictive policing, but do reach the level of developing a full and independent prediction model. The reason why we include these retrospective studies in the category of evidence of effectiveness (albeit a relatively weak category) is that while the ultimate goal is to achieve outcomes such as crime reduction by applying actual crime prediction models, it is also a prerequisite for effective BDPP to ensure sufficient data analysis capabilities before applying them to policing.

3. Methods

3.1. The Objective of Systematic Review

The aim of this systematic review is to examine the effectiveness of BDPP strategies in the specific case of place-based predictive policing. Are they effective in reducing and preventing crime as claimed by governments and police forces, or are the predictive algorithms credible enough to be used as a first step in BDPP, if the new strategy has not been tested sufficiently to identify “real world” crime reduction effects?

Specifically, the research question “What do the existing evidence-based studies say about the effectiveness of BDPP” is systematically reviewed using the PICO format. PICO refers to Population, Intervention, Comparison and Outcome (Higgins et al., 2019) in medical research, where systematic review has been most widely applied. Population refers to the target of interest, and Intervention is a treatment that is being tested on the population. Comparison and outcome clarify the effect of the intervention between the treatment group and the non-treatment group.

However, there is some variation in what PICO stands for across different research fields (see Table 1 in Munn et al., 2018). Therefore, in this study we would adopt the revised PICO format as shown below. The main change is to move from “comparison” to “context.” This is not only because we found the variations that used “Context” according to the types of systematic reviews, but also for a practical reason. Comparison’ implies and even insists on Randomised Controlled Trial (RCT) studies (type 1 and 2 studies above), which are relatively rare in criminal justice and policing studies and, moreover, are not needed in studies that retrospectively assess the predictive accuracy of algorithms (type 3 and 4).

1. Population: Countries with the Big Data-driven policing strategy

2. Intervention: Predictive policing strategy using big data

3. Context: in the place-based policing context

4. Outcome: demonstration of effectiveness (crime reduction, the accuracy of prediction, etc.) in policing

3.2. The Database and Data Management

The following databases were searched: ProQuest Central, SCOPUS, Web of Science, College of Policing (UK), Global Policing Database (AU) and National Criminal Justice Reference Service (US). For the first three civil databases, full search terms were used as described below. However, for the later three government databases, simplified search terms were used as there were no hits for the full search terms. All materials were managed in Endnote X7.

3.3. Inclusion and Exclusion Criteria

The criteria for inclusion and exclusion are set out in Table 3. Firstly, policing strategies using big data are the “population” of the study. Given the concerns surrounding Big Data policing, the collection, linking and analysis of multi-sourced data by LEAs, and the infringements of private rights and excessive state power that this may indicate, this study will identify multi-sourced data in policing as an integral part of the problem. Material on single-source data-driven policing (such as predictive models based solely on crime data originally held by the police) will therefore be excluded.

Table 3. Inclusion and exclusion standard.

Criteria		Inclusion	Exclusion
PICO	Population	Policing strategy using big data (big data: multiple data sources including crime data)	Not related to the work of police Only relying on the conventional crime data of police
	Intervention	Predictive policing	Non-predictive policing
	Context	Place-based policing	Person-based policing
	Outcome	Crime reduction Rate of crime prediction (possibly) Efficient crime prevention(quantifiable)	Unquantifiable literature review in terms of efficiency
Technical		Quantitative studies in : academic journal articles Master and doctoral dissertations Books	Qualitative studies Non-academic writings in media (i.e., newspapers or journals)
		Grey Literatures : Policy reports are written by academics or researchers or law enforcement professionals. Governmental policy reports Conference papers and proceedings	Opinion pieces from the individual sphere (i.e., blogs or social media) Policy reports or opinions not written by academics or researchers or law enforcement professionals.
		Language of study: English	Languages other than English
		Studies within 15 years (17 May 2007 − 17 May 2022)	Studies before 2007
		Obtainable	Unobtainable in full-text and abstract

Second, the intervention criteria will reflect the issue of prediction. Thus, cases that rely only on conventional crime control strategies with non-predictive aspects will be excluded.

Thirdly, in terms of “context”, this study only covers predictive policing in a place-based context. Thus, “prediction” should be based on the algorithmic prediction of crime in specific geographical (physical) spaces. As such, prediction(s) of individual behaviour is not included. According to these exclusion criteria, most recidivism studies that deal with the prediction of reoffending are out of scope.

Finally, in terms of outcome, we start from the premise that central to the dialogue through which legitimacy is established and reproduced is the claim by police and governments that BDPP is “effective”: this is the “outcome” criterion, and the research will assess the veracity of this claim. Thus, given the multiple concepts of effectiveness, this review will only consider the forms of effectiveness claimed by governments and police forces, which are “crime reduction” or “accuracy of crime prediction” or other related quantifiable variables of effective crime prevention.

For the technical criteria, only quantitative studies that consider effectiveness against crime are included. Academic literature will be considered, including master’s and doctoral theses. Institutional policy reports and policy reports by eligible authors will also be considered as grey literature. The language of the study must be English, while the date should be within the last 15 years, from 2007 to 2022. The reason for setting the period from 2007 is that big data analysis is a relatively recently developed analytical method, and prior to this period, the various data sources that can be used for big data analysis were not sophisticated enough to be applied into the public sector, and the conditions for applying and analysing them for crime prediction were not mature enough. This is confirmed in Table 3 below.

3.4. Search Terms for Academic DBs

Based on the above discussion of the terms associated with the “BDPP” and PICO format, the following search terms were applied to databases that searched for abstracts, titles and keywords (essentially the entire record, excluding the full text). Specifically, each category of terms was applied to represent: the “Population” criteria of ① the characteristics of big data ② in the context of policing against, the “Intervention” criteria of the application of predictive methods, and the “Context” was a place-based perspective. The “Outcome” criteria were screened manually as it was difficult to find appropriate search terms to screen the quantitative and qualitative studies. For additional search terms, three widely discussed predictive policing methods (COMPSTAT, Hunchlab and Predpol) were used in cases where studies specifically using these algorithms were identified. Finally, the following terms were used.

• Terms related to big data: “big data” OR “data sources “OR dataset* OR “large data” OR “multiple data” OR “multiple sources” OR “high volume” OR “large volume” OR “high velocity”

  AND

• Terms related to policing: crim* OR secur* OR policing OR police OR “law enforcement” OR policed

  AND

• Terms related to predictive methods: algorithm* OR “AI” OR “artificial intelligence” OR forecast* OR predict* OR “machine learning” OR “super-learner” OR “super learner”

  AND

• Terms related to place-based context: place OR “hot spot” OR spot OR spatial* OR area OR geo*

  OR

• Terms related to the actual commercial programmes including “COMPSTAT” OR “Hunchlab” OR “Predpol”

3.5. Grey Literature Sources and Search Terms

Searches for policy documents (reports, government documents) and conference papers were included in the above databases to reduce the bias of including only published papers.

For additional grey literature, the following sources were searched: College of Policing (UK); Global Policing Database (AU); and National Criminal Justice Reference Service (US). The search strategy for each of the above databases is shown in Table 4. In brief, given the characteristics of these governmental DBs, simplified search terms were applied (applying the academic literature search terms above resulted in zero hits). In particular, as the first two sources (College of Policing and Global Policing Database) are DBs specifically for policing, terms related to policing were not necessarily required for the search. However, the last source (National Criminal Justice Reference Service) covers the wider criminal justice system and here a policing-related term (“predictive policing”) was used.

Table 4. Grey literature search.

Sources	Address	Methods
College of Policing	https://www.college. police.uk/	In the search bar, below search terms were applied respectively. “big data” “predict” “predictive”
Global Policing Database	https://gpd.uq.edu.au/s/gpd/ page/about	In the Browse bar(search full-text option), the below search terms were applied respectively. “big data” “predict” “Predictive”
the National Criminal Justice Reference Service	https://www.ojp.gov/ncjrs/ virtual-library/search	In General Search Bar, the below search terms were applied respectively. “big data” “predictive policing”

3.6. Citation Chaining Search (Backward and Forward Search)

Backward and forward searches were also conducted to mitigate the risk of missing relevant material in the keyword-based searches described above. Given the inconsistent terminology in place-based predictive policing (Kounadi et al., 2020), backward and forward searches were particularly important to mitigate the risk of erroneous exclusion due to terminological limitations. Typically, a backward search refers to a search of the references in a key article (in this study, the materials screened during the keyword-based search), and a forward search refers to the search strategy that identifies materials that cite the original article (again, in this study, the original articles will be the keyword-based search results).

The backward and forward searches were carried out in the same three databases (ProQuest Central, Web of Science and SCOPUS) used for the keyword searches, as they systematically provide reference indexes for backward searches and lists of cited articles for forward searches. Reference management software (Endnote) was used to list data exported from the databases. Duplicates were eliminated using Endnote X7. The PICO-based inclusion and exclusion criteria mentioned above were then applied for a more rigorous search.

4. Results

4.1. Basic Statistics

The first keyword search was carried out on 17 May 2022 in the three academic databases (ProQuest Central, SCOPUS and Web of Science). The three government databases were searched on 8 June 2022. Backward and forward searches were performed on 6 June for documents listed in the three academic databases and on 8 June for documents listed in the three government databases. Subsequently, the search was then repeated for the period between 17 May and 7 September 2022 as an update. Backward and forward searches were not performed for the newly listed documents in the updated search period of May 2022 and September 2022. In total, the search took about a month.

A total of 161 documents remained after screening. Specifically, the keyword search identified 69 documents using the 2007 to 17 May 2022 search period. In addition, 90 documents were identified by forward and backward searches on the 69 documents originally identified. Then, by updating the search period between 17 May and 7 September 2022, 2 additional studies were added using the same keywords. The screening process for each search strategy is shown in Figures 1 and 2. As many studies were identified through backward and forward searches, this indicates the use of inconsistent terminology in this area.

Figure 1. Keyword-based search.

Figure 2. Citation chaining-backward and forward search.

4.2. Search Result Statistics

Of the 161 documents, only 3.7% (n = 6) reported a study testing a predictive algorithm in a real-world policing application (type 1 and 2). All other studies (n = 155) were retrospective (type 3 and 4). These retrospective studies were mostly type 3 studies (n = 134), while type 4 studies (n = 21) were a relatively small proportion.

The number of selected papers per year gradually increased over the period covered by the search (the number of papers in 2022 was relatively small as the search period was from 17 May 2007 to 7 September 2022), as shown in Figure 3 below. First, there was a gradual increase in the number of papers searched. Even though the number of searched papers are the papers identified before applying the inclusion and exclusion criteria, it still says the growing interest in the technology-driven policies around us and might imply the trend of technology-driven policies and applications in various fields. The number of selected papers also increased during the search period. This is not surprising as the selected papers are also part of technology-based policies, but just focused papers on certain topics, as we have reflected in Table 3: Inclusion and exclusion criteria.

Figure 3. Number of studies searched and selected.

Of the final 161 papers, the top 5 countries covered were the USA (n = 101), China (n = 11), Canada (n = 9), the UK (n = 8) and Brazil (n = 5). However, as this review only considered research published in English, this result should be interpreted with caution.

4.3. Analysis of Type 1 and 2 Studies

Recall that of the 161 included studies, only 6 fall into type 1 (n = 3) and type 2 (n = 3). This review will look at all of these individually; a general summary of the 6 is given in Table 5 below.

Table 5. Type 1 and 2 studies.

Type	Authors (year)	Area	Data used for prediction	Target Crime	Police intervention	Result	Additional information
1	Braga and Bond (2008)	US	Emergency call, various qualitative data (local place characteristics, officers’ opinion etc.)	Various types of crimes	Situational interventions, Aggressive interventions (arrest, patrol, etc.), etc.	19.8 % reduction (in general, statistically significant than the control group)	Displacement was tested but no significant effect was detected
1	Carter et al. (2021)	US	Police crime data, drug overdoes data (ER data), crime cost estimation data	Social harm index	Vehicle patrol or foot patrol	The effect of the intervention was significant (p=.0295, social harm cost decreased $38.6 per 10.4 min of policing)	Displacement was tested but no significant effect was detected
1	Ratcliffe et al. (2021)	US	Crime data, demographic data, weather data, etc.	Property and violent crimes	Officer awareness, marked car patrol, unmarked car patrol.	31% reduction only on property crime when applying marked car patrol	No displacement was detected (diffusion of benefit was detected)
2	Wyatt and Alexander (2010)	US	Crime data (number of crimes), traffic accident data, driving under the influence data.	Traffic crashes	Vehicle stops (with visibility like blue lights)	Decreased number of fatal(15.9%) and injury(30.8%) accidents etc after the intervention.	There was no control group compared in the paper.
2	Florence et al. (2011)	UK	Crime data, demographic data, health service records	Violent crimes	Targeted deployment of police resources (presence, CCTV)	Effective in hospital admissions from violence(7->5 per 100k people in treatment while 5->8 in control), and recorded wounding(54 > 82 per 100k people in treatment while 54->114 in control)	There was a significant increase in less serious assaults than the control group
2	Hunt et al. (2014)	US	Crime data, disorder calls, seasonal variations, juvenile arrests	Property crime	Directed patrol etc.	No significant effect on reducing crimes than the control group	The research found that intervention was not properly made in the field.

Looking first at type 1 studies (implemented in the field and testing for displacement effects), Braga and Bond (2008) conducted a randomised controlled trial (RCT) with the Lowell Police Department in the US to test the effect of various tailored interventions on crime. The target areas (hot spots) were predicted by analysing police 911 call data and other qualitative data (site characteristics, local dynamics and officer perceptions). They used temporal analysis and ranking procedures based on the call data to identify the locations with highest citizen demand, and calibrated the boundaries of the hot spots with qualitative data. The 34 targeted hotspots represented 2.7% of the city. Half were randomly selected as the treatment group for problem-oriented policing (the officer in charge of the selected area was to analyse and implement tailored policing). Comparing the 6 months before and after the intervention, they found a 19.8% reduction in general crime calls. Additionally, there were some increase of calls in surrounding areas (displacement effect) but there were statistically not significant. Although their place-based prediction technique was not the same as in more recently published papers, Braga and Bond demonstrate the importance of tailoring the intervention to the target area and testing for displacement effects. Carter et al. (2021) targeted places with higher social harm characteristics (they called these social harm spots, derived from police crime data, drug overdose data and crime cost estimation data) and compared them with crime hot spots (mainly specified by violent crime data) to direct police patrols in Indianapolis (US). They found a significant effect in the social harm hotspots on violent crimes etc., with no displacement effects, but no significant effect in the crime hotspots. They also found no disproportionate effects (i.e., arrests) of the intervention on minorities. Ratcliffe et al. (2021) tested a predictive model in Philadelphia. The model used crime data, demographic data and census data (household income) to make predictions. There were two phases, one for property crime and one for violent crime. In each phase, three implementations (officer awareness, marked car patrol and unmarked car patrol) were compared in target areas by dividing 20 wards in equal proportions into 4 groups (3 implementation groups and 1 control group). Only in the case of property crime did the marked car patrol intervention have a significant effect (31% reduction), with no displacement effect - in fact, a diffusion of benefits was observed. This experiment did not find an effect of unmarked car patrols on property crime or of predictive policing on violent crime. In terms of violent crime, the researchers noted that the absolute lack of incidents made it difficult to find results, and descriptive crime statistics increased in the treatment group (24% increase during marked car patrols).

In terms of type 2 studies, Wyatt and Alexander (2010) implemented visible police enforcement as an intervention for targeted traffic crash site selected from crime, traffic crash and DUI data in Nashville (USA). They found a reduction in the number of fatalities and crash injuries after implementation. However, this study appears to be less robust than those above. For example, there was no information on the control group, displacement effects were not tested, and the period of the treatment effect was one year (i.e., each outcome was measured on an annualised basis). Florence et al. (2011) trial in Cardiff (UK) compared the city with control areas (i.e., other UK cities) to test a new strategy to prevent injuries from violence. They adjusted areas for the use of police resources (i.e., police presence, informing the use of CCTV) using police, demographic and health service (i.e., hospital) data and found a significant reduction in serious injuries. However, they found that there was an increase in less serious injuries in the treatment area. Finally, Hunt et al. (2014) examined the effectiveness of a predictive policing model in Shreveport (USA). They set up 3 treatment areas and 3 control areas, and implemented targeted car patrols etc. in the target areas, selected using different data sources, for 7 (intervention based on predictive algorithms) and control groups, although in one specific treatment case, where an officer was dedicated to this predictive policing project, there was a temporary reduction (35%) in property crime in the first four months of the experiment, but overall the effect was not statistically significant. (In the other treatment case, officers in patrol cars responded to calls for service in addition to the predictive policing project). The authors suggest that the intervention (allocation of police resources to target areas) failed because of organisational issues that prevented the correct dosage (for example, the police failed to maintain fidelity to participation in the experiment) or programme design.

In summary, of the six type 1 and 2 studies that applied multi-source prediction models to the real world, five showed some effectiveness. Even the one that showed no significant effect specified the reason (intervention failure) for this result, which gives some scope for future modification to be effective. However, this high proportion of results showing effectiveness could reflect publication bias. Furthermore, in most cases effectiveness is limited to some conditions, that is, particular combinations of crime type and police activity (e.g., property crime and marked cars in Ratcliffe et al., 2021). It is therefore difficult to generalise the results and claim that BDPP applied in real-life policing contexts reduces crime.

4.4. Analysis of Type 3 and 4 Studies

Type 3 (n = 134) and type 4 (n = 21) studies accounted for 96.2% (n = 155) of the studies identified. As these studies are retrospective, no police intervention was applied. These studies are reviewed in a simplified form - a summary is provided in the appendix. The results of this process are summarised in Figure 4. Of the 134 type 3 studies that tested novel algorithms on historical data to find an effective prediction model, 122 compared the effectiveness of their model (generally prediction accuracy) with baseline algorithms or other methods, while 12 developed a single model and reported the level of prediction accuracy. Of the 134 (n = 122 + 12) type 3 studies, 130 (n = 119 + 11) reported optimised best prediction result s(n = 119) or presented favourable results suggesting that their algorithm is effective in predicting crime (n = 11).

Figure 4. Type 3 and 4 studies.

Rather than list all 130 of these studies, we present an extract of some commonly observed characteristics of these studies to provide some context. The first set of studies (i.e., Chen, 2019; Park et al., 2016; Wheeler & Steenbeek, 2021) was the most common type among the 130 and involved comparing different candidate models to identify the machine learning technique or algorithm that can build the most optimal predictive model using the data obtained. These studies compared and analysed the superiority of their proposed methods for combining data and modelling. The second line of studies (e.g., Bogomolov et al., 2015; Bowen et al., 2018; Marchant et al., 2018) have either shown that predictive models using different data sources have better predictive accuracy than those using only criminal data, or that better predictive models can be built by using more data than before (e.g., 3 data sources instead of 2 data sources). The third type of research looks at how the same data (or variables) can be used in a predictive model, but in a more effective way to improve the accuracy of predictions. For example, Rummens et al. (2021) argued that including the number of people active at the time as a variable, rather than just the number of people living in the area, resulted in a better predictive model. In Fatehkia et al. (2019), they presented that prediction accuracy would be better if the adult population using Facebook was considered, rather than just the adult male population.

The final four studies were Elluri et al. (2019), Kostakos et al. (2019), Vomfell et al. (2018) and Wu et al. (2017), which are listed in the appendix as numbers 44, 73, 123 and 135 in order. Elluri et al. (2019) tested whether weather and time data, in addition to crime data model, would improve their model, but no significant difference was found. Kostakos et al. (2019) applied “hotel review data” to the crime rate prediction model, but the results were not significant. Vomfell et al. (2018) tested their model with crime, taxi flow, census and other data, but found no difference in violent crime from the baseline model (but a 19% increase in accuracy for property crime). Wu et al. (2017) also tested their predictive model with the addition of street view and satellite imagery, but also found no significant difference from baseline modelling.

While these results (large number of studies that made improvements or demonstrated accuracy versus small number of studies that admitted they didn’t make improvements or demonstrated accuracy was not significant) may seem to support the idea that predictive policing could help address crime problems, it almost certainly also represents a form of publication bias (as results with improved or good prediction rates are more likely to be submitted and published). The four studies that reported other types of outcomes all reach conclusions that seem to challenge the claimed strength of BDPP, which is that using more data resources improves prediction

In terms of type 4 studies (n = 21), these did not compare the robustness of their methods to other methods or baseline models, but only identified predictor variables (n = 18) or hot spots (n = 3) that can be used as predictive material for better preventative policing. These studies are significant in that they use different data sources to explore the background to crime prediction, but they did not go to the stage of building their own novel algorithm (which is probably the next stage after these), so they were classified as a separate type.

5. Discussion

5.1. Has the Effectiveness of BDPP Been Proven?

Answers to the question “What do existing evidence-based studies say about the effectiveness of BDPP” from this review appear contradictory. Of the 161 studies identified, only 6 were applied police interventions in the field. The majority, 96% (n = 155), were retrospective studies that tested models on past data. Of the six “real-world” studies, only three tested for displacement effects (type 1), which is important in determining the true effectiveness of place-based policing interventions. Moreover, even within the Type 1 studies, results varied according to the type of crime targeted and the type of intervention. Among the Type 2 studies (n = 3), one (Hunt et al., 2014) showed no significant effect after the intervention, while another (Wyatt & Alexander, 2010) did not have a control group, implying weak evidential credibility. In this sense, it can be concluded that the number of studies that provide evidence for the effectiveness of BDPP is extremely limited, and we cannot yet say that this type of intervention is effective.

However, of the six real-world application-based studies of BDPP, five showed some degree of effectiveness (Braga & Bond, 2008; Carter et al., 2021; Florence et al., 2011; Ratcliffe et al., 2021; Wyatt & Alexander, 2010). Among the retrospective studies (n = 155), which make up a large proportion of the total, most showed some efficacy or suggested an optimised prediction model. Of course, a high proportion of supportive results in retrospective studies is not in itself proof of effectiveness, precisely because they are “retrospective” and there is possible publication bias. However, testing predictive algorithms is worthwhile in itself. As Ratcliffe (2015) notes, building an accurate predictive model is one of the key parts of predictive policing. If building a good algorithmic model is the first part of creating effective BDPP (with initiating the right police intervention as the second part), then we may have some evidence of effectiveness.

Then, to get the result right, what perspective of interpretation should we focus on? In the view of this review, proportionality between risks and benefits should be the standard for interpreting the results.

5.2. Proportionality Between Risks and Benefits

From some perspectives, developments such as BDPP pose an almost existential threat to privacy rights and raise concerns about the potential “chilling effect” they could have on participation in the public sphere (BBW, 2018; LPEP, 2019). As Weber (1978) and many others since have argued, increases in the technical capacity of bureaucratic systems have a self-serving and self-replicating nature. The power that the police have is extremely difficult to take back once it has been given to them. Once police use of these forms of technology becomes commonplace, it will be difficult to roll back the additional surveillance and intervention capabilities they provide, whether or not they have a concrete effect on desired outcomes.

High-risk interventions should be carefully considered in terms of the benefits they are expected to bring. In the case of the BDPP, the expected benefit is effectiveness in tackling crime problems. However, based on the review of existing literature above, it is not yet known whether the potential benefits of this type of technology are sufficient to outweigh privacy concerns. In order to properly consider the legitimacy of implementing BDPP, we need further evidence that it is worth trying. Once this is provided, we might then be able to begin to discuss whether BDPP can be operated in a democratic way. Of course, such a discussion might reasonably conclude that even if some effectiveness can be demonstrated, this is not enough to offset the erosion of privacy brought about by the new technology. But without knowing whether basic criteria of effectiveness can be met, it is unclear on what basis government and police actors are entering the conversation in the first place (since justifications for the use of new technologies in policing almost always revolve around their alleged usefulness in “fighting crime”).

5.3. The Next Steps Required

The steps that need to be taken before BDPP can be actively used (if it is to be used) would therefore appear twofold. The first would be to prove, in terms of place-based policing, the effectiveness of using big data analysis (multi-source data) over crime data analysis alone. The second step is to test the model more rigorously in the field to ensure proper implementation.

With regard to the first step, if we accept that it is the multi-sourced nature of BDPP that is its fundamental risk point - i.e., that bringing together multiple sources of non-police data in risk prediction algorithms poses very significant privacy risks - then one aspect of the evidence base for the potential use of this technology should be tests of the superiority of the multi-sourced data-driven model over “simple” crime data driven models. As can be seen from Figures 1 and 2, the number of studies excluded from the full-text screening process because they do use multi-source data is over two hundred. This means that there are many studies that have applied crime data-driven algorithmic models to place-based crime prediction. In this review, we found a few studies (like Bowen et al., 2018; Castro et al., 2020; Marchant et al., 2018) that compared multi-sourced models with crime data-sourced models and showed higher predictive accuracy of the multi-sourced models. However, these studies are all retrospective and have not been implemented in the field. Therefore, it should be investigated whether the application of the multi-sourced prediction model is significantly different from the crime data-sourced prediction model when applied in the field.

When it comes to the second step, the need for evidentially stronger testing, this review identified a small number of studies that used multi-sourced prediction models in the field that gave useful insights. Taken together, it appears that a number of issues need to be addressed in order to test BDPP more rigorously in practice. These are ① proper sets of control and treatment groups as in the general protocol of randomised controlled trials, ② better-specified interventions (what kind of policing is to be applied and how), ③ more specificity in targeting crime types, ④ examination of crowding-out effects, and ⑤ tests of algorithmic bias (whether the algorithmic model was applied disproportionately to the particular group of people).

5.4. Limitations

This review is of course not free of limitations. The first concerns definitions of key terms: If we had defined big data as a specific size of data, for example, we could have included various studies on prediction using one data source (crime data), This, though, would have have required defining sheer size as objective measure of “big data,” which as we demonstrated is not a widely shared view.

Second, because this study reviewed papers to test the government’s claims about the effectiveness of the BDPP, it could have neglected the other important aspects (e.g., algorithmic fairness) of the BDPP. It is important to note that this review only scratches the surface of the various discourses on the subject of BDPP.

Third, for the reasons outlined above, this study only covered place-based BDPP. However, given that predictive techniques that target individuals or specific groups of people raise serious concerns for democratic policing, this aspect needs to be addressed in future studies.

Fourth, this study only reviewed papers written in English. Therefore, caution should be exercised in generalising the results.

Disclosure statement

No potential conflict of interest was reported by the authors.

Appendix.

Type 3 and 4 studies

Table

No	Author, Year	Target Crime	Region	Data Used (except crime data)
1	Ait El Bour et al., 2018	general crime	US	Local data(site information)
2	Al Boni, 2017	various crime types	US	Social media data
3	Al Boni & Gerber, 2017	various crime types	US	tweet data + etc.
4	Almanie et al., 2015	various types of crime(drug, disorder, theft and etc))	US	Demographics
5	Alves et al., 2018	homicide	Brazil	census data
6	Amiri, 2014	burglary	US	qualitative data
7	Amiruzzaman et al., 2021	crime rate	US	street view
8	Andresen, 2015	various crime types	CA	census data
9	Andresen & Hodgkinson, 2018	property crimes	CA	land use data + etc.
10	Araujo et al., 2019	crime rate	Brazil	urban features
11	Ariel et al., 2015	general crime call	UK	emergency call data (ambulance)
12	Baloian et al., 2017	burglary	Chile	facilities data
13	Bappee et al., 2021	crime rate	CA	streetlight data + etc.
14	Bappee et al., 2021	crime rate	CA	demographics + etc.
15	Belesiotis et al., 2018	various types	UK	street businesses + etc.
16	Bogomolov et al., 2014	crime rate	UK	mobile activity data + etc.
17	Bogomolov et al., 2015	crime rate	UK	weather + etc.
18	Borowik et al., 2018	various crime types	POL	open source
19	Bowen et al., 2018	violence	US	neighborhood census data + etc.
20	Brindha & Thillaikarasi, 2021	assault	US	census(socio-econ data)
21	Briz-Redon et al., 2022	Property / robbery / vandalism	Spain	socio-demographic data + etc.
22	Caplan et al., 2015	burglary	US	spatial factors
23	Caplan et al., 2020	Robbery	US	environmental features
24	Castro et al., 2020	crime rate	Brazil	crime data 2(unoffical) + etc.
25	Chase et al., 2019	none (police deployment and response)	Singapore	Demographics + etc.
26	Chen, 2019	burglary	US	land use data + etc.
27	Chen et al., 2015	Theft	US	Weather + etc.
28	Chen et al., 2016	traffic accident	JAP	mobility data
29	Cichosz, 2020	various types of crime	UK	street features
30	Connealy, 2019	violent crimes(assault robbery, homicide, rape)	US	environment data
31	Connealy, 2021	Street robbery	US	business data + etc.
32	Connealy & Piza, 2019	Robbery	US	business data
33	Corso, 2015	crime rate	US	tweet + etc.
34	Da Silva Neto et al., 2021	next movement of stolen vehicle	Brazil	traffic data
35	Daley et al., 2016	child abuse	US	local features + etc.
36	Dash et al., 2018	various crime types	US	library data + etc.
37	Delts, 2020	various crime types	US	image data + etc.
38	Deshmukh et al., 2020	robbery, theft, murder	India	crime data
39	Do Rego et al., 2020	violent crime	Brazil	regional features
40	Drawve et al., 2016	gun crimes	US	business data
41	Duan et al., 2017	crime rate		311 data(civil complaint)
42	Dugato et al., 2018	bulgrary	Italy	neighbourhood data
43	Ellison et al., 2021	policing demand	UK	census(socio-demographic) + etc.
44	Elluri et al., 2019	crime rate	US	weather + etc.
45	Fatehkia et al., 2019	burglary, assault, robbery	US	Facebook data + etc.
46	Fitterer et al., 2015	Break and entries	CA	urban environment data + etc.
47	Flores et al., 2015	crime risk(homicide)	not given	geodata of public services
48	Garnier et al., 2018	Robbery	US	land use data
49	Gezer, 2017	crime rate	US	education data + etc.
50	Gimenez et al., 2018	car crime accidents	Spain	environment data
51	Gerell, 2018	violent crimes(assault burglary)	Sweden	public environment data + etc.
52	Hajela et al., 2021	various types	US	census data + etc.
53	Han et al., 2020	theft	US	weather
54	He et al., 2020	larceny-theft	China	mobile phone data + etc.
55	Hibdon & Groff, 2014	illegal drug use	US	emergency medical call data
56	Hou et al., 2022	theft and assault	US
57	Hu et al., 2018	burglaries	China	demographics + etc.
58	Huang et al., 2015	various crime types	US	social network data + etc.
59	Huang et al., 2018	various types of crime	US	311 data(civil complaint) + etc.
60	Huang et al., 2019	traffic accident	US	urban complaint data + etc.
61	Hwang et al., 2017	burglary	KOR	social data + etc.
62	Iqbal et al., 2013	crime type	US	census data + etc.
63	Jendryke & McClure, 2021	hate crimes	US	demographic data + etc.
64	Johnson et al., 2021	property crimes	CA	ambient population data
65	Kadar & Pletikosa, 2018	serious crimes(robbery, burglary, assault and etc)	US	census data + etc.
66	Kadar et al., 2016	various crime types	US	demographic data + etc.
67	Kadar et al., 2015	burglary	Swiss	foursquare(local data)
68	Kang & Kang, 2017	crime rate	US	demographic data + etc.
69	Kennedy et al., 2011	gun violence	US	business data
70	Kennedy et al., 2016	assault	US	public housing data
71	Khairuddin et al., 2019	crime rate	US	census data + etc.
72	Kim et al., 2018	crime rate	CA	neighbourhood data
73	Kostakos et al., 2019	crime rate	UK	hotel review data
74	Kounadi et al., 2018	Robbery	Austria	twitter data + etc.
75	Kwon et al., 2021	Theft	KOR	environmental data(land use and etc)
76	Lamari et al., 2020	various crime types	US	socio-economic data + etc.
77	Li et al., 2019	property cirme and violent crime	US	socio-ecnomic + etc.
78	Li et al., 2021	post impact(clean up time and etc)	US	trafic data
79	Liang et al., 2022	Burglary, Robbery, Felony Assault, Grand Larceny	China	Demographics + etc.
80	Liang et al., 2022	various types of crime	China	urban features + etc.
81	Liesenfeld et al., 2017	various types of crime	US	socio-economic data
82	Liu, 2017	violent crime	US	weather + etc.
83	Liu et al., 2021	traffic accident	US	traffic flow data
84	Marchant et al., 2018	violent crimes(domestic violence, burglary, vehicle theft)	AU	demographics
85	Matijosaitiene et al., 2018	Larceny / Harassment / Assault	US	land use geographic data
86	McClendon & Meghanathan et al., 2015	various crime types	US	census data
87	Meng et al., 2018	car crash accident	China	weather data
88	Moosavi et al., 2019	traffic accident	US	weather + etc.
89	Nguyen et al., 2017	various types(street, other, motor vehicle theft, burglary)	US	demographic data
90	Ohyama & Amemiya, 2018	Theft from Vehicle	JAP	facility data
91	Parent et al., 2020	crime rate	CA	socio-economic + etc.
92	Park et al., 2016	traffic accident	KOR	accident data
93	Peng & Kurland, 2014	burglaries	China	demographics + etc.
94	Piña-García & Ramírez-Ramírez, 2019	various types of crime	Mexico	tweet data + etc.
95	Piza & Carter, 2018	Burglary & Motor Vehicle Theft	US	geospatial feature data + etc.
96	Qian et al., 2020	crime rate	US	311 data(civil complaint) + etc.
97	Rahakrishnan & Dervarasan, 2016	crime rate	US	census data
98	Rastogi et al., 2020	crime against women	India	census data
99	Reed, 2015	identity theft	US	demographic data
100	Reinhart, 2016	various crime types	US	socio-economic + etc.
101	Ristea et al., 2020	property crimes	US	tweets + etc.
102	Roses, 2020	various crime types	US	street segment + etc.
103	Roses et al., 2021	crime rate	US	census + etc.
104	Ruiz & Sawant, 2019	various crime tpyes	US	business data + etc.
105	Rumi & Salim, 2020	crime rate	US	check-in data
106	Rumi et al., 2018	theft	US & AU	foursquare(local data) + etc.
107	Rummens & Hardyns, 2020	burglary	Belgium	census data + etc.
108	Rummens & Hardyns, 2021	burglary	Belgium	demographics + etc.
109	Rummens et al., 2017	burglary, robbery, battery	Netherlands	demographics + etc.
110	Rummens et al., 2021	various types of crime	Belgium	mobile data(to find ambient population)
111	Salama et al., 2021	various types of crime	US	weather + etc.
112	Schaffter, 2020	violent crime	US	census data
113	Shukla et al., 2021	crime rate	US	econ variables (like employees) + etc.
114	Solomon et al., 2022	burglary	US. Israel	weather + etc.
115	Sujatha & Ezhilmaran, 2014	general crime rate	US	LEA data + etc.
116	Sujatha & Ezhilmaran, 2016	crime rate	US	census data + etc.
117	Sun et al., 2021	crime rate	US	census data
118	Tang et al., 2019	crime rate	China	building boundary + etc.
119	Tarlekar et al., 2021	crime rate	India	media data + etc.
120	Tavares & Costa, 2021	property crimè	Portugal	demographics + etc.
121	Tian & Zhang, 2021	traffic accident	China	traffic flow data
122	Vinnia & Felix, 2019	crime rate	US	zoning data
123	Vomfell et al., 2018	violent / property	US	taxi flow data + etc.
124	Wang & Brown, 2012	breaking and entering	US	geograhic(highways, businesses, schools and etc) + etc.
125	Wang et al., 2019	property / violent	CA	census data
126	Wang et al., 2017	crime rate	US	Weather
127	Wang et al., 2017	various crime types	US	venue data
128	Wang et al., 2020	various types of crime	US	census data + etc.
129	Wang et al., 2020	crime risk(homicide occurance)	China	demographics + etc.
130	Wang et al., 2013	burglary / robbery / larceny	US	demographics + etc.
131	Wang et al., 2012	personal and property crimes	US	census data
132	Wawrzyniak et al., 2018	various crime types	Poland	meteological data + etc.
133	Wei et al., 2020	various types of crime	US	demographic data + etc.
134	Wheeler & Steenbeek, 2021	Robbery	US	census data
135	Wu et al., 2017	crime rate	US	street view + etc.
136	Xia et al., 2021	drug crime	US	socio-ecnomic data + etc.
137	Yang et al., 2018	varuiys types	US	twitter data + etc.
138	Yang et al., 2020	street crimes	US	night light image based urban transitional zones
139	Yao et al., 2020	crime rate	US	demographic data + etc.
140	Ye et al., 2021	theft	US	311 call data
141	Yi et al., 2018	crime rate	US	census data + etc.
142	Yoo & Wheeler, 2019	homeless related crime	US	environmental features
143	Yuan et al., 2018	traffic accident	US	high-resolution rainfall data + etc.
144	Zhang et al., 2019	theft and assault	US	tweet data
145	Zhang et al., 2020a	various types of crime(person offense, property, theft and etc)	US	GSV(google street view) + etc.
146	Zhang et al., 2020b	theft and assault	US	twitter data
147	Zhang et al., 2021	property crime	China	weather data
148	Zhang et al., 2022	theft	China	skynet camera data + etc.
149	Zhao & Tang, 2017	crime rate	US	meteorological data + etc.
150	Zheng et al., 2023	various crime types	US	tweeter data
151	Zhou et al., 2020	crime rate	US	demographic data + etc.
152	Zhou et al., 2021	crime risk	US	weather + etc.
153	Zhou et al., 2022	road crime	US	road network + etc.
154	Zhu, 2018	burglary	US	social data
155	Zuniga-Garcia et al., 2022	Pedestrian crash	US	road inventory + etc.