Text mining and machine learning for crime classification: using unstructured narrative court documents in police academic

Abstract

This paper proposes a novel approach to utilizing open-source legal databases in academic education, especially in the fields of law and police investigations. Our framework provides a way to organize and analyze this data and extract reports that are associated with crime scenes, addressing the challenge of classifying unstructured legal documents by using text mining, natural language processing, and machine learning techniques. We developed a supervised machine learning model capable of accurately classifying court documents based on two classifiers: one identifies the documents containing crime scenes, and the other classifies them into five types of crimes. The experimental results were promising, as the random forest algorithm achieved an accuracy of 91.07% for the first classifier and support vector machines achieved an accuracy of 82.46% for the second classifier. What distinguishes our work is the creation of a crime dictionary that includes 70 crime tools and 151 related terms extracted from various forensic sources. It is considered relatively small, but it contributed to giving good classification results. The proposed crime dictionary can be generalized, developed, used in advanced searches, and integrated with police databases to improve crime scene analysis. Overall, the research highlights the use of court databases in police academic education and attempts to utilize them in a more effective manner.

Keywords:

• Crime scene
• crime classification
• text mining
• machine learning
• unstructured data
• the CAP dataset
• legal documents

REVIEWING EDITOR:

• Jenhui Chen, Chang Gung University, Taiwan

SUBJECTS:

• Data Preparation & Mining
• Artificial Intelligence
• Computer Science (General)
• Machine Learning
• Education & Training
• Educational Technology

1. Introduction

Police officers often spend a substantial amount of their time writing reports, which are required for recording different situations. These reports, often known as police reports, comprise all relevant paperwork and narratives that document the events, persons involved, and officers’ activities. Accuracy in reporting a broad variety of scenarios, from small occurrences to major crimes, is critical. These reports act as a replacement for officers in court, assisting the judicial system in understanding their findings. Despite the importance of report writing, many police schools provide little instruction in this area. While police are trained in many parts of law enforcement, report writing is sometimes overlooked and should be included in a more comprehensive curriculum that covers all aspects of the criminal justice system (Yu & Monas, 2020).

Studies have shown that there is a link between reading and writing abilities, since reading is an effective source of writing that helps pupils overcome language and topic knowledge issues. Reading may help to create writing models by composing and organizing phrases, paragraphs, and essays. It also improves critical thinking skills by requiring students to evaluate and analyze the content they encounter, which is necessary for writing compelling and logical arguments (Bai & Wang, 2023; Graham & Hebert, 2010; Shanahan & Shanahan, 2008). According to these studies, reading actual reports can help forensics and law enforcement students improve their writing skills and gain practical knowledge, such as crime scene analysis, hypothesis development, legal terminology, and identifying court-worthy points of interest. Academics can also use these cases as teaching tools, presenting them to students as case studies and leading in-depth discussions about various aspects of the cases, such as flaws in the investigations, improper evidence presentation, and the possibility of overturning the ruling. However, the public is often unable to access police reports due to legal restrictions on data use agreements, concerns about victims’ privacy information, and the need for permission from the relevant authorities. As a result, despite the educational value of these reports, their distribution is a challenge. To address this issue, we reviewed existing research (HeinOnline, 2022; Lage-Freitas et al., 2022; Novotná & Harašta, 2019; The Caselaw Access Project (CAP), 2022; Zhang, 2021) and discovered projects from various countries that make legal and court documents available online as a ‘case law dataset’. However, they have the disadvantage of including reports that are not appropriate for our work, whether they are brief or cover topics unrelated to our field.

The main goal of this study is to prepare the criminal documents found in these databases by processing and classifying them using text mining and Machine Learning (ML) techniques and building an educational tool. This tool contains a large group of criminal cases that provide details about the crime and help investigators, whether professionals or trainees, access criminal documents easily. Homicide cases were selected as a case study for our research because they are one of the most popular criminal cases among students due to the many factors and hypotheses requiring logical thinking and analysis. In addition, the Caselaw Access Project (CAP) (2022), a database made available by the Harvard Law School Collection, served as the adopted database for our experiments. The contributions are as follows:

Propose an educational tool for real-life cases using court reports to improve education at police academies. This is the first study of its kind that suggests the utilization of court reports in this field.Build a crime dictionary from different external resources in criminal field.Design a two-phase ML model with a small number of features from the proposed crime dictionary to classify court documents based on homicide crime types.

The rest of the paper is organized as follows: Section 2 presents the related works, and Section 3 describes the materials and method of the proposed framework. In Section 4, the experimental results of the model are illustrated by tables, statistics, and graphs with discussion. Finally, Section 5 concludes the paper and offers suggestions for future investigation.

2. Related works

Crime classification is a complex and multifaceted field that involves categorizing criminal activities using various criteria. There are various systems for classifying crimes, which often differ by jurisdiction and legal framework. Studies on the classification of crimes using crime tools will be reviewed, as we searched for the following keywords: ‘crime classification’, ‘narrative reports’, ‘text mining’, and ‘machine learning’. All studies using statistical police reports to classify crimes were excluded. This is because our topic is about providing crime details to students as an educational resource, preferably narrative police reports. Then we conducted another search, with the goal of finding a suitable open-source database. Because we could not find open-source narrative reports from the police, we turned to alternative sources: ‘legal court documents’ published as databases in various countries. We reviewed studies that used these databases in various fields in general, as well as the CAP database, which we utilized in our research. We also reviewed the methods and machine learning algorithms used in these studies to get a general idea of effective approaches to crime classification and legal document analysis.

2.1. Crime classification

This section explores recent applications and contributions in crime classification using narrative records across various contexts from diverse sources, such as police reports, child welfare records, and social media content, with text mining, natural language processing (NLP), and ML techniques. The literature reviewed demonstrates the potential of these techniques for automating crime analysis tasks and extracting valuable insights from free-text data routinely collected by law enforcement agencies. However, there are limitations and challenges that have been identified in the research.

Some studies have used open-source datasets derived from news websites (Carnaz et al., 2021; Thaipisutikul et al., 2021), a legal website (Li & Qi, 2019), and social media platforms like Twitter (Lal et al., 2020) and YouTube (Ashraf et al., 2020). These datasets are not suitable for our study because they lack details about crimes in the reports, and some of them use datasets in languages other than English. In contrast, several studies were acquired which classify crimes according to unstructured narrative police reports and for various objectives. The systems implemented in the studies (Birks et al., 2020; Kuang et al., 2017; Mohemad et al., 2020; Percy et al., 2018) aimed to identify and prevent crime patterns such as burglaries, housebreakings, and murder. Domestic violence has also been the subject of research, including that of Karystianis et al. (2022). It emphasizes the various forms of abuse and the injuries sustained by victims to offer distinctive perspectives on domestic violence patterns that can be utilized to monitor and regulate. To reduce violent deaths, Arseniev-Koehler et al. (2022) identified patterns surrounding suicides, homicides, and other violent fatalities. Furthermore, in an effort to formulate social service initiatives, Victor et al. (2021) identified instances of violence in the writings of abused children. In addition, Geurts et al. (2023) and Baek et al. (2021) each conducted research that resulted in the development of a risk assessment tool and an early identification of risk situations, respectively. The mentioned studies utilize various ML algorithms for different crime-related tasks like analysis, classification, crime detection, and prevention. A summary of these studies is presented in Table 1.

Table 1. Summary of related studies to the crime classification.

Study	Purpose	Classification	Dataset			ML algorithms
			Source	Access	Size
Thaipisutikul et al. (2021)	Illustrate the temporal and spatial occurrence of violent incident.	Burglary, Accident, Drug, Corruption and Murder.	Thai online news articles.	Open	7,879	NB LR GBM	RF KNN SVM
Carnaz et al. (2021)	Information extraction from Portuguese criminal reports.	Named entities extraction and classification using the (Who, What, Why, Where, When, and How) method.	Portuguese crime reports and criminal news.	Open	163	Perceptron
Li and Qi (2019)	Detecting serial robbery crimes committed by the same suspects.	Serial or non-serial robbery crimes.	OpenLaw website from Zhengzhou City in China.	Open	334	LR SVM KNN	NN DT
Lal et al. (2020)	Analyzing and classifying crime-related tweets.	Crime and non-crime.	Twitter.	Open	369	NB RF	DT (J48) ZeroR
Ashraf et al. (2020)	Detection of violent threat language in YouTube comments.	Violent threats and non-threats.	YouTube.	Open	28,643 sentences & 9,845 comments from 19 videos.	1D-CNN LSTM BiLSTM
Kuang et al. (2017)	Crime prevention and reduction.	226 crime types	Los Angeles Police Department	Closed	1,027,168	NMF
Percy et al. (2018)	Crime prevention and reduction.	Identify regions with similar patterns of crimes.	Dyfed-Powys Police in UK	Closed	N/A	Affinity propagation Spectral clustering
Birks et al. (2020)	Crime prevention and reduction.	Burglary	West Yorkshire Police Department	Closed	9621	LDA
Mohemad et al. (2020)	Crime prevention and reduction.	Modus operandi from Housebreaking reports: method, role, oddity, weapon, and location.	Malay Royal Police Department	Closed	100,383	k-means k-medoids
Karystianis et al. (2022)	Domestic violence surveillance and monitoring.	44 abuse types 17 Injury types	New South Wales Police Force.	Closed	492,393	N/A Text mining
Arseniev-Koehler et al. (2022)	Reduce violent death cases.	Discovering patterns about suicides, homicides, and other violent deaths.	US National Violent Death Reporting System	Closed	272,964	DATM
Victor et al. (2021)	Identify domestic violence as an active service need in child welfare investigation summaries.	Domestic violence presence or absence.	Substantiated referrals of child maltreatment in the State of Michigan.	Closed	75,809	RF
Geurts et al. (2023)	Developing a risk assessment instrument.	Repeat victimization of severe and less severe violent crimes.	Dutch police records.	Closed	116,680	DT RF
Baek et al. (2021)	Early recognition of risk situations.	21 crime types.	Korea Information System of Criminal Justice Services	Closed	5000	DNN CNN	NB SVM

ARL: Association Rule Learning; CNN: Convolutional Neural Network; DT: Decision Trees; DNN: Deep Neural Network; DATM: Discourse Atoms Topic Model; GBM: Gradient Boosting Machine; KNN: K-Nearest Neighbor; LDA: Latent Dirichlet Allocation; LR: Logistic Regression; NB: Naive Bayes; NN: Neural Network; NMF: Non-negative Matrix Factorization; RF: Random Forest; SVM: Support Vector Machines.

One of the constraints of these studies is their reliance on private or restricted data sets, which could hinder the progress of the research and prevent external validation of the findings. Online-published and open-source court databases were proposed as a potential solution to this problem. A potential analytical bias is introduced because these investigations mine and extract features for classification from the same dataset, which is the second limitation. The presence of bias in text mining and ML models with respect to groups of individuals or categories of crimes can result in inaccurate or unjust outcomes. Furthermore, the functionality of the classifier cannot be applied to different datasets that fall under the same domain. As a result, a proposition was made to establish a crime dictionary, comprising terms gathered from reliable external sources within the domain of crime, with the intention of incorporating them into various datasets.

2.2. Legal documents

Legal document analysis has become an increasingly important area of research in recent years, with a growing need for automated methods to process and analyze legal documents. Many studies have focused on developing ML and NLP techniques using different datasets that include a variety of legal documents, reflecting the complexity and variety of legal contexts. The dataset comprises legal judgments, court case documents, and other legal texts, providing a rich source of unstructured data for analysis. The studies have utilized datasets from various legal domains, including Indian Supreme Court judgments, US state court decisions, European Court of Human Rights cases, and more, as listed in Table 2.

Table 2. Summary of studies using legal documents.

Study	Purpose	Dataset
		Source	Access	Size	Language
De Oliveira and Nascimento (2022)	Brazilian court decision document similarity detection.	The Ordinary Appeal Brought (ROI).	Open	210,000	Portuguese
Garat and Wonsever, (2022)	Protect the privacy of case participants while making court documents public.	The National Jurisprudence Base (NJB) from different courts of the Judicial Branch in Uruguay.	Open	80,000	Spanish
Nuranti et al. (2022)	Predict Indonesian court punishment category and length.	The Indo-law dataset from the Indonesian Supreme Court Decision (ISCD) website.	Open	22,630	Indonesian
Purnomo et al. (2021)	Assess Kalimantan Forest fire actors’ relationships, activities, and funding, particularly in land clearing and burning.	Forest fire cases from the official website of the Supreme Court of the Republic of Indonesia (SCRI).	Open	N/A	Indonesian
Prasad et al. (2022)	Explore decision classification of large unstructured and unannotated legal documents and formulate it as a real-life decision prediction problem.	The Indian Legal Documents Corpus (ILDC) from the Supreme Court of India.	Request	42,409	Hindi
Mandal et al. (2017)	Automate the process of similarity measurement between court case documents.	The LIIofIndia website hosted by the Legal Information Institute of India.	Open	33,545	Hindi
Gupta et al. (2023)	Use topic modeling and sentiment analysis to help users understand legal judgments and identify relevant sentiments faster.	The Supreme Court of India website.	Open	700	Hindi
Tyss et al. (2023)	Introduce article-aware classification for European Court of Human Rights (ECHR) cases to improve legal judgment prediction.	The LexGLUE ECtHR dataset from the public court database HUDOC.	Open	11,000	English
Habernal et al. (2023)	Develop methods for mining legal arguments in court decisions and automatically identifying, classifying, and analyzing them.	The CaseLaw dataset from the Harvard Law School Library and Ravel Law.	Open	65,908	English
		ECHR dataset.	Open	65,908	English
		The annotated LAM: ECHR corpus.	Open	373	English
Vatsal et al. (2023)	BERT-based methods for broad and fine-grained US Supreme Court case classification are available.	The Supreme Court Database (SCDB) from Washington University School of Law.	Open	8419	English
Song et al., (2022)	Create effective multi-label classification algorithms for legal motion detection and case prediction.	The POSTURE50K dataset from U.S. courts.	Open	50,000	English
Savelka et al., (2019)	Evaluate methods to simplify case law sentence retrieval for statutory interpretation.	The U.S. Caselaw Access Project (CAP) from federal and state courts.	Partially open Request the rest.	> 6.7 million	English
Savelka and Ashley, (2020)	Discover sentences from court documents for explaining the meaning of statutory terms.	The CAP dataset from federal and state courts.	Open	26,959 sentences	English
Sokol et al., (2020)	Analyze court case usage of common antitrust terms to determine how economics affects antitrust cases.	The CAP dataset from federal and state courts.	Partially open Request the rest.	N/A	English
Chang et al. (2020)	Solve big data caselaw research issues like data interpretation and limited caselaw extraction.	The CAP dataset from federal and state courts.	Open	9,899	English
Tang et al. (2020)	Create a semi-automatic legal knowledge graph building system and test its semantic annotation capabilities.	The CAP dataset from federal and state courts.	Open	280 legal texts	English
Castano et al. (2020)	Prove that the LATO-KM can find new legal terminology to enrich the LATO ontology.	The CAP dataset from the State of Illinois.	Open	180,000	English
Petrova et al. (2020)	Set baselines for outcome extraction and prediction.	CASELAW4 dataset subset from The CAP dataset.	Open	350,000	English

The dataset has been employed for a multitude of tasks, such as legal judgment prediction in many studies (Habernal et al., 2023; Nuranti et al., 2022; Prasad et al., 2022; Petrova et al., 2020; Tyss et al., 2023), sentiment analysis to assess public sentiments on legal issues (Gupta et al., 2023), document classification (Prasad et al., 2022; Song et al., 2022; Tyss et al., 2023; Vatsal et al., 2023), and information extraction (Chang et al., 2020; Castano et al., 2020; Savelka et al., 2019; Savelka & Ashley, 2020; Tang et al., 2020). The studies have also explored the use of the dataset for building semantic annotation systems and knowledge graphs to represent legal knowledge in a structured format (Tang et al., 2020). It is important to note that the dataset has been instrumental in advancing research in the field of legal document analysis, enabling the development and evaluation of ML and NLP techniques tailored to the legal domain. The utilization of this dataset across multiple studies underscores its significance as a valuable resource for training and testing models, as well as for gaining insights into legal decision-making processes and public perceptions of legal issues.

We concluded, therefore, that we could utilize one of the databases in a new domain, namely academic police education, given that most of them are freely available, open source, and readily accessible. The CAB database, which contains numerous court decisions published in the Harvard Law School Library, was selected because it is one of the most comprehensive legal databases available. Ideal for big data initiatives and streamlining the data mining procedure, the CAP dataset is presented in a pristine, machine-readable format. Previous research in the field of legal document classification and analysis using CAP datasets has not specifically addressed the classification of legal documents based on crime scenes or crime types, which is the primary focus of our intended study. The results of our work seek to assist in educating investigators and helping them organize crime-related materials and information so they can study crime patterns more abstractly.

3. Proposed framework

This section presents our novel framework aimed at classifying legal documents into five criminal types: beating, shooting, stabbing, strangulation, and multiclass. The overall architecture presented in this research is depicted in Figure 1. The key challenge in achieving our goal relates to the unstructured nature of the documents, which contain narrative information that is not related to the crime scene. To get an overview of its topics and understand its contents, the usual solution is to utilize clustering algorithms. Thus, a Term Frequency-Inverse Document Frequency (TF-IDF) technique with the K-means clustering algorithm was applied as an initial experiment. TF-IDF is a simple technique for converting text into a feature vector based on the Bag-of-Words (BoW) model (Kim & Gil, 2019). Different topics were created depending on the word frequency in the document and the importance of the word in the dataset. However, the results were not suitable for our study, as the outcome topics and their keywords were far from what was required. The reason is that the CAP database is primarily for court decisions and not police reports. Most of the documents consist of trial and appeal procedures with other information like the offender’s life history, while the details of the crime scene and scenario are mentioned as background in the document, and their length varies from document to document. For example, the document contains 150 pages, and the important part is about one and a half pages. Therefore, the important keywords in our research may be neglected or of little significance. To solve this problem, a framework was proposed, as shown in Figure 1, that takes advantage of text mining to explore a big database rapidly and efficiently by using the prior definition of topics with their own keywords as well as ML techniques for analyzing court documents to achieve our objectives. Our primary contribution consists of compiling terms from various sources in the fields of crime and forensics into a crime dictionary. This enables us to extract features from the database that are not restricted to the terms found in the database under investigation, thereby enhancing its generalizability and applicability to other databases. According to the proposed framework, the following main steps are involved in building ML models:

Figure 1. The proposed framework architecture.

1. Collect and filter the court decisions from the CAP dataset and label some of them for the ML models. Then, save the filtered documents with labels in the database.

2. Get the labeled documents from the database to be preprocessed and converted to word tokens.

3. Create a crime dictionary manually using some websites and research articles on homicides. It will be lemmatized and then stored in the database.

4. Next, based on the dictionary and BoW with TF-IDF techniques, the text tokens will be embedded and turned into appropriate vectors that can be used as features for the ML models.

5. Build a two-phase ML model that consists of two models: Crime Scene Existence (CSE) to find out if the document has information about the crime scene or not (binary classification). The second model is Crime Type (CT), which determines the type of crime committed based on the context of the words (multiclass classification). These models apply an appropriate ML algorithm to the TF-IDF vectors with their labels and create the classifiers. For the CT model, TF-IDF vectors of documents with only positive CSE will be used.

6. The resultant two classifiers from the system can help predict the classification of new court documents automatically.

The proposed model was programmed using the Python language with different libraries like Panda, Natural Language Toolkit (NLTK), and Scikit-Learn (sklearn). It consists of different modules, and the details of each are as follows.

3.1. Data collection

As mentioned previously, our work requires a publicly accessible database containing crime-related information. This does not apply to police reports, as they are subject to privacy laws due to the sensitivity of their details. Therefore, the CAP database was chosen, as it was collected by Harvard University and it is a comprehensive legal database with much information that contains 6,920,596 documents, which represent everything published in the last three hundred years. Although it was not designed specifically to analyze crime scenes or types of crime, the opinions, legal data, and information it provides are valuable sources for educational and research purposes, including the study of cases-related to the crime. After reviewing some cases, we found that the CAP dataset can provide details about crimes that can be used in our work. In addition, it is characterized by the presence of a special website that contains search and retrieval tools, as well as metadata and tagging, to make it easier for users to find specific legal cases or concepts, which is the main reason for choosing it, as it has benefited us in the process of selecting the reports we need to classify and organize cases related to crime scenes and types of crimes. This section outlines the processes used to collect the data and select the legal documents in the CAP dataset. The documents were downloaded as bulk data from the CAP website (The Caselaw Access Project (CAP), 2022). The summary of our data collection processes is displayed in Figure 2.

Figure 2. Flowchart of data collection from CAP website.

3.1.1. Documents selection

According to the latest statistics, the database contains 6,920,596 documents. For our experiment, 36,230 documents were selected using three filter stages. Firstly, 6,920,596 documents were filtered down to n = 76,931 by using the phrase ‘homicide murder’ to meet the case study for our research. Secondly, the rest of these documents were filtered to n = 45,679 according to their publication date. As all the documents over the past three hundred years are represented in the CAP dataset, the documents for the last fifty years, from 1970 to 2018, were the only ones considered. Lastly, a minimum threshold of 2,000 words was set to eliminate any document that may contain very modest information or lack details. As a result, the final total was n = 36,230 documents. Statistics for the CAP dataset depending on words are shown in Table 3.

Table 3. Statistics of the CAP dataset used for experimentation.

No. documents	No. words
	Longest document	Shortest document	Average
36,230	158,992	2001	7,372

3.1.2. Content analyzing

Each document in the CAP dataset has a range of metadata, such as the case name, citation, court, date, etc. Some cases were read randomly to find the important details of the case, which were found in the ‘head_matter’ and ‘Opinions’ under the case body section. It is noted that the documents vary, whether in length as stated in Table 3, writing style, or content, as it depends on the writer. Content may include court decisions, general regulations, the constitution, and court proceedings. In addition, some documents contain a crime scene description, witness statements, and the crime scenario, which is relevant to our study and often found under the headings ‘Facts’ and ‘Background’. An example of a document from the CAP dataset with different content is shown in Table 4. This information helped facilitate the identification of documents for the annotation process.

Table 4. An example of the different information contained in the court document.

Content type	The narrative text from the document
	Unrequired information
Court opinion	In People v. Leach, 391 Ill. App. 3d 161, 908 N.E.2d 120 (2009), we affirmed defendant’s conviction. On September 29, 2010, pursuant to its supervisory authority, the Illinois Supreme Court directed this court to vacate our judgment in Leach, 391 Ill. App. 3d 161, 908 N.E.2d 120, and to consider whether, in light of the holding of People v. Williams, 238 Ill. 2d 125 (2010), a different result is warranted. We conclude it is not and for the following reasons affirm the judgment of the circuit court.
Offender’s life history	Defendant related that for the past two years he lived at 13795 Grace Street in Robbins with Latyonia and their three children, Emanuel, age 11, Sakiyo, age 7, and Sakera, age 6. Defendant was 30 years old. He had left Thornton High School in 1993, after his sophomore year, and supported himself and later his family doing temporary work and odd jobs.
Trial procedure	Following presentation of the State’s case, the defense moved for a finding of not guilty, asserting that the evidence failed to establish that Curtis Leach was guilty of first degree murder. At best, the defense argued, the State had proved defendant guilty of involuntary manslaughter, based upon defendant’s recklessness, or second degree murder in the heat of intense passion while engaged in mutual combat with Latyonia, who was the aggressor and outweighed the defendant by 100 pounds.
Common law text	Section 115—5(a) of the Code of Criminal Procedure of 1963 thus provides: Any writing or record, whether in the form of an entry in a book or otherwise, made as a memorandum or record of any act, transaction, occurrence, or event, shall be admissible as evidence of such act, transaction, occurrence, or event, if made in regular course of any business, and if it was the regular course of such business to make such memorandum or record at the time of such act, transaction, occurrence, or event or within a reasonable time thereafter.
	Required information
Crime scenario	According to defendant, Latyonia then ‘got to pushing me’ and he ‘pushed her back.’ They got ‘into an altercation where we was wrestling’ and Latyonia then began to hit defendant in his face, first with an open hand and then with her fist. He tried to contain Latyonia by grabbing her hands, but she was stronger and ‘had so much anger she broke loose.’ They continued wrestling on the bed until defendant managed to push Latyonia down on the bed and restrain her as she lay on her back.
Crime scene description	Evidence technician Lilliebridge processed the scene with her partner. She observed some blood on the pillow beneath Latyonia’s head as well as on the bedding underneath her hand. The bedroom was pretty neat; nothing looked out of place or thrown about. Additionally, the bed had a wooden headboard containing toiletry items which ‘were upright and undissturbed’. After processing the scene, Officer Lilliebridge later photographed defendant at the Robbins police department.
Forensic statement	His external examination revealed that Latyonia was clothed in a house shirt and underwear, and her body was accompanied by an elastic bandage. Latyonia was 5 feet 9V2 inches tall and weighed 295 pounds. Her right eye had ‘multiple petechiae hemorrhages on the upper scleral’ and ‘diffuse scleral hemorrhages’ on the lower portions; her left eye also had ‘diffuse hemorrhage’ on the lower portions. A foamy fluid had escaped from Latyonia’s nostrils and there were bluish marks on both sides of her neck. There were no wounds to Latyonia’s face or upper extremities of her body.

(Snippet from Case_id: 3716989).

3.1.3. Documents annotation

For classification purposes, some documents from the CAP dataset need to be tagged. Due to the huge number of documents in the dataset, previous information was used to guide our document selection. A subset of metadata was obtained from documents that met our requirements, and two other features were extracted, namely word count and the terms ‘Facts’ and ‘Background’. Accordingly, the documents were sorted and divided into various groups based on these new features. Then, 374 documents were selected in a random fashion from each group and labeled manually for the suggested ML models: CSE and CT.

3.2. Text preprocessing

As the CAP dataset documents are row data, the data must be prepared and cleaned before being analyzed to get higher quality results. This step is important to emphasize features that are used as inputs to our ML model (Joshi, 2020; Torfi et al., 2020). Basic techniques were implemented as follows:

• Convert text to lowercase.

• Remove digits, special characters, and extra space from the text.

• Through the tokenization of the text into small parts called tokens, word tokens were chosen, and all tokens that had fewer than two letters were dropped.

• Remove stop words that have no significant effect in the text, such as conjunctions, determiners, and helping verbs. The predefined stop words from the NLTK package were used.

• Lemmatization of the word is an improved method of stemming. This process will return the word forms to their root, which exists in the language’s dictionary. The one that exists in the NLTK package was applied to nouns, verbs, and adjectives. The term ‘wound’ is an important term in our work. An exception was made for it because it was returned to ‘wind’ as the root, which is an inappropriate word in the context.

3.3. Crime dictionary

As previously stated, the CAP dataset includes mixed information in an unstructured format. Thus, the output text from the preprocessing remains inadequate. To improve the performance of the ML classifiers, a dictionary of the topic and its keywords was built, and it contained only the most distinguished keywords. Our field is homicide crimes, and the evidence collected from crime scenes plays an essential role, such as fingerprints, blood, saliva, drugs, and others. The most important is the crime tool, as the investigator can find the weapon at the crime scene or infer it from the marks that exist on the scene and the wounds on the body. All this information is recorded in police reports. In our experiment, the proposed crime dictionary was focused on the associated vocabulary of the crime tool to demonstrate the presence of the crime scene description in the documents. Further, the methods of murder were classified into four types: beating, shooting, stabbing, and strangulation. A crime dictionary was manually gathered, which includes crime types, tools, and associated vocabulary that is repeated in the context. It depends on a site that reviews physical evidence and some articles on homicides (Amita et al., 2021; Bohnert et al., 2006; Papi et al., 2020). The suggested keywords in the dictionary were shown in Table 5. In fact, there are eight different lists of topics. It is noticeable that the keywords vary in terms of usage, from common to rare. As the dictionary’s terms are hand-picked, no preprocessing procedures are required. However, it is best to perform only one procedure, lemmatization, to guarantee that the terms are in the root forms and can be compatible with the CAP database after preprocessing the documents.

Table 5. Crime dictionary (list of crime tools and associated vocabulary).

Type	Crime tool			Associated vocabulary
(a) Beating	Axe Bat Board Bottle Club Crowbar	Cup Fist Foot Hammer Hand Iron Bar	Mallet Pipe Poker Rock Stick Truncheon	Abrasion Abuse Beat Blood Blow Blunt	Bone Bruise Contusion Crush Ecchymosis Eye	Fall Fracture Head Hemorrhage Hit Jaw	Kick Laceration Lip Punch Push Shak	Skull Smash Strike Throw Torsion Trauma
(b) Shooting	Colt Firearm Gun	Handgun Pistol Revolver	Rifle Shotgun	Abdomen Ammunition Ballistic Blood Bloodstain Bullet Cartridge	Chest Defect Direction Fire Fragment Gunpowder Gunshot	Head Hemorrhage Hit Hole Injury Neck Path	Pellet Penetrate Perforate Projectile Shock Shot Slug	Soot Speed Stipple Track Trajectory Travel Wound
(c) Stabbing	Axe Blade Cleaver Fork Glass	Ice Knife Machete Pen Pick	Razor Saw Scissor Screw Sword	Abdomen Abrasion Amputation Angle Arm Attack Blood Bloodstain Bruise Chest	Chop Crush Cut Deep Depth Edge Finger Force Forearm Fracture	Hand Hemorrhage Hit Inch Incise Injury Intensity Laceration Length Mutilation	Neck Palm Penetrate Penetration Puncture Shape Sharp Size Slash Slit	Speed Stab Tear Thick Throat Track Trunk Width Windpipe Wound
(d) Strangulation	Acid Acrolein Bag Bed Blanket Cable Carbon Clothesline Cord Cyanide	Duct Gas Hose Hydrogen Ligature Monoxide Necktie Nitrogen Pillow	Plastic Scarf Sheet Sulfide Tape Tie Towel Twine Wire	Abrasion Asphyxia Asphyxiate Asphyxiation Aspirate Bind Bluish Choke	Compression Contusion Cyanotic Eardrum Engorgement Fingernail Hang	Head Knot Laceration Mark Mouth Neck Nose	Over Pressure Pull Purplish Smother Strangle Strangulation	Suffocate Suffocation Swell Swollen Tongue Vomit White

3.4. Feature extraction

The BoW and TF-DF models are the main techniques in our feature extraction process. The classic BoW model extracts words from all the contents of the document. This causes computational complexity, especially if the data volume is huge (Kim & Gil, 2019). TF-IDF helps encode words in documents by retrieving words with numerical statistics. For a particular document, TF calculates the ratio of the frequency of a keyword in the document. In contrast, the IDF denotes the dataset’s significance for the keyword (Thaipisutikul et al., 2021). Here, only associated vocabulary terms that exist in the crime dictionary were extracted from the documents, and the rest of the words were filtered out, which makes it more specific to what is required and allows faster analysis by reducing the BoW dimension to 151 keywords only. The ‘TfidfVectorizer’ function from sklearn was adopted for this task. The following equations were used to find the $\mathit{\text{tfidf}}$ value for each document (Borcan, 2020; Kim & Gil, 2019): (1) $$t\mathrm{f}\left(w,d\right)=\mathrm{\text{log}}\hspace{0.17em}\left(1+f\left(w,d\right)\right)$$(1) (2) $$\mathit{\text{idf}}\left(w,D\right)=\mathrm{\text{log}}\hspace{0.17em}\left(\frac{N}{f(w,D)}\right)$$(2) (3) $$\mathit{\text{tfidf}}\left(w,d,D\right)=\mathit{\text{tf}}\left(w,d\right)\times \mathit{\text{idf}}(w,D)$$(3)

Where $f(w,d)$ is the frequency of word $w$ in document $d,$ $N$ is the number of documents in the dataset, and $f(w,D)$ is the frequency of word $w$ in the whole dataset $D.$ An example of a portion of a BoW representation with TF-IDF is presented in Figure 3, which shows a $\mathit{\text{tfidf}}$ value for some vocabulary associated with a shooting-type crime.

Figure 3. An example of BoW representation with TF-IDF for five documents from the CAP dataset.

3.5. ML building

This phase contains two phases: training and testing with two-stage classification using CSE and CT models, as indicated in Figure 1. Different functions and methods from the sklearn library were applied to execute our experiment. Several supervised classification algorithms have been implemented (Sen et al., 2020; scikit-learn: Machine Learning in Python, 2023): Decision Trees (DT), K-Nearest Neighbor (KNN), Logistic Regression (LR), Naive Bayes (NB), Random Forest (RF), and Support Vector Machines (SVM). The CSE model is a binary classification that is responsible for checking if a document contains a crime description. It has two classes: ‘Not Exist’ labeled with 0, and ‘Exist’ labeled with 1. In the next stage, the CT model will work only on documents that have an ‘Exist’ class resulting from the first model. It is a multiclass classification labeled from one to five that is responsible for classifying documents according to crime types.

3.6. Model prediction

The purpose of this step is to apply the CSE and CT classifiers that were acquired from the previous step to new data. The remaining unlabeled documents from the CAP dataset were utilized as inputs to the prediction process. To perform this step, the ‘predict’ function in the sklearn library was used to get the finding, which will be saved in the dataset later.

3.7. Experimental setup

3.7.1. Dataset division

The CAP dataset was divided into 70% for the training set and 30% for the testing set using the ‘train_test_split’ function. It corresponds to 262 and 112 documents in the CSE model, respectively. Further, the CT model was trained on 130 documents and tested against 57 documents. Details of the data division are shown in Table 6.

Table 6. Statistics of the dataset splitting for the experiment.

Model	Class	Training	Testing	Total
CSE	Not Exist	131	56	187
	Exist	131	56	187
	Total	262	112	374
CT	Beating	22	10	32
	Shooting	45	20	65
	Stabbing	30	13	43
	Strangulation	15	6	21
	Multiclass	18	8	26
	Total	130	57	187

3.7.2. Performance metrics

To evaluate our work, commonly applied metrics were compared between ML algorithms. The first metric is the accuracy score, which was calculated by the equation: (4) $$\mathit{\text{Accuracy}}=\mathit{\text{tp}}/P$$(4) where $\mathit{\text{tp}}$ is the number of correct predictions and $P$ is the total number of all predictions. In addition, the k-fold cross-validation method was applied with 10 folds to validate the model effectiveness and to avoid the underfitting problem because of the small dataset. With this method, the training set is partitioned into k equal-sized folds (groups), where $k$ is the total number of partitions. If $k=10,$ for instance, the dataset will be divided into ten subsets. In this scenario, nine divisions were employed for training purposes and one for evaluation. Iteratively, the system was trained ten times, each time with a new partition serving as the test set and the remaining nine partitions serving as the training set. The findings are then summarized by averaging them to get the validation accuracy (Ramezan et al., 2019). To choose the best value of the random state ($n$) parameter in the ‘train_test_split’ function that splits the dataset into an appropriate distribution, different experiments with the value of $0\le n<300$ were implemented, and then the difference between the validation accuracy and the model accuracy was calculated. (5) $$d=\mathit{\text{Accuracy}}-\mathit{\text{Validation}}$$(5) (6) $$N=n,\mathit{\text{if}}\mathrm{ }\left(-\mathit{\text{th}}<d<\mathit{\text{th}}\right)\mathit{\text{\hspace{0.17em}and\hspace{0.17em}}}\mathrm{\text{Max}}\left(\mathit{\text{Accuracy}}\right)$$(6) where $N$ is the selected random state parameter and $\mathit{\text{th}}$ is the threshold used to determine the suitable difference between the two accuracies, which in our experiment was 0.6. The ML algorithm that gives the best $\mathit{\text{Accuracy}}$ will be deployed, and the trained classifier will be saved in the database for use in future prediction. For the CT model, only documents that contain a crime scene will be used. The selected $N$ that was adopted in our experiment for different algorithms and models is listed in Table 7.

Table 7. The selected value of the Random State parameter ($N$) used in our experiment.

Model	DT	KNN	LR	NB	RF	SVM
CSE	254	194	20	220	163	202
CT	223	96	54	8	51	86

In contrast, the evaluation metrics for each class in the model will be calculated using the parameters from the confusion matrix (Krüger, 2016), as illustrated in Figure 4. The following equations can be used to find the metrics: (7) $$\mathit{\text{Precision}}=\mathit{\text{TP}}/(\mathit{\text{TP}}+\mathit{\text{FP}})$$(7) (8) $$\mathit{\text{Sensitivity}}=\mathit{\text{TP}}/(\mathit{\text{TP}}+\mathit{\text{FN}})$$(8) (9) $$\mathit{\text{Specificit}}=\mathit{\text{TN}}/(\mathit{\text{TN}}+\mathit{\text{FP}})$$(9) (10) $$F_{1}s\mathit{\text{core}}=2\times \frac{\mathit{\text{Precision}}\times \mathit{\text{Sensitivity}}}{\mathit{\text{Precision}}+\mathit{\text{Sensitivity}}}$$(10) where $\mathit{\text{TP}}$ indicates the number of the class was predicted while it is real, $\mathit{\text{TN}}$ indicates the number of the class was not predicted and it is not real, $\mathit{\text{FP}}$ indicates the number of the class was predicted but it is not real, and $\mathit{\text{FN}}$ indicates the number of the class was not predicted but is real. $\mathit{\text{Precision}}$ is used to measure all positive predictive values. $\mathit{\text{Sensitivity}}$ also known as recall represents the true positive rate while $\mathit{\text{Specificity}}$ represents the true negative rate. Finally, $F_{1}s\mathit{\text{core}}$ is the harmonic mean of precision and recall. For all these metrics, the higher the value, the better.

Figure 4. Confusion matrix for the classification model.

4. Results and discussion

This work is research on an increasingly important academic area of research on the utilization of published court reports in the police academy for educational purposes, classifying court reports that contain murders using the crime dictionary, and predicting the class from the description of the crime scene even if the crime instrument is not declared in the document. All that is done using advanced computer techniques such as text mining and supervised ML methods.

4.1. Crime dictionary evaluation

In this section, the statistical data of the keywords that were extracted from all documents in the CAP will be presented and illustrated, which totaled 36,230 documents as mentioned in Table 3. Different statistics were generated depending on the suggested crime dictionary, which presented in Table 5. Our goal in providing this overview of the documents’ contents is to figure out whether the CAP dataset and crime dictionary are appropriate for our work. These statistics will be obtained using the BoW model based on the presence of the keyword in the document without regard to its frequency. A document that contains the keyword once is equal to a document in which the keyword is repeated.

The initial stage was to identify crime tools that were directly mentioned in the documents; the results of this analysis are shown in Figure 5. According to the statistical data, the most common criminal tools from each type stated in the documents are hands, guns, ice from an ice pick, and cords. However, several instruments are never mentioned in the documents at all, like the truncheon and acrolein. Tools such as poker, mallet, fork, scissor, necktie, and scarf were also cited by fewer than 1.0% of respondents. That means many of the documents in the dataset are written in common terms, with only a few references to scientific terminology. In other words, they are lacking in fine details. Additionally, terms including hand, bar, foot, ice, pen, cord, etc. are frequently used in public life and are not categorized as well-known criminal tools such as guns, pistols, knives, and others. Therefore, the number of documents for these terms is high. For example, the terms ‘ice’ and ‘cord’ appear in 99.67% and 97.53% of the documents, respectively, but the words ‘gun’ and ‘knife’ appear in 59.15% and 23.62%, respectively. Since the CAP dataset is comprised of legal documents as opposed to police reports, it should not be assumed that every given document containing these instruments is necessarily associated with criminal activity. These tools may be stated in the report but have little to do with the incident itself, and not all crime reports include a description of the crime scene.

Figure 5. Statistics of crime documents (%) in the CAP dataset by crime tools. (a) Crime type: Beating. (b) Crime type: Shooting. (c) Crime type: Stabbing. (d) Crime type: Strangulation.

A crime dictionary heatmap was also presented, as shown in Figure 6. It shows the number of documents that include the crime tool and its associated vocabulary in the same document, to explore the correlation between them. Dark cells indicate a larger number of documents. From the examples that illustrate the relationship, in more than 20% of the documents with multiple associated vocabulary, in a beating-type crime, there is a relationship between ‘hand’ and the associated vocabulary like ‘throw’, ‘strike’, ‘lip’, ‘hit’, ‘head’, ‘fall’, ‘eye’, ‘blood’, ‘beat’, and ‘abuse’. The term ‘gun’ is used in the shooting crime and is related to terms such as ‘wound’, ‘shoot’, ‘injury’, ‘hole’, ‘hit’, ‘head’, ‘gunshot’, ‘fire’, ‘bullet’, and ‘blood’. Also, it is noticeable that the association of ‘pen’ in stabbing crime with the following terms: ‘wound’, ‘stab’, ‘size’, ‘neck’, ‘length’, ‘injury’, ‘hit’, ‘hand’, ‘force’, ‘finger’, ‘edge’, ‘cut’, ‘chest’, ‘blood’, ‘attack’, and ‘arm’. In strangulation crime, a term like ‘cord’ is associated with terms like ‘white’, ‘pull’, ‘neck’, ‘mark’, ‘head’, ‘hand’, and ‘bind’. Therefore, when manually classifying the reports, most of them fell under the following crimes: beating, shooting, stabbing, and strangulation.

Figure 6. Heatmap of crime documents (%) in the CAP dataset. (a) Crime type: Beating. (b) Crime type: Shooting. (c) Crime type: Stabbing. (d) Crime type: Strangulation.

Some cases have more than one body or more than one way to kill, like beating with stabbing, or stabbing with strangulation. Because of this, they were put in the ‘multiclass’ category, as shown in Table 6. Additionally, the heat map proves what was mentioned earlier, from Figure 5. The term ‘cord’ from strangulation is in about 97.53% of the documents as present in Figure 5(d), and after filtering using the associated vocabulary ‘hang’ in Figure 6(d), it becomes 63.8%. Also, the crime of hitting is typically committed with the hand, so there is a relationship between them in the heatmap; however, the hand is not typically considered a crime tool; thus, the percentage decreased from 77.19% to 50% with the terms ‘head’ and ‘hit’. In contrast, the well-known methods of murder, such as shooting with a gun and stabbing with a knife, showed similar results before and after, as they were 59.15% and 23.62%, which then became 50.1% with the term ‘shoot’ and 22.5% with the term ‘stab’, respectively. This makes it more likely that they are crime tools, even if they do not have the associated vocabularies.

In general, heatmaps show a connection between crime tools and their associated vocabulary. Therefore, descriptive data is often helpful at crime scenes. Even if the name of the crime instrument was not mentioned in the report, this clue can be used to infer it from the context of the text. These findings are also useful in our work, as the documents in the CAP dataset are unstructured court reports, and not all of them relate to homicides. Therefore, the dictionary was used to identify the important features that distinguish each type of crime to use it in the classification and organization of documents, thereby facilitating its application as a reference in academic education, which is our goal in this work.

4.2. Evaluation of ML models

To evaluate the performance of CSE and CT models, six different algorithms were compared based on different evaluation metrics. The metrics are confusion matrix, precision, sensitivity, specificity, F1-score, and accuracy scores.

4.2.1. Evaluation of CSE model

The results of the CSE model were displayed in Table 8 and Figure 7(a), which is a binary classification. The accuracy scores indicated in Table 8 were between 82.46% and 91.07%. As mentioned earlier, several tests were run to determine the optimal value of ($N$), shown in Table 7, for the distribution of the dataset so that the values of the two scores are relatively close. Among different algorithms, RF gave the best test accuracy with 91.07% and validation with 90.85%. Therefore, the results of the RF algorithm will be discussed in detail. As noted, the ‘not exist’ class had scores of 88.33%, 94.64%, 87.5%, and 91.28% for precision, sensitivity, specificity, and the F1-score, respectively. In contrast, the second class obtained the following scores: 94.23%, 87.5%, 94.64%, and 90.74%. Notably, the F1-score is relatively close for both classes in each algorithm, suggesting that the classifier is balanced in recognizing the correct class regardless of the error percentage. According to the confusion matrix in Figure 7(a), three documents out of 56 were misclassified as not containing a crime scene, while another seven were misclassified as containing a crime scene. The error rate may be due to the difference in the length of the documents, as there is a wide variation in the size of the reports as stated in Table 3.

Figure 7. Confusion matrix of ML models using different algorithms. (a) CSE model; (b) CT model.

Table 8. Results of CSE model using different algorithms.

Algorithm	Class	Precision	Sensitivity	Specificity	F1-score	Accuracy
						Test	Validation
RF	Not Exist	88.33	94.64	87.50	91.38	91.07	90.85
	Exist	94.23	87.50	94.64	90.74
DT	Not Exist	92.00	82.14	92.86	86.79	87.50	87.39
	Exist	83.87	92.86	82.14	88.14
LR	Not Exist	90.38	83.93	91.07	87.04	87.50	87.01
	Exist	85.00	91.07	83.93	87.93
NB	Not Exist	81.54	94.64	78.57	87.60	86.61	86.64
	Exist	93.62	78.57	94.64	85.44
SVM	Not Exist	84.75	89.29	83.93	86.96	86.61	86.31
	Exist	88.68	83.93	89.29	86.24
KNN	Not Exist	84.91	80.36	85.71	82.57	83.04	82.46
	Exist	81.36	85.71	80.36	83.48

Similar search to ours, as presented in Table 9, utilizes Twitter tweets. Our analysis reveals that the accuracy rate is 98.10%, since the tweets contain limited words, and they used these words as features of the classification model. On the other hand, the accuracy rate in our work is also considered high, about 91.07%, despite the discrepancy in the length of the reports and the fact that they are mixed with a lot of information, as the features extracted from the crime dictionary helped classify the documents well by highlighting the important words in the document and eliminating distractions.

Table 9. Comparing result of CSE model with previous study.

Study	Labels	Database	Size	Algorithm	Accuracy
Lal et al. (2020)	Crime and non-crime	Twitter	500 tweets	TF-IDF + RF	98.10
Our CSE model	Not Exist and Exist	The CAP	374 documents	TF-IDF + RF	91.07

4.2.2. Evaluation of CT model

The findings for the second model, a multiclass classification model based on crime type, are significantly different, as presented in Table 10 and Figure 7(b). The KNN, DT, and NB algorithms had an accuracy of less than 76% based on Table 10. The reason is that the training set is small with imbalanced data and needs to increase the number of labeled documents in each class. However, the SVM, RF, and LR algorithms had good accuracy scores of over 80%. The SVM algorithm provides the best score for the CT model, and therefore its classifier will be adopted in the discussion of the results. Its test accuracy is about 82.46%, and the validation score is around 82.31%. For the other metrics in Table 10, the strangulation class achieved the best levels of precision and specificity with 100%, while the shooting class had the highest levels of sensitivity with 95% and the highest F1-score with 92.68%. Conversely, the crime of multiclass type obtained the lowest percentage in all metrics except specificity, as follows: precision (60%), sensitivity (37.5%), and F1-score (46.15%). The lowest specificity is in the stabbing class at 93.18%. From the confusion matrix in Figure 7(b), there were two misclassified in the beating class, five misclassified in multiclass, and one misclassified in the rest of the classes, for a total of 10 misclassified out of 57 reports. The most crimes were confused in the multiclass crimes in all algorithms. This is because some crime documents include a detailed description of one method but not the other, and hence the classifier prefers the detailed method. For example, in a stabbing crime, the offender may also hit or strangle the victim, and the report includes facts about these events.

Table 10. Results of CT model using different algorithms.

Algorithm	Class	Precision	Sensitivity	Specificity	F1-score	Accuracy
						Test	Validation
SVM	Beating	72.73	80.00	93.62	76.19	82.46	82.31
	Shooting	90.48	95.00	94.59	92.68
	Stabbing	80.00	92.31	93.18	85.71
	Strangulation	100	83.33	100	90.91
	Multiclass	60.00	37.50	95.92	46.15
LR	Beating	80.00	80.00	95.74	80.00	80.70	80.77
	Shooting	86.96	100	91.89	93.02
	Stabbing	72.22	100	88.64	83.87
	Strangulation	100	66.67	100	80.00
	Multiclass	50.00	12.50	97.96	20.00
RF	Beating	80.00	80.00	95.74	80.00	80.70	80.77
	Shooting	95.00	95.00	97.30	95.00
	Stabbing	65.00	100	84.09	78.79
	Strangulation	100	83.33	100	90.91
	Multiclass	50.00	12.50	97.96	20.00
KNN	Beating	75.00	60.00	95.74	66.67	75.44	75.38
	Shooting	90.91	100	94.59	95.24
	Stabbing	55.00	84.62	79.54	66.67
	Strangulation	100	33.33	100	50.00
	Multiclass	80.00	50.00	97.96	61.54
DT	Beating	70.00	70.00	93.62	70.00	70.18	70.77
	Shooting	82.61	95.00	89.19	88.37
	Stabbing	81.82	69.23	95.45	75.00
	Strangulation	57.14	66.67	94.12	61.54
	Multiclass	16.67	12.50	89.80	14.29
NB	Beating	64.29	90.00	89.36	75.00	66.67	66.92
	Shooting	94.12	80.00	97.30	86.49
	Stabbing	50.00	53.85	84.09	51.85
	Strangulation	100	50.00	100	66.67
	Multiclass	33.33	37.50	87.76	35.29

In relation to the comparison with prior research, Table 11 illustrates that the first study (Petrova et al., 2020) classified documents based on legal outcomes utilizing deep learning (LSTM) and the CAB database. However, due to the limited number of documents examined, the accuracy score was approximately 81%. The second study (Thaipisutikul et al., 2021), which utilized the same algorithms with pre-labeled Thai news articles, attempted to categorize crimes. In doing so, it was extremely like our research. Despite the experiment utilizing a substantial number of articles, the obtained accuracy score of approximately 79.41% appears to be indicative of a lack of feature extraction customization, as evidenced by the fact that the whole article was utilized. The accuracy rate of the model proposed in this study, on the other hand, was approximately 82.46%, despite the experiment utilizing a smaller number of documents. This indicates that the crime dictionary had a positive impact on the results, particularly since the terms for this dictionary were collected from forensic reports and websites and not from the same documents used in the experiment.

Table 11. Comparing result of CT model with previous studies.

Study	Labels	Database	Size	Algorithm	Accuracy
Petrova et al. (2020)	Affirm, Reverse, and Mixed	The CAP	500 documents	LSTM	81.00
Thaipisutikul et al. (2021)	Burglary, Drug, Murder, Accident, and Corruption.	Thailand Online News Articles	7,879 articles	TF-IDF + SVM	79.41
Our CT model	Beating, Shooting, Stabbing, Strangulation, and Multilabel	The CAP	374 documents	TF-IDF + SVM	82.46

Although conducting further comparative experiments would enhance the quality of the work, direct comparisons are not possible due to the distinctive characteristics of our research design and dataset. To guarantee a rigorous assessment of our proposed method, we have undertaken thorough experimentation with a multitude of algorithms. Furthermore, these experiments function as an indirect comparison, showcasing the efficacy of our methodology in relation to the array of machine learning techniques presently accessible.

4.3. Results of ML prediction

As we mentioned earlier in the data collection section, the number of documents in the database after the filtering process is about 35,856 unclassified documents. The outcome classifiers from the proposed models will be applied to annotate these cases. The first classifier used to predict CSE, and 18,179 documents that describe the crime scene have been found, as displayed in Figure 8(a). These documents are classified by crime type using the CT classifier. From Figure 8(b), we find that crimes committed by shooting are the highest, at about 9,189 crimes. The crimes of stabbing and beating are close to each other with 3,648 and 3,346, respectively. There are also 1,333 strangulation offenses, with the multiclass type being the least at 663 crimes.

Figure 8. Classification of documents from the CAP dataset according to our experience. (a) Crime scene existence model. (b) Crime type model.

Our proposed method for classifying crime types in legal documents is a big step forward, but it’s important to look at where it might fall short. One potential drawback is that our technique strongly relies on the crime dictionary, which, while extensive, may be limited. Unusual or unique criminal instruments and tactics may be underrepresented in classification results. Furthermore, categorization performance is influenced by the quality and detail of the narrative in crime reports, which can vary. In addition, while the accuracy rates seem promising, they could be more flawless, and misclassifications may compromise the system’s trustworthiness in real-world applications. Another limitation is the possibility of bias in the crime vocabulary compiled from existing forensic reports and websites. If the sources of these terms contain inherent biases, the classification results may reflect them.

In summary, while our method is highly accurate and offers a novel approach to classifying crime types in unstructured texts, we are constantly working to enhance and improve the model. Future studies will try to expand the criminal dictionary, investigate the integration of other contextual aspects, and solve the observed flaws to improve the approach’s robustness and application.

5. Conclusions

Our work is just the first step toward activating and bringing attention to the legal databases in academic education. We presented a novel framework to assist students studying law and police procedures in effectively organizing and analyzing crime scene data. This framework involves classifying unstructured crime scene documents and making predictions about the types of crimes that occurred. Therefore, legal and case-law documents in this study were collected to study crime scenes through criminal investigations. We have successfully built a machine learning model using a limited number of features, which yields highly accurate outcomes in the classification of court documents according to crime scenes and types of crimes. These legal documents have been categorized to make it easier for people who are interested to find the information they need. Finally, we tested various classification algorithms to determine the best classifier for correctly predicting new crimes classes.

In NLP applications, it is difficult to classify huge, unlabeled texts with a narrative writing style. Therefore, the present work proposes text mining with NLP and supervised ML techniques to build a framework model that utilizes two classifiers: one that identifies documents that contain crime scenes, and another that labels them as one of five crime types: beating, shooting, stabbing, strangulation, and multiclass. This work has led to the development of a substantial crime dictionary that includes 70 criminal instruments and 151 words for terminology related to them, so that crime types, tools, and associated vocabulary are easily understood and can thus be utilized by other researchers in the future. The experimental test shows that the results obtained are promising. The random forest algorithm was adopted for the first classifier with an accuracy score of 91.07%, while the second classifier achieved an accuracy score of 82.46% with support vector machines.

As an initial exploration of text mining court documents in the field of crime scene investigation, our work has some limitations. One limitation of our research is the small size of the dataset for each class, which is also unbalanced. This issue can be addressed by increasing the number of classified documents used to train and test ML classifiers. Additional information from forensic reports can be incorporated to enhance the crime dictionary. Utilizing the dictionary in conjunction with the police database is anticipated to yield superior outcomes compared to the court database. We propose extracting large portions of the court database rather than simply categorizing it by crime type and focusing on the data that police students require.

Future work in this paper may investigate a variety of approaches to improve and extend the research findings. Our efforts can be expanded by implementing and testing this educational tool in police academies, as well as conducting research on its educational and practical implications. Deep learning techniques can also be used to analyze and classify these documents, allowing for a better understanding of the crimes in question. Another promising research area for improving analysis is predicting the exact crime instrument used by expanding the crime vocabulary with wound description terms from forensics. Finally, we suggest evaluating the system using real police reports and comparing the outcomes to verify the suitability of the crime dictionary for analyzing descriptive police reports.

Authors’ contributions

The authors confirm contribution to the paper as follows: study conception and design: E. Bifari and W. Alhalabi; data collection: E. Bifari; analysis and interpretation of results: E. Bifari and S. Albaradei; original draft manuscript preparation: E. Bifari; writing, review, and editing: E. Bifari, A. Basabrin, R. Mirza, A. Bafail and S. Albaradei. All authors reviewed the results and approved the final version of the manuscript.

Availability of data and materials

The dataset supporting the conclusions of this article is available in the [Harvard Law School Collection] repository, [https://case.law/about/].

Disclosure statement

The authors declare that they have no conflicts of interest to report regarding the present study.

Correction Statement

This article has been corrected with minor changes. These changes do not impact the academic content of the article.

Additional information

Funding

This research work was funded by Institutional Fund Projects under grant no. (IFPRC-119-611-2020). Therefore, the authors gratefully acknowledge technical and financial support from Ministry of Education and Deanship of Scientific Research (DSR), King Abdulaziz University (KAU), Jeddah, Saudi Arabia.

Notes on contributors

Ezdihar Bifari

Ezdihar Bifari, Department of Computer Science, King Abdulaziz University, Jeddah, Saudi Arabia, [email protected]

Arwa Basbrain

Arwa Mohmmed Basbrain, Department of Computer Science, King Abdulaziz University, Jeddah, Saudi Arabia, [email protected]

Rsha Mirza

Rsha Mirza, Department of Computer Science, King Abdulaziz University, Jeddah, Saudi Arabia, [email protected]

Alaa Bafail

Alaa Bafail, Artificial intelligence, Department of Computer Science, King Abdulaziz University, Jeddah, Saudi Arabia, [email protected]

Somayah Albaradei

Somayah Albaradei, Department of Computer Science, King Abdulaziz University, Jeddah, Saudi Arabia, [email protected]

Wadee Alhalabi

Wadee Alhalabi, Virtual Reality Research Group, Department of Computer Science, King Abdulaziz University, Jeddah, Saudi Arabia, [email protected]