Development and Evaluation of an Intelligent System for Calibrating Karaoke Lyrics Based on Fuzzy Petri Nets

ABSTRACT

In the home entertainment system, karaoke is a popular leisure facility in our daily life. Via the karaoke system, users can sing along with the lyrics based on the recordings of pop songs. However, a lot of karaoke systems can display lyrics semi-automatically. Traditionally, some lyrics are input manually and need to be synchronized with the tonal music stepwise, which is time-consuming. One of the famous musical phrase segmentation theories is a generative theory of tonal music, through which we have implemented a karaoke system in C# programming language. This intelligent system can automatically segment music phrases and use a high-level fuzzy Petri net model to calibrate the lyrics in pop songs. Fifty Chinese pop songs are selected to evaluate its performance. The experimental results have shown that the average calibration precision value (92.78%) and recall value (90.46%) are highly acceptable.

Introduction

In the home entertainment system, karaoke playing has become a popular activity, which is enjoyed by many people. Via a karaoke system, users can sing along with lyrics based on the recordings of pop songs, and receive a score based on their performances (Tsai and Lee 2017). When reading the karaoke lyrics on the screen, users can recognize how to adjust their singing intonation to improve their scores, which is really simple and interesting. However, a difficulty in the traditional karaoke lyrics to be calibrated manually is encountered. It takes a large amount of time to segment the musical phrases, which is quite inefficient. Karaoke songs have become more difficult to make and the manufacturing costs are increasing. Thus, the selling price of a karaoke system is harder to be accepted by users.

In recent years, with advances in computer technology, the music albums have gradually been replaced by digital music purchased online. Nowadays, researchers suggest that automatically calibrating karaoke lyrics may be possible. Firstly, the musical phrase segmentation is required. One of the famous music segmentation theories is a generative theory of tonal music (GTTM) (Chan and Hsiao 2016; Frankland and Cohen 2018), which was conceived by a music theorist, Fred Lerdahl, and a linguist, Ray Jackendoff; and presented in the book with the same title. Grouping preference rules (GPRs) are established in GTTM for the purpose of making a music grouping structure (Goto 2016). Quantification and testing of the local grouping rules for four local GPRs are provided (Frankland and Cohen 2018). In the approach based on the preference rules, the sound onset detection is applied to find all onsets in the song (Wu and Cao 2018).

The high-level fuzzy Petri net (HLFPN) model (Chiang et al. 2020; 2021; Lin et al. 2020a) is now used to perform the music segmentation. It is based on Petri net theory and fuzzy reasoning. Petri net theory, proposed by Dr Carl Adam Petri in 1962, is a graphical and mathematical modeling tool, which is concurrent, asynchronous, distributed, parallel, non-deterministic, stochastic, and so on. It can be used to model and analyze various systems (Sung and Chang 2019). However, along with the advances in information system, the description of Petri nets is becoming more and more complex with use of fuzzy production rules (Matsumoto and Hori 2019). In general, a fuzzy production rule can be used to describe the fuzzy relationship between antecedent and consequent.

Due to the above reasons and motivation, the goal of this paper is to make a karaoke system which can automatically calibrate karaoke songs with lyrics. A C# program has been designed to implement GTTM and HLFPN model for the purpose of segmenting each musical phrase. Musical phrases are used to automatically calibrate the lyrics that form a database system, and music notes are automatically calibrated for a Chinese character which has only one syllable. Then, an automatic calibration system for Chinese karaoke lyrics is completed.

Problem Statement: Although the karaoke technology has advanced with multi-functions for many years, it still lacks a robust database system for musical analysis. The grouping preference rules are diversified (Frankland and Cohen 2018). Also, the technical analysis parameters are hard to make musical decision. Consequently, the original semi-automatic lyrics calibration system needs to be further improved.

The remainder of this paper is organized as follows: Section 2 discusses the literature review of GTTM which is applied to fuzzy reasoning of HLFPN model. The system architecture and music segmentation are described in Section 3. The experimental results are discussed in Section 4. Finally, the conclusion is remarked in Section 5.

Literature Review

First, the background knowledge of a generative theory of tonal music (GTTM) rules and the musical context combination method are described. Second, some basic definitions regarding the HLFPN model, and the related works are presented.

Generative Theory of Tonal Music (GTTM)

The traditional method of musical phrase segmentation is based on psychological perspectives, and their relationship is discovered by psychology in music grouping. In 1920s, German psychologists proposed Gestalt theory to group similar objects. In Gestalt theory, they proposed the cognitive psychological aspects of musical groups, including three basic principles, namely, principle of proximity, principle of similarity, and principle of continuation.

In Gestalt theory, the research work in music grouping, composed of Jackendoff‘s and Lerdale’s ideas, proposed a generative theory of tonal music (GTTM) (Chan and Hsiao 2016; Frankland and Cohen 2018). GTTM constitutes a formal description of the musical intuitions of a listener who is experienced in a musical idiom. It includes the grouping preference rules (GPRs) defined as follows:

Rule 1: Single Event: “Avoid analyses with very small groups- the smaller, the less preferable.”

Rule 2: Proximity: “Consider a sequence of four notes, n1 – n4, the transition n2 – n3 may be heard as a group boundary if A. Slur/Rest: “the time interval from the end of n2 is greater than that from the end of n1 to the beginning of n2 and that from the end of n3 to the beginning of n4; or if B. Attack-Point: “the time interval between the attack points of n2 and n3 is greater than that between the attack points of n1 and n2 and that between the attack points of n3 and n4.”

Rule 3: Change: “Consider a sequence of four notes, n1 – n4, the transition n2 – n3 may be heard as a group boundary if marked by Register, Dynamics, Articulation, Length, and Timbre,” which are shown in Figure 1–5.

Figure 1. Register change.

Figure 2. Dynamics change.

Figure 3. Articulation change.

Figure 4. Length change.

Figure 5. Timbre change.

High-Level Fuzzy Petri Net

Based on the original Petri net theory, scholars conduct their research works with evolutionary Petri nets one after another, such as colored Petri nets (Zhang, Zhao, and He 2022), timed Petri nets (Lin et al., 2020b; Lin et al. 2022), fuzzy Petri nets (Chowdhury et al. 2019), high-level fuzzy Petri nets (Chiang et al. 2020; Lin et al. 2020a). The HLFPN model was adopted to make decision on the segmentation of music phrases. It provides the characters of Petri net and fuzzy logic theories, which can be used to model fuzzy production rules and to conduct fuzzy reasoning (Chiang et al. 2021). The basic definitions and fuzzy reasoning approach are presented as follows:

Definition 1. The HLFPN is defined as an 8-tuple:

$HLFPN=\left(P,T,F,C,V,\alpha ,\beta ,\delta \right)$

where:

$P=\left\{p_1,p_2,\dots ,p_k\right\}$: a finite set of places.

$T=\left\{t_1,t_2,\dots ,t_l\right\}$: a finite set of transitions.

$P\cup T=\mathrm{\varnothing }$

$F\subseteq \left(P\times T\right)\cup \left(T\times P\right)$: called the flow relation which is also a finite set of arcs, each representing the fuzzy set (i.e. fuzzy term) for an antecedent or a consequent; where the positive arcs (i.e. THEN parts) are denoted by →.

$C=\left\{X,Y,Z\right\}$: A finite set of linguistic variables, e.g. $X,Y$, and $Z$, where $X=\left\{x_1,x_2,\dots ,x_n\right\}$, $Y=\left\{y_1,y_2,\dots ,y_m\right\}$, $Z=\left\{z_1,z_2,\dots ,z_q\right\}$.

$V=\left\{v_1,v_2,\dots ,v_l\right\}$: A finite set of fuzzy truth values known as the fuzzy relational matrix between the antecedent and the consequent of a fuzzy production rule.

$\alpha :P\to C$: An association function, mapping from places to linguistic variables. $\alpha \left(p_i\right)=c_i,i=1,\dots ,I$, where $C=c_i$ is a set of linguistic variables in the knowledge base (KB) and is the number of linguistic variables in the KB;

$\beta :F\to \left\{0,1\right\}$: An association function, mapping from the flow relations to the fuzzy truth values between zero and one.

$\delta :T\to V$: An association function, mapping from transitions to fuzzy relational matrices.

Definition 2. Input and Output Functions:

$I\left(t\right)=\{p\in P|\left(p,t\right)\in F\}$: A set of input places of transition $t$.

$I\left(p\right)=\{t\in T|\left(t,p\right)\in F\}$: A set of input transitions of place $p$.

$O\left(t\right)=\{p\in P|\left(t,p\right)\in F\}$: A set of output places of transition $t$.

$O\left(p\right)=\{t\in T|\left(p,t\right)\in F\}$: A set of output transitions of place $p$.

Definition 3. Membership Function:

The mapping function $Mem\left(p\right):P\to \left[0,1\right]$ assigns each place a real value, where $Mem\left(p\right)=DOM\left(\alpha \left(p\right)\right)$, $DOM$ represents the degree of membership in the associated proposition and data tokens are available in a set of places $P$.

Definition 4. Max-Min Compositional Rule:

In the HLFPN model, $\mathrm{\forall }$ transition t, V(t) = min (fuzzy sets in I(t)); $\mathrm{\forall }$ place p, V(p) = max (fuzzy sets in I(p)). The Max-Min composition operator is denoted by ○.

Definition 5. Input Place, Hidden Place, and Output Place:

In the HLFPN model, $\mathrm{\forall }$ place p_i $\in$ P, if $\mathrm{\forall }$ t_j $\in$ T, p_i $\notin$ O(t_j), then p_i is called an input place (IP) of t_j. if $\mathrm{\forall }$ t_j $\in$ T, p_i $\notin$ I(t_j), then p_i is called an output place (OP) of t_j; otherwise, p_i is called a hidden place.

Fuzzy Reasoning

For the fuzzy reasoning, fuzzy production rules are used (Lin et al. 2020a). In general, a fuzzy production rule is used to describe fuzzy relationship between the antecedent and the consequent. Let R be a set of fuzzy production rules, where $R=\left\{R_1,R_2,\dots ,R_n\right\}$. The general form of the i-th fuzzy production rule $R_i$ is shown as follows:

$R_i:IF{d}_j\left(XisA\right),THEN{d}_k\left(YisB\right);ELSE{d}_w\left(ZisC\right)\dots \left(V\right)$

where $d_j,d_k,\mathrm{a}\mathrm{n}\mathrm{d}{d}_w$ denote propositions; $X$ is called the input linguistic variable; $Y$ and $Z$ are called the output linguistic variables, respectively; $A$ is called the input fuzzy set; $B$ and $C$ are called the output fuzzy sets, respectively; the fuzzy truth values of the propositions “$X$ is $A$”, “$Y$ is $B$” and “$Z$ is $C$” are restricted to [0, 1]; $d_j$ is the antecedent of a fuzzy production rule R_i; d_k and d_w denote the consequents of the fuzzy production rule $R_i$. Let $V$ represent the fuzzy relational matrix between antecedent and consequent of a fuzzy production rule.

Illustrative Example:

Let us consider the fuzzy production rule $R_1$ shown as follows:

$R_1:IF\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{t}\mathrm{e}\mathrm{m}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e}\left(X_1\right)is\mathrm{h}\mathrm{o}\mathrm{t}\left(A_1\right)AND\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{s}\mathrm{k}\mathrm{y}\left(X_2\right)is\mathrm{c}\mathrm{l}\mathrm{o}\mathrm{u}\mathrm{d}\mathrm{y}\left(A_2\right),$ $THEN\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{h}\mathrm{u}\mathrm{m}\mathrm{i}\mathrm{d}\mathrm{i}\mathrm{t}\mathrm{y}\left(Y\right)is\mathrm{h}\mathrm{i}\mathrm{g}\mathrm{h}\left(B\right).$

Based on the transformation procedure presented in (Casey et al. 2018), the above fuzzy production rule $R_1$ is transformed into the following first-order logic form:

$R_1^{\prime }:IF{X}_1\left(A_1\right)AND{X}_2\left(A_2\right),THENY\left(B\right)$

Then, the HLFPN model is shown in Figure 6.

Figure 6. HLFPN model of illustrative example.

Assume that the fuzzy sets $A_1$, $A_2$ and $B$ are shown as follows:

$A_1=\frac{0.20}{a_{11}}+\frac{0.54}{a_{12}}+\frac{0.24}{a_{13}}$

$A_2=\frac{0.11}{a_{21}}+\frac{0.77}{a_{22}}+\frac{0.42}{a_{23}}$

$B=\frac{0.31}{b_1}+\frac{0.66}{b_2}+\frac{0.17}{b_3}$

By the cylindrical extension operations (Lin et al. 2020a), that is, a Cartesian product, we can obtain the antecedent fuzzy set A, shown as follows:

$A=A_1\times A2=(0.200.540.24{)}^{T_{\wedge }}\left(0.110.770.42\right)=\left[\begin{array}{ccc}0.11& 0.20& 0.20\\ 0.11& 0.54& 0.42\\ 0.11& 0.24& 0.24\end{array}\right]$

Then, the fuzzy relational matrices $V_1\left(t_1\right),V_2\left(t_2\right),\mathrm{a}\mathrm{n}\mathrm{d}{V}_3\left(t_3\right)$ between antecedent and consequent of the fuzzy production rule $R_1$ can be obtained, shown as follows:

$V_1\left(t_1\right)=\left[\begin{array}{ccc}0.11& 0.20& 0.20\\ 0.11& 0.31& 0.31\\ 0.11& 0.24& 0.24\end{array}\right]\in A\times B\times b_1$

$V_2\left(t_2\right)=\left[\begin{array}{ccc}0.11& 0.20& 0.20\\ 0.11& 0.54& 0.42\\ 0.11& 0.24& 0.24\end{array}\right]\in A\times B\times b_2$

$V_3\left(t_3\right)=\left[\begin{array}{ccc}0.11& 0.17& 0.17\\ 0.11& 0.17& 0.17\\ 0.11& 0.17& 0.17\end{array}\right]\in A\times B\times b_3$

The most widely used fuzzy reasoning method is the max–min composition inference (Yang et al. 2019). Assume that the input fuzzy sets ${A_1}^{\prime }$ and ${A_2}^{\prime }$ are shown as follows:

${A_1}^{\prime }=\frac{0.09}{a_{11}}+\frac{0.85}{a_{12}}+\frac{0.29}{a_{13}}$

${A_2}^{\prime }=\frac{0.29}{a_{21}}+\frac{0.89}{a_{22}}+\frac{0.45}{a_{23}}$

Then, we can get

${A_1}^{\prime }\circ V_1\left(t_1\right)=\left(0.090.850.29\right)\circ V_1\left(t_1\right)=\left(0.110.310.31\right)$

${A_1}^{\prime }\circ V_2\left(t_1\right)=\left(0.090.850.29\right)\circ V_2\left(t_1\right)=\left(0.110.540.42\right)$

${A_1}^{\prime }\circ V_3\left(t_1\right)=\left(0.090.850.29\right)\circ V_3\left(t_1\right)=\left(0.110.170.17\right)$

Finally, we can obtain

$B{ }^{\mathrm{\prime }}=\left(0.290.890.45\right)\circ \left[\begin{array}{ccc}0.11& 0.11& 0.11\\ 0.31& 0.54& 0.17\\ 0.31& 0.42& 0.17\end{array}\right]$

$=\left(0.310.540.17\right)$

$=\frac{0.31}{b_1}+\frac{0.54}{b_2}+\frac{0.17}{b_3}$

The above description is the fuzzy reasoning process of HLFPN model.

Fuzzy Reasoning Algorithm

In this sub-section, a fuzzy reasoning algorithm (FRA) (Chiang et al. 2020) is briefly reviewed to determine whether there exists a fuzzy relational matrix between antecedent and consequent of a fuzzy production rule or not.

INPUT: Input Place (IP), $Mem\left(p_i\right)\hspace{0.17em}\mathrm{\forall }{p}_i\in IP$, (or fuzzy set), where $IP$ denotes a set of input places.

OUTPUT: Output Place (OP), $Mem\left(p_i\right)\hspace{0.17em}\mathrm{\forall }{p}_i\in OP$, (or fuzzy set), where $OP$ denotes a set of output places.

Procedure

Step 1:

Initially, assume that only the Degree of Memberships (DOMs) in the propositions operating on input variables are available. Consequently, the initial marking function is shown as follows:

$M\left(p_i\right)=0,\mathrm{i}\mathrm{f}{p}_i\notin IP$

$M\left(p_i\right)=\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{d}\mathrm{a}\mathrm{t}\mathrm{a}\mathrm{t}\mathrm{o}\mathrm{k}\mathrm{e}\mathrm{n}\mathrm{s},\mathrm{i}\mathrm{f}{p}_i\in IP$

Step 2:

Calculate the fuzzy relational matrices V($t_i$) of the current transition, and use the input to perform cylindrical extension.

$\mathrm{\forall }{t}_j\in T,$ compute $V\left(t_j\right)=W_a\times W_c=(w_{a1},w_{a2},\dots ,w_{am}{)}^T\wedge \left(w_{c1},w_{c2},\dots ,w_{cn}\right),$ where $T$ denotes a set of transitions; $V\left(t_j\right)$ is a fuzzy relational matrix between the antecedent and the consequent of rule $t_j$; $W_a=\left(w_{a1},w_{a2},\dots ,w_{am}\right)$ is a fuzzy set of weights in the antecedent; $W_c=\left(w_{c1},w_{c2},\dots ,w_{cn}\right)$ is a fuzzy set of weights in the consequent; and each element of a fuzzy set is denoted as a fuzzy weight interval.

Step 3:

Input a data pattern to $W_a$-input.

Step 4:

Fire the enabled transitions and execute Zadeh’s max-min operation. Let $t_j$ be any enabled transition. Then, compute $t_j\in T/\mathrm{\forall }{p}_k,M\left(p_k\right)$ = the number of data tokens

$\hspace{0.17em}{W}_a^{\prime }$ = $W_a$-input

$W_c^{\prime }=W_a^{\prime }{\ast }_{\circ }V\left(t_j\right)or\mathrm{\neg }{W}_a^{\prime }{\ast }_{\circ }V\left(t_j\right)$

if an ELSE part is available.

Step 5:

Send the results to the output place. For every output variable $O$, its associated membership distribution is ${W_c}^{\prime }=\left\{{w_{ci}}^{\prime }\right\}=\vee {w_{ci}}^{\prime },i=1,2,\dots ,l$, where $l$ denotes the in-degree of output variable $O$. Then, $W_c^{\prime }$ becomes an actual output.

Step 6

Go back to Step 4, while $\mathrm{\exists }{t}_j\in T/M\left(p_i\right)=1,\mathrm{\forall }{p}_i\in IP\left(t_j\right)$ (That is, while the enabled transitions still exist, go to Step 4).

Step 7

The weighted average defuzzification method is applied, and the real operation value is obtained.

Ieee 1599

“Music is much more than listening to audio encoded in some unreadable binary format” (Wick, Hartelt, and Puppe 2020). So, we need to integrate each karaoke song, score, information, and musical phrase. The standard is named as IEEE1599 (Baggi 2015), proposed by Denis L. Baggi and Goffredo M. Haus in 2009. IEEE 1599 is a standard to encode music with XML symbols. The main distinguished characteristics use blocks to represent music and the concept of layers, as shown in Figure 7.

Figure 7. All layers in IEEE 1599 (Wick, Hartelt, and Puppe 2020).

General Layer: The layer provides a general description of the music. It contains the following items: music title, authors, number, data, type, and so on. Structural Layer: The layer stores information of musical structure. Logical Layer: The layer provides music description from a symbolic point and includes scores with symbols, as well as the spine with logically organized symbols, and a sorted list of music events. Notational Layer: The layer includes notations of music, like a great variety of pictures. Performance Layer: The layer contains the information about musical performance, including midi, Csound, etc. Audio Layer: The audio layers including media contents are still available in their original encoding.

Related Works

According to reference (Meier 2013; Wu and Cao 2018), for the problem of song sentiment classification based on Chinese lyrics, there are two methods that have been widely used at present, namely, a sentiment-dictionary-based method and a machine-learning-based method. The former sentiment dictionary is difficult to expand, while the latter may cause “dimensional disaster” when generating feature vectors. It proposes a song sentiment classification method that combines the advantages of these two ideas. The method obtains the positive and negative tendency values by the process of word segmentation and weight calculation based on sentiment dictionary, which are used as feature vectors. Then, we use Logistic Regression algorithm to carry binary classification of Chinese songs. According to reference (Shen and Chen 2014), it was the initial idea to develop this lyrics calibration system, which had lower precision value (71.43%) and no recall values available. More useful parameters and detailed discussions are needed to improve its system performance. The fuzzy reasoning algorithm also needs to be further improved. According to reference (Bilbao et al. 2020; Ruangsang and Assawinchaichote 2019), as the computational costs for physical modeling synthesis of sound are often much greater than those for conventional sound synthesis methods, most techniques currently rely on simplifying assumptions. They include digital waveguides and modal synthesis methods. However, it can be difficult to approach some of the more detailed behavior of musical instruments.

Proposed Approach

For a karaoke system, it is important to make musical phrases, which are concepts and practices related to grouping consecutive melodic notes in performance and composition. A musical phrase is a unit that has a complete musical sense of its own length, in which a singer or instrumentalist can sing or play in one breath. We have developed a C# program to implement GTTM and HLFPN model to segment musical phrases. Musical phrases are used to automatically calibrate the lyrics that form a database system. And the music notes are automatically calibrated for a Chinese character, which has only one syllable.

Data Preprocessing System

A phrase segmentation technique is highly required because a phrase is the basic unit for organizing and analyzing the music signals. In music scores, it is usually marking a phrase with a slur, as shown in Figure 8.

Figure 8. A phrase.

For phrase segmentation, as rule 3 in grouping preference, “Consider a sequence of four notes, n₁ – n₄, the transition n₂ – n₃ may be heard as a group boundary if marked by register, dynamics and length.” (Haus and Ludovico 2017; Haus, Longari 2019; Hu, Chen, and Yin 2020), the score function (Liu 2010) is defined for the grouping preference rule 3 of GTTM as follows:

The register change rule is assumed that the pitches (in cent) of the four music notes n₁n₂n₃n₄ are c₁c₂c₃c₄. If |c₃-c₂| > |c₂-c₁| and |c₃-c₂| > |c₄-c₃|, then it is defined as:

(1) $f_{register}=\left\{1-\frac{\left|c_2-c_1\right|+\left|c_4-c_3\right|}{2\left|c_3-c_2\right|}\right\}$(1)

else, defined as $f_{register}=0$. It represents that the pitch has not been changed.

Similarly, the dynamics change rule is assumed that the intensities (in dB) of the four music notes n₁n₂n₃n₄ are d₁d₂d₃d₄. If |d₃-d₂| > |d₂-d₁| and |d₃-d₂| > |d₄-d₃|, then it is defined as

(2) $f_{dynamics}=\left\{1-\frac{\left|d_2-d_1\right|+\left|d_4-d_3\right|}{2\left|d_3-d_2\right|}\right\}$(2)

else, defined as $f_{dynamics}=0$. It represents the intensity not being changed.

Also, the length change rule is assumed that the lengths (in second) of the four music notes n₁n₂n₃n₄ are l₁l₂l₃l₄ defined as:

(3) $f_{length}=\left\{1-\frac{Min\left(l_2,l_3\right)}{Max\left(l_2,l_3\right)}\right\}$(3)

First, the membership function can be defined in accordance with the input parameters. The fuzzy reasoning rule and the corresponding HLFPN model can be established. The $f_{register}$, $f_{dynamics}$ and $f_{length}$ associated with the membership functions for the fuzzifier are input into the HLFPN model. With the characters of fuzzy reasoning, each input parameter is reasoned by the fuzzy rule. Finally, all indices are integrated to make decision on the musical phrase segmentation and to perform automatic calibration of karaoke lyrics (Bastanfard, Amirkhani, and Naderi 2020; Chen 2021).

HLFPN Model Based on GTTM

To make more precise in the musical phrase segmentation, it is a good way to implement grouping preference rules of GTTM. The decision based on input parameters uses the calculated values to set the position of segmentation. However, an input parameter needs to be considered as a fuzzy term, high or low. Grouping preference rules of GTTM including three parameters are used to determine where the position of performance or segmentation is. In these unclear, obscure, and ambiguous cases, it is appropriate to use the fuzzy logic theory. Thus, the HLFPN model is applied to make decision (Shen et al. 2017), as shown in Figure 9.

Figure 9. Block diagram of HLFPN model.

Calibration Mechanism in Membership Degrees

According to the corresponding membership functions and fuzzy sets defined, we input each parameter to a fuzzifier and calculate the membership degrees in each set. Based on the size of the input parameter, each parameter is transformed to 「If … then … 」statement and the fuzzy rules are established. In the decision method, three parameters are used, namely, the $f_{register}$, $f_{dynamics}$ and $f_{length}$. As a result, each index is defined in membership degrees of Low and High. Then, three sets of technical indices include the S-Type and Z-Type membership degrees, as shown in Figure 10.

Figure 10. The type of membership degrees for input parameter.

In addition, the decision is also divided into two cases, namely, “Performance” and “Segmentation.” Therefore, two sets of performance and segmentation belong to the Λ-Type of membership degree, as shown in Figure 11.

Figure 11. The type of membership degrees for technical decision.

In this paper, three input parameters are used, namely, the $f_{register}$, $f_{dynamics}$ and $f_{length}$. As a result, the membership functions of “high” and “low” are defined for each input parameter. Therefore, all of them are defined in Table 1. The membership degrees of the decision on music performance and segmentation are listed in Table 2.

Table 1. Membership degrees of input parameter.

Input Parameter	Low		High
$f_{register}$	0.2	0.7	0.3	0.8
$f_{dynamics}$	0.2	0.7	0.3	0.8
$f_{length}$	0.2	0.7	0.3	0.8

Table 2. Membership degrees of technical decision.

Performance			Segmentation
0.0	0.2	0.4	0.3	0.5	0.7

In the analysis, the values of each input parameter are usually set between 0 and 1. When $f_{register}$,$f_{dynamics}$, and ${ }_{length}$ values larger than 0.3 are high, and those values less than 0.7 are low. However, due to the membership degree as defined above, whose scope of a variable is set between 0 and 1. Thus, the values of each input parameter must be converted to the values between 0 and 1 before they are substituted into the calculation of basic rules.

Assume that ${\nu }_H$ and ${\nu }_L$ values represent membership degrees of “high” and “low,” respectively. The fuzzification procedure is defined as follows:

(4) $v_H\left(x\right)=\{\begin{array}{c}1,\hspace{0.17em}x\ge 0.8\\ 2\left(x-0.4\right),0.3<x<0.8\hspace{0.17em}\\ 0,\hspace{0.17em}x\le 0.3\end{array}$(4)

(5) $v_L\left(x\right)=\{\begin{array}{c}0,\hspace{0.17em}x\le 0.2\\ 2\left(x-0.1\right),0.7>x>0.2\hspace{0.17em}\\ 1,\hspace{0.17em}x\ge 0.7\end{array}$(5)

(6) $v_{Segmentation}\left(x\right)=\left\{\begin{array}{c}0,\hspace{0.17em}x\ge 0.7\\ \begin{array}{c}-\frac{1}{0.3}\left(x-1.0\right),0.5\le x<0.7\\ \frac{1}{0.3}\left(x-0.4\right),\hspace{0.17em}0.3<x<0.5\hspace{0.17em}\end{array}\\ 0,\hspace{0.17em}x\le 0.3\end{array}\right.$(6)

(7) $v_{Performance}\left(x\right)=\left\{\begin{array}{c}0,\hspace{0.17em}x\ge 0.4\\ \begin{array}{c}-5\left(x-0.5\right),0.2\le x<0.4\\ 5\left(x-0.0\right),\hspace{0.17em}0.0<x<0.2\hspace{0.17em}\end{array}\\ 0,\hspace{0.17em}x\le 0.0\end{array}\right.$(7)

Fuzzy Reasoning and Building HLFPN Model

According to fuzzy sets and their corresponding membership degrees as defined previously, each input parameter is imported to a fuzzifier and the technical index is calculated to find its own membership degree. Based on the size of an input parameter, it has been transformed into an「If … then … 」statement in order to establish a rule base. The activity diagram of fuzzy reasoning is shown in Figure 12.

Figure 12. The activity diagram of fuzzy reasoning.

Assume that $f_{register}$,$f_{dynamics}$, and $f_{length}$ denote input linguistic variables with fuzzy terms: high (H) and low (L). And assume that the decision (D) is an output linguistic variable with fuzzy terms: strong (S) and weak (W). The fuzzy production rules are defined as follows:

R1: If $f_{register}$ is H, then D is S.

R2: If $f_{register}$ is L, then D is W.

R3: If $f_{dynamics}$ is H, then D is S.

R4: If $f_{dynamc}$ is L, then D is W.

R5: If $f_{length}$ is H, then D is S.

R6: If $f_{length}$ is L, then D is W.

Then, the fuzzy production rules are transformed into the HLFPN model, as shown in Figure 13, and the parameters are listed in Table 3.

Figure 13. The HLFPN model representing six fuzzy production rules.

Table 3. Description of parameters.

Name of Parameter	Description of Parameter
$f_{register}$	Denotes the value of $f_{register}$ in input place P1,
$f_{dynamics}$	Denotes the value of $f_{dynamics}$ in input place P2.
$f_{length}$	Denotes the value of $f_{length}$ in input place P3.
D	Denotes a decision in an output place P4.
H, L	Denote high and low fuzzy sets, respectively.
S, W	Denote strong and weak fuzzy sets, respectively.
$V(ti),i=1,2,3,\dots ,6$	Denotes the fuzzy relational matrix of $f_{register}$, $f_{dynamics}$, $f_{length}$ and decision.
H’, L’	Denote high and low fuzzy sets of input value.

According to the proposed HLFPN model, the fuzzy inference is performed with the technical indices of a fuzzier and the fuzzy rules in the rule base. Within the process of fuzzy inference, the standard operators for calculations are used. Finally, the crisp outputs are obtained to decide performance or segmentation by the weighted average defuzzification method, and the real operation value of “Decision” can be computed.

Preprocessing Results

However, the music lyrics, score, and other music information are integrated into an automatic calibration system. A new music standard named IEEE 1599 and a software system are used to show the preprocessing results. The IEEE 1599 standard (Baggi 2015) is used to encode music with XML symbols. This is an example in IEEE 1599 from a song named “Silence,” composed by Jay Chou, as shown in Figures 14–17.

Figure 14. Layers in IEEE 1599.

Figure 15. Description in general layer of IEEE 1599.

Figure 16. The related files in general layer of IEEE 1599.

Figure 17. Layers in IEEE 1599.

Experimental Results

To demonstrate that the proposed HLFPN-based calibration system is feasible, this Section is intended to evaluate the proposed system performance. A fuzzy reasoning algorithm (FRA) (Chiang et al. 2020) is used to determine whether a fuzzy relational matrix exists between antecedent and consequent of a fuzzy rule. Then, our proposed approach is applied to the illustrative example.

Example of HLFPN Model for Fuzzy Reasoning

In this sub-section, an example is used to illustrate the viability of the fuzzy reasoning process. Two digits of significant numbers represent the result of each calculation.

Step 1: Initially, assume that only the DOMs in the propositions operating input variables are available. Assume that four fuzzy sets are shown as follows:

$H=\frac{0.00}{h_L}+\frac{0.70}{h_H}$,$L=\frac{0.30}{l_L}+\frac{0.00}{l_H}$, $S=\frac{0.00}{s_L}+\frac{0.70}{s_H}$,$W=\frac{0.30}{w_L}+\frac{0.00}{w_H}$

Step 2: Compute the fuzzy relational matrices, shown as follows:

$V\left(t_1\right)=H\ast S=(0.000.70{)}^T\wedge \left(0.000.70\right)=\left[\begin{array}{cc}0.00& 0.00\\ 0.00& 0.70\end{array}\right]$

$V\left(t_3\right)=V\left(t_1\right)=\left[\begin{array}{cc}0.00& 0.00\\ 0.00& 0.70\end{array}\right]$

$V\left(t_5\right)=V\left(t_1\right)=\left[\begin{array}{cc}0.00& 0.00\\ 0.00& 0.70\end{array}\right]$

$V\left(t_2\right)=L\ast W={\left(0.300.00\right)}^T\wedge \left(0.300.00\right)=\left[\begin{array}{cc}0.30& 0.00\\ 0.00& 0.00\end{array}\right]$

$V\left(t_4\right)=V\left(t_2\right)=\left[\begin{array}{cc}0.30& 0.00\\ 0.00& 0.00\end{array}\right]$

$V\left(t_6\right)=V\left(t_2\right)=\left[\begin{array}{cc}0.30& 0.00\\ 0.00& 0.00\end{array}\right]$

Step 3: Input data pattern, shown as follows:

$H_{fregister}^{\prime }=\frac{0.00}{h_L^{\prime }}+\frac{0.20}{h_H^{\prime }}$

$L_{fregister}^{\prime }=\frac{0.80}{l_L^{\prime }}+\frac{0.00}{l_H^{\prime }}$

$H_{fdymanics}^{\prime }=\frac{0.00}{h_L^{\prime }}+\frac{0.80}{h_H^{\prime }}$

$L_{fdymanics}^{\prime }=\frac{0.00}{l_L^{\prime }}+\frac{0.00}{l_H^{\prime }}$

$H_{flength}^{\prime }=\frac{0.00}{h_L^{\prime }}+\frac{0.40}{h_H^{\prime }}$

$L_{flength}^{\prime }=\frac{0.20}{l_L^{\prime }}+\frac{0.00}{l_H^{\prime }}$

Step 4: Fire the enabled transitions:

$S_{fregister}^{\prime }=H_{fregister}^{\prime }\circ V\left(t_1\right)=\left[\begin{array}{cc}0.00& 0.20\end{array}\right]\circ \left[\begin{array}{cc}0.00& 0.00\\ 0.00& 0.70\end{array}\right]=\left[\begin{array}{cc}0.00& 0.20\end{array}\right]$

$W_{fregister}^{\prime }=L_{fregister}^{\prime }\circ V\left(t_2\right)=\left[\begin{array}{cc}0.80& 0.00\end{array}\right]\circ \left[\begin{array}{cc}0.30& 0.00\\ 0.00& 0.00\end{array}\right]=\left[\begin{array}{cc}0.30& 0.00\end{array}\right]$

$S_{fdynamics}^{\prime }=H_{fdynamics}^{\prime }\circ V\left(t_3\right)=\left[\begin{array}{cc}0.00& 0.80\end{array}\right]\circ \left[\begin{array}{cc}0.00& 0.00\\ 0.00& 0.70\end{array}\right]=\left[\begin{array}{cc}0.00& 0.70\end{array}\right]$

$W_{fdynamics}^{\prime }=L_{fdynamics}^{\prime }\circ V\left(t_4\right)=\left[\begin{array}{cc}0.00& 0.00\end{array}\right]\circ \left[\begin{array}{cc}0.30& 0.00\\ 0.00& 0.00\end{array}\right]=\left[\begin{array}{cc}0.00& 0.00\end{array}\right]$

$S_{flength}^{\prime }=H_{flength}^{\prime }\circ V\left(t_5\right)=\left[\begin{array}{cc}0.00& 0.40\end{array}\right]\circ \left[\begin{array}{cc}0.00& 0.00\\ 0.00& 0.70\end{array}\right]=\left[\begin{array}{cc}0.00& 0.40\end{array}\right]$

$W_{flength}^{\prime }=L_{flength}^{\prime }\circ V\left(t_6\right)=\left[\begin{array}{cc}0.20& 0.00\end{array}\right]\circ \left[\begin{array}{cc}0.30& 0.00\\ 0.00& 0.00\end{array}\right]=\left[\begin{array}{cc}0.20& 0.00\end{array}\right]$

Steps 5–6: Finally, the fuzzy reasoning result is shown as follows:

$D={S_{fregister}^{\prime }}^{\hspace{0.17em}}\cup {W_{\mathrm{f}\mathrm{r}\mathrm{e}\mathrm{g}\mathrm{i}\mathrm{s}\mathrm{t}\mathrm{e}\mathrm{r}}^{\prime }}^{\hspace{0.17em}}\cup S_{\mathrm{f}\mathrm{d}\mathrm{y}\mathrm{n}\mathrm{a}\mathrm{m}\mathrm{i}\mathrm{c}\mathrm{s}}^{\prime }\cup W_{\mathrm{f}\mathrm{d}\mathrm{y}\mathrm{n}\mathrm{a}\mathrm{m}\mathrm{i}\mathrm{c}\mathrm{s}}^{\prime }\cup S_{\mathrm{f}\mathrm{l}\mathrm{e}\mathrm{n}\mathrm{g}\mathrm{t}\mathrm{h}}^{\prime }\cup W_{\mathrm{f}\mathrm{l}\mathrm{e}\mathrm{n}\mathrm{g}\mathrm{t}\mathrm{h}}^{\prime }$

$\hspace{0.17em}=\frac{0.30}{d_L}+\frac{0.70}{d_H}$

Step 7: If the weighted average defuzzification method is applied, then the real operation value of “Decision” can be computed, shown as follows:

$decision=\frac{0.30\times d_L+0.70\times d_H}{0.30+0.70}=\frac{0.30\times 0.30+0.70\times 0.70}{0.30+0.70}=0.58$

Due to $0.3\le decision<0.7$, the decision is made as “Segmentation.”

Main Results

The experimental results illustrate the effectiveness of the proposed system. An automatic calibration system for karaoke lyrics is useful, but its musical phrase segmentation may have a little deviation. The comparison results are shown in Figures 18 and 19.

Figure 18. Deviation in automated system.

Figure 19. The actual score.

According to manual experiments, the automatic music segmentations which are measured by the recall and the precision values defined as follows:

(11) $recall=\left\{\frac{numberofphrasescorrectlydetected}{numberofphrases}\right\}\times 100\mathrm{\%}$(11)

(12) $precision=\left\{\frac{numberofphrasescorrectlydetected}{numberofphrasesdetected}\right\}\times 100\mathrm{\%}$(12)

Finally, in the experiment, 50 Chinese pop songs were used. In total, there are 2076 musical phrases manually labeled. As shown in Table 4, there are 1463 musical segmentations that are correctly detected by the proposed system. Therefore, the average recall is 90.46% and the average precision is 92.78%.

Table 4. Experimental results.

No.	Number of Phrases	Number of Phrases Correctly Detected	Number of Phrases Deviated	Number of Phrases Detected	Recall (%)	Precision (%)
1	43	39	3	42	90.70	92.86
2	46	39	4	43	84.78	90.70
3	49	45	3	48	91.84	93.75
4	48	42	3	45	87.50	93.33
5	22	19	2	21	86.36	90.48
6	34	31	2	33	91.18	93.94
7	27	25	2	27	92.59	92.59
8	34	31	3	34	91.18	91.18
9	49	45	3	48	91.84	93.75
10	44	40	3	43	90.91	93.02
11	24	22	1	23	91.67	95.65
12	40	35	3	38	87.50	92.11
13	39	36	2	38	92.31	94.74
14	27	24	2	26	88.89	92.31
15	45	39	3	42	86.67	92.86
16	43	40	2	42	93.02	95.24
17	48	43	3	46	89.58	93.48
18	36	33	2	35	91.67	94.29
19	39	35	3	38	89.74	92.11
20	32	29	2	31	90.63	93.55
21	48	44	2	46	91.67	95.65
22	44	39	4	43	88.64	90.70
23	42	38	3	41	90.48	92.68
24	43	39	3	42	90.70	92.86
25	53	47	5	52	88.68	90.38
26	36	33	2	35	91.67	94.29
27	49	45	3	48	91.84	93.75
28	31	28	2	30	90.32	93.33
29	35	32	2	34	91.43	94.12
30	53	47	4	51	88.68	92.16
31	60	54	4	58	90.00	93.10
32	44	40	3	43	90.91	93.02
33	36	32	3	35	88.89	91.43
34	59	53	4	57	89.83	92.98
35	45	41	3	44	91.11	93.18
36	38	34	3	37	89.47	91.89
37	40	36	3	39	90.00	92.31
38	42	38	3	41	90.48	92.68
39	46	42	3	45	91.30	93.33
40	50	45	4	49	90.00	91.84
41	61	56	4	60	91.80	93.33
42	46	42	4	46	91.30	91.30
43	47	43	3	46	91.49	93.48
44	22	20	2	22	90.91	90.91
45	40	36	3	39	90.00	92.31
46	37	33	3	36	89.19	91.67
47	32	32	0	32	100.0	100.0
48	36	33	2	35	91.67	94.29
49	58	52	4	56	89.66	92.86
50	34	30	2	32	88.24	93.75
Average					90.46	92.78

Functional Comparison

Based on the above experimental results, we have made a functional comparison with each other among different approaches, including ours, Wu’s (Wu and Cao 2018), and Bilbao’s (S. Bilbao et al. 2020). The results of functional comparison are listed in Table 5. In summary, our proposed approach is more feasible and powerful than others.

Table 5. Results of functional comparison.

Functions Approaches	Correct Phrase Detection	Deviated Phrase Detection	Missing Phrase Detection	Recall (%)	Precision (%)
Our Proposed	∨	∨	∨	90.46	92.78
X. Wu’s et al	∨	∨			∨
S. Bilbao’s et al	∨	∨			∨

“ˇ” denotes “Yes” or “Available.”

Conclusion

In this paper, those datasets including karaoke lyrics, songs, titles, and author names downloaded from the Internet have been used, and a database system of Chinese pop songs for automatic calibration has been built. The grouping preference rules of GTTM are adopted to perform the parameter analysis as a basis for determination. The appropriate decisions are made based on the above parameters by applying the reasoning algorithm of HLFPN model. A C# program has been developed to implement this automatic calibration system. Users can use this system to sing a pop song from which the singer’s voices are removed, and the lyrics can be calibrated automatically. The contributions of this paper are stated as follows:

1. A robust database system with Chinese pop songs has been successfully built, providing a useful environment for musical analysis.

2. The grouping preference rules of GTTM are easily used to group each song.

3. The parameters obtained from the register change rule, the dynamics change rule, and the length change rule are all used to easily conduct technical analysis for musical decision.

4. Through the fuzzy reasoning algorithm of HLFPN model, a fast decision can be made on music performance or segmentation.

5. This system can provide users with an easy way to sing a pop song and to get the lyrics automatically.

In the future, the learning algorithms of the HLPFN model will be adopted to accurately detect music segmentations. Also, a specific tool of HLFPN model will be created to predict the music segmentation and to make a difference between HLFPN model and GTTM. Since this system is now designed for Chinese karaoke songs only, it has been limited to some users. Therefore, it will be enhanced for English songs or songs in other languages to provide more applications.

Acknowledgments

The authors are grateful to the anonymous reviewers for their constructive comments, which have improved the quality of this paper.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the Ministry of Science and Technology, Taiwan MOST 107-2221-E-845- 001-MY3, ROC, under grants MOST 110-2637-E-131-005- and MOST 110-2221-E-845-002-.