5 Lanes 5 Fingers Portable Touch-Typing Interface “ParoTone” for Efficient Sensorimotor Learning: Musical Performance Acquisition Much Faster than Piano

Abstract

Being able to play a musical instrument requires complicated sensorimotor learning. Many beginners give up practicing Piano or Guitar because they require a long time of practice to overcome difficulties. It is not possible to practice and perform Piano or Guitar anytime and anywhere. We fundamentally rethought how the device and software should be, and tested the hypothesis that a new interface “ParoTone” with music game style notation, one-to-one correspondence between 5 lanes and 5 fingers with octave shifts, playing chords with one button can balance the three elements: easiness to learn, a wide range of expression, and portability. We found that compared to Piano, ParoTone is portable and enables beginners to learn ∼3 times faster for melody, five times faster for chords. The ParoTone interface in the future may be useful not only for music but in other fields to learn to operate multiple commands in a timely manner.

1. Introduction

Being able to play a musical instrument requires complicated sensorimotor learning, and it is one of the ways of self-expression and communication. Playing a musical instrument is useful for stress relief (Toyoshima et al., 2011), development of children’s cognitive ability (Moreno, 2009; Moreno et al., 2009; Thompson et al., 2004), and music therapy for autism (Reschke-Hernández, 2011) and dementia (Moreno-Morales et al., 2020).

To enjoy playing musical instruments, such as Piano and Guitar, we need to practice for a long time to overcome the difficulties. Many beginners want to play musical instruments to relieve stress, but they give up practicing during learning because the practice is stressful. Among 800 people who have experience of playing musical instruments outside of education, 87.7% want to play musical instruments, but 76.6% quitted playing the instruments. 34.4% among the 76.6% answered that the main reason they quitted playing the instruments is they couldn’t improve their skills (IID, Inc., 2013). Music games or rhythm games are easier ways to get started, but playing a music game does not directly lead to playing musical instruments that allow them to play melody and chords as they want with their own hands.

Difficulties in reading sheet music and controlling fingers prevent beginners from enjoying the practice. Playing a keyboard instrument is a kind of complicated sensorimotor transformation learning consisting of a series of processing from cognition through action to getting feedback (Emond & Comeau, 2013; Nakahira & Kitajima, 2017; Wolf, 1976). We diagrammed the series of processing consisting of four steps (Figure 1):

Figure 1. Four kinds of difficulties of sensorimotor transformation in playing keyboard instrument.

• [Step 1] Deciding which key to be pressed by reading sheet music

• [Step 2] Grasping the current finger positions

• [Step 3] Generating a motor command

• [Step 4] Getting feedback on whether you pressed the correct keyboard position

Difficulty at each step disturbs the learning process. Regarding [Step 1], reading sheet music (e.g., staff notation) requires a high cognitive load on learning how to read. The complexity of the visual information is too high for beginners. The cost of learning the correspondence between sheet music and keyboard position is high. As for [Step 2], touch-typing requires grasping the position of fingers only by tactile presentation without visual clues. In music, we do not “type,” but this time we purposefully chose the term “touch-type” rather than “sight-read” to emphasize playing by touch without hand-eye coordination. Since there are too many keys on Piano, the cognitive load to find the key position to press is high (Ohsawa et al., 2017). Since vision is used not only for [Step 1] but also for [Step 2], competition (cannot be processed in parallel) occurs, and the cognitive load becomes too concentrated on visual processing, which results in frequent eye movements between the hands and sheet music. As for [Step 3], motor learning difficulty consists of two causes. First, it is not possible to uniquely determine which finger should be used to press the target key because the number of keys is too large for the number of fingers (Sloboda et al., 1998). Second, holding chords requires simultaneous complicated multi-finger muscle control for pressing multiple keys at the same time. As for [Step 4], getting feedback for corrections (whether the correct key is pressed) only by the sense of pitch or touch is difficult for beginners, requires visual feedback. By explicitly diagramming the process of sensorimotor transformation in playing keyboard instruments (Figure 1), we realized that existing approaches dealt with only part of the steps in a whole process, and they are not enough to solve the difficulties (Figure 2). Regarding [Step 1], approaches, such as music game style sheet music notation, such as Piano games, AR assists (Hackl & Anthes, 2017; Sagawa et al., 1999; Trujano et al., 2018), and key lighting Piano keyboards (Mochizuki, 1982) have been proposed, improved to some extent, but unsolved problems remain. Music game style notation has the same number of lanes as the number of keys, and musical notes fall on the lanes. The keyboard should be pressed when musical notes reach the line representing the hitting point. Since there are many keys on Piano compared to the number of fingers, it is difficult to recognize instantly which lane corresponds to which key. Lighting a keyboard is an approach in which the next keys to be pressed are lit up. This approach still has the problem of not being able to look ahead for more than one note, and not being able to know the duration of the note. As for [Step 2] and [Step 3], only improving software is not enough to solve the problems. The basic Piano interface has not been updated for more than 300 years and is still the most common musical interface. We are now living in the digital era, it is necessary to fundamentally review how the digital hardware device should be fit for humans beyond the limitation of analog acoustic instruments. As for [Step 4], an existing approach that gives feedback to the player by moving the cursor on the digital sheet music on the PC monitor to the next note when the correct key is pressed (Dannenberg et al., 1990) works well.

Figure 2. Comparison between existing approaches and ParoTone to balance three elements.

It is not possible to practice existing musical instruments (Piano, Guitar, etc.) anytime and anywhere, the usage situations are limited, so it is difficult to improve skills. Even if they can improve their skills, there are limited places to perform, which reduces the player's motivation to begin with the existing musical instruments. If we try to make the Piano more portable, it is necessary to reduce the number of keys or make the keys smaller. On a Piano with a small number of keys, the number of octaves that can be played is extremely limited, and it is difficult to play chords and melody at the same time. It’s hard to say the guitar is portable enough. Even if they can carry guitars, unless they can mute the guitars, they can’t practice anytime, anywhere. To be able to practice and perform anytime, anywhere, portability and mute are required.

There are also problems in measuring brain activity during playing a musical instrument. To elucidate the mechanism of sensorimotor transformation during playing musical instruments, the researchers need to measure the brain activities of subjects during playing musical instruments. However, it is difficult for the subjects to do so in a natural situation in MRI scanners. Piano-type interfaces (such as midi Piano keyboards) have been mainly used in previous studies (Limb & Braun, 2008; McPherson & Limb, 2013), but technical restrictions remain. It is physically impossible to bring a horizontally long Piano with many keys into an MRI, so the available number of keys and the octave movements are limited in these kinds of experiments. Parallel movements of the palms are required for playing the Piano, but if the subject moves his or her arms too much, the noise will be added to the brain activity measurement, and the measurement itself will become difficult. It is also difficult to play with both hands with a limited number of portable midi Piano keyboard interfaces. Due to these restrictions, it is only possible to measure with a limited degree of freedom in playing in a considerably different situation compared to playing in a natural state. It is necessary to devise a mirror in the MRI scanner so that the subject can see the hands (Limb & Braun, 2008). In addition, the Piano-type interface is difficult to learn, so the number of subjects who can participate in the experiment is limited. To measure brain activity while playing an instrument in a more natural and flexible manner, we need a new musical instrument interface that is portable, octave-movable, minimizes palm and finger movements without adding noise to brain measurements, and is easy for many people to play with.

Therefore, we propose a new interface “ParoTone” (Figure 3) that balances three elements: easiness to learn, a wide range of expression, and portability (Figure 2). We fundamentally rethought how the device and software should be for efficient sensorimotor learning. We tested the hypotheses (Figure 4) that the new interface “ParoTone” with music game style notation, one-to-one correspondence between 5 lanes and 5 fingers (with octave shifts), playing chords with one button can balance the three elements. ParoTone is a brand new independent interface. It is not mainly an introductory device for the Piano. Of course, by learning ParoTone, beginners will learn the basic three elements of music (melody, chords, rhythm) and the skill of moving 5 fingers, so ParoTone also has the secondary value of making it easier to learn various other musical instruments.

Figure 3. Five lanes 5 fingers interface “ParoTone.” (a) A prototype of the ParoTone interface including sheet music display and hardware devices. (b) The hands fit into the actual hardware devices. (c) Detailed correspondence between the musical notes in sheet music and buttons of the hardware devices. (d,e,&f) ParoTone can switch 3 kinds of playing modes.

Figure 4. Hypotheses to test in this study.

2. Materials and methods

2.1. About “ParoTone” interface

Figure 3 and Supplementary Videos show the ParoTone interface (Mita & Sato, 2021).

2.1.1. One-to-one correspondence between 5 lanes, 5 buttons, and 5 fingers

The most characteristic point of the ParoTone interface is a one-to-one correspondence between 5 lanes of sheet music, 5 buttons, and 5 fingers (Figure 3(c)). The reason why there are six buttons is to be compatible both for the left hand and right hand. The rightmost button is inactive when using the right hand, and the leftmost button is inactive when using the left hand.

2.1.2. Three states transition in each button

Each button makes three states transition (top, middle, and bottom) smoothly in the flexion and extension directions of each finger (Figure 3(c); Supplementary Video 2). Users can feel the boundary between the three states by touch. Three colors of musical notes in sheet music notation correspond to the three states.

2.1.3. Radially arranged buttons

Buttons are radially arranged to ergonomically fit 5 fingers including the thumb.

2.1.4. Switch between melody mode and chord mode for each hand

There are two playing modes: melody mode and chord mode. Users can choose which mode to play for each hand, e.g., melody mode for both hands (Figure 3(d)), chord mode for both hands (Figure 3(e)), and chord mode for the left hand and melody mode for right hand (Figure 3(f)). In the case of melody mode, the five most frequently used sounds (C, D, E, G, A; major pentatonic scale) are arranged in the middle states (Figures 3(c,d)). The top states are semitone higher than the middle states. The bottom states are semitone lower than the middle states. These arrangements can cover all the 12 semitones of the octave from B to B flat. When playing a melody with both hands, by default, the right hand is one octave higher than the left hand. In the case of chord mode, users can play chords with one state in the button without the need for simultaneous control of multiple fingers. Frequently used chords are placed in the middle states, and frequently progressing pairs of chords are arranged next to each other (Figure 3(e)).

2.1.5. Octave shifts

Octaves are switched by sliding the device to the right (octave up) or left (octave down) (Supplementary Video 3).

2.2. Experiment 1: Comparison between piano (staff notation) and ParoTone

The hypotheses to be tested in experiment 1 are as follows: whether ParoTone is faster to learn than Piano (staff notation), whether ParoTone is easier to touch-type than Piano, and whether ParoTone reduces learning time more compared to Piano for beginners than for experienced people. Figure 5(a) and Supplementary Video 5 show the example scenes of experiment 1. There were 17 subjects (15 adult males, two adult females). Eleven persons were beginners (<1 year of keyboard instrument experience outside compulsory education) and six were experienced people (1 year or more of keyboard instrument experience outside compulsory education). Experiment sessions on Piano and ParoTone were done. Eight subjects did the Piano session first, and nine subjects did the ParoTone session first. For 1 min at the beginning of each session, the subjects looked at the correspondence table between the sheet music and the keyboard positions (pages 2 and 10 in Supplementary Figure 1) to understand the notation rules and played as they liked to practice. In the main part of the experiment, we measured how long it took for the subject to play the song “Jingle bells” chorus four bars three times, for three patterns of playing modes: (1) Melody only, (2) Chords only, (3) Melody and chords. Visual feedback about whether the correct keys were pressed was given to the subjects during playing. In the Piano session, the cursor moved to the next note when the correct key was pressed and did not move when the wrong key was pressed. In the ParoTone session, the musical notes stopped at the hitting point until the correct key was pressed (Supplementary Video 4). In both sessions, the correctness was judged only by whether the key with the correct pitch was pressed. Only the onsets of the sounds were used, and the offsets were not used for judgment. The durations of the notes were not used for correctness judgment. Metronome sounds with a target tempo of 120 beats per minute (0.5 s per beat) were played as background sounds. Regarding the acceptable timing of the onset, for example, if the time span from the onset of the previous note to the onset of the next note is one beat (corresponding to the time span of a quarter note), the timing to start accepting the next note is 0.5 s after the timing of the onset of the previous note. If the next note is played earlier than the timing, it will not be accepted even if the correct key is pressed. If the note is played later than the timing, it will be accepted no matter how late. The longer the delay of the timing of playing each note compared to the correct timing, the longer time it took for the subject to complete playing all the notes. The ideal is to play all the notes without mistakes at the target tempo without delay. The number of times the subjects moved their line of sight from the screen to their hands during the performance was counted (Figure 5(e)).

Figure 5. Experiment 1: Comparison of learning speed and the number of gaze movements between Piano (staff notation) and ParoTone. (a) The experiment setup. (b,c,&d) Comparison of the time to play the the song “Jingle bells”' chorus 4 bars 3 times correctly, for 3 patterns of playing modes. (e) Gaze recording to count how many times the subjects moved their line of sight to their hand during playing. (f,g,&h) Comparison of the number of gaze movement during playing.

2.3. Experiment 2: Comparison among 4 types of notation

The hypotheses to be tested in experiment 2 are as follows: whether the ease of reading, touch typing, and motor control contribute to the improvement of learning speed (if so, to what extent each step contributes), whether the music game style notation is faster to learn than the staff notation, whether ParoTone is faster to learn than the Piano game style sheet music notation. There were 10 subjects (9 adult males, one adult female). Six persons were beginners (<1 year of keyboard instrument experience outside compulsory education) and four were experienced people (1 year or more of keyboard instrument experience outside compulsory education). Experiment sessions on Piano and ParoTone were done. Four subjects did the Piano session first, and six subjects did the ParoTone session first. Subjects played four kinds of sheet music notation (Figure 7(a)):

• [Notation X] Piano (staff notation)

• [Notation Y] ParoTone (ParoTone notation)

• [Notation Z] Piano (indicating positions to be pressed on keyboard diagram, similar to music game style notation)

• [Notation W] ParoTone (indicating positions to be pressed on ParoTone diagram)

We measured how long it took for the subjects to play a random 5-tone sequence for two patterns of playing modes: (1) Melody, and (2) Chord. Visual feedback about whether the correct keys were pressed was given to the subjects during playing (the same way as in Experiment 1).

2.3.1. How to analyze the contribution of each step in sensorimotor transformation

Compared to [Notation X] and [Notation Y], [Notation Z] and [Notation W] exclude the difficulty of [Step 1] in learning the correspondence between sheet music and keyboard position (Figure 7(a)). Therefore, by comparing the learning speeds of [Notation X] and [Notation Y], it is possible to understand to what extent all the factors including [Step 1], [Step 2], [Step 3], and [Step 4] affect the learning speed difference between the Piano (staff notation) and ParoTone. By comparing the learning speeds of [Notation Z] and [Notation W], it is possible to understand to what extent only [Step 2], [Step 3], and [Step 4], excluding [Step 1], affect the learning speed difference between Piano (staff notation) and ParoTone.

2.3.2. Staff notation vs. music game style notation

By comparing [Notation X] and [Notation Z], we can investigate whether there is a difference in the learning speed between the staff notation and music game style notation.

2.3.3. Piano (music game style notation) vs. ParoTone

By comparing [Notation Z] and [Notation Y], we can investigate whether there is a difference in the learning speed between the Piano (music game style like notation) and ParoTone.

3. Results

3.1. Experiment 1: Comparison between piano (staff notation) and ParoTone

3.1.1. Comparison result of learning speed

Figures 5(b–d) and Table 1 show learning speed comparison results. The learning speed of the ParoTone was faster than that of the Piano (staff notation) (p < 0.01 in each case, Wilcoxon signed-rank test, 2.3 times faster on average for playing melody (Figure 5(b)), 3.8 times faster on average for playing chords (Figure 5(c)), 3.6 times faster on average for playing both melody and chords (Figure 5(d)).

Table 1. How many times faster to learn ParoTone than Piano (staff notation).

	Beginners’ group (n = 11)	Experienced group (n = 6)	All subjects (n = 17)
Melody	2.8**	–	2.3**
Chord	4.8**	2.2*	3.8**
Melody + Chord	4.2**	–	3.6**

**p < 0.01, *p < 0.05, –: no significant difference (Wilcoxon signed-rank test).

3.1.2. Comparison result of the number of gaze movements

Figures 5(f–h) show a comparison of how many times the subjects moved their line of sight to their hand during performance. In the case of ParoTone, the number of times the line of sight moved to the hands during the performance was less than that of the Piano (p < 0.01 in each case, Wilcoxon signed-rank test, as for melody in Figure 5(f), chords in Figure 5(g), both melody and chords in Figure 5(h)). This result shows that ParoTone is superior to Piano in touch-typing. In other words, by actively using the sense of touch to grasp the position of fingers without visual clues in [Step 2], the load of visual information is reduced. This enables the players to process parallelly reading sheet music in [Step 1] and grasping the current position of fingers by touch-typing in [Step 2].

3.1.3. Comparison between beginners and experienced people

Figure 6 and Table 1 show the detailed comparison of learning speed between groups with and without more than 1 year of keyboard instrument experience. Especially in the beginner group (<1 year of keyboard instrument experience outside compulsory education), the difference in learning speed between Piano (staff notation) and ParoTone was remarkable. In the case of beginners’ group, learning melody performance was 2.8 times faster on average, chord performance was 4.8 times faster on average, both melody and chord performance were 4.2 times faster on average than Piano (staff notation) (p < 0.01 in each case, Wilcoxon signed-rank test). As for the experienced group (more than 1 year of keyboard instrument experience outside of compulsory education), there was a significant difference in the learning speed of chord performance (p < 0.05, Wilcoxon signed-rank test), but there was no significant difference in melody performance and both melody and chord performance.

Figure 6. Detailed comparison of learning speed between beginners and experienced groups. (a) The data in Figure 5b was divided into two groups. (b) The data in Figure 5c was divided into two groups. (c) The data in Figure 5d was divided into two groups.

3.2. Experiment 2: Comparison among 4 types of notation

3.2.1. To what extent which steps contribute to the improvement of learning speed of ParoTone compared to piano (staff notation)

Figure 7(b) and Table 2 show the investigation of which steps in sensorimotor transformation contribute to improve the learning speed of melody performance. The learning speed of [Notation Y] was 2.7 times faster on average than [Notation X] (p < 0.05, Wilcoxon signed-rank test). There was no significant difference in learning speed between [Notation Z] and [Notation W] (p = 0.55, Wilcoxon signed-rank test). From these results, in the case of learning melody performance, ease of reading sheet music in [Step 1] contributed to improve the learning speed of ParoTone compared to Piano (staff notation).

Figure 7. Experiment 2: Which steps contribute to speed up learning, and comparison between Piano (music game style notation) and ParoTone. (a) Abbreviated diagram of Figure 1, and 4 types of notations used in experiment 2. (b) Comparison of the time to play the assigned melody tone sequences correctly among 4 types of notations. (c) Comparison of the time to play the assigned chord tone sequences correctly among 4 types of notations. (d) Comparison of the time to play the assigned tone sequences correctly between the notation [Z] and [Y].

Table 2. How many times faster to learn ParoTone than Piano.

	[X] vs. [Y] (n = 10)	[Z] vs. [W] (n = 10)	[Y] vs. [Z] (n = 10)
Melody	2.7*	–	–
Chord	13.3**	4.6**	4.1**

**p < 0.01, *p < 0.05, –: no significant difference (Wilcoxon signed-rank test).

Figure 7(c) and Table 2 show the investigation of which steps contribute to improving the learning speed of chord performance. The learning speed of [Notation Y] was 13.3 times faster on average than that of [Notation X] (p < 0.01, Wilcoxon signed-rank test). The learning speed of [Notation W] was 4.6 times faster on average than [Notation Z] (p < 0.01, Wilcoxon signed-rank test). From these results, it was possible to improve the learning speed only by the ease of touch-typing in [Step 2] and motor control in [Step 3], but the main factor that contributed was the ease of reading sheet music in [Step 1].

3.2.2. Comparison of learning speed between staff notation and music game style notation

[Notation X] vs. [Notation Z] in Figures 7(b,c) show the results of comparing learning speed between staff notation and music game style notation. Music game style notation was faster to learn than staff notation for both melody and chords (p < 0.05, Wilcoxon signed-rank test).

3.2.3. Comparison of learning speed between piano (music game style notation) and ParoTone

Figure 7(d) and Table 2 show the result of comparing learning speed between Piano (music game style notation) and ParoTone. There was no significant difference in the learning speed of melody performance (p = 0.44, Wilcoxon signed-rank test). Regarding the learning speed of chord performance, ParoTone was 4.1 times faster on average than Piano (music game style notation) (p < 0.01, Wilcoxon signed-rank test).

4. Discussion

The main hypotheses of this study (Figure 4) are to test whether the new ParoTone interface with music game style notation, one-to-one correspondence between 5 lanes and 5 fingers (with octave shifts), and playing chords with one button can balance the three elements: easiness to learn, a wide range of expression, and portability. We will discuss whether the hypotheses (Figure 4) are valid or not compared to existing approaches (Figure 2).

4.1. Balancing portability and a wide range of expression

We will discuss whether the interface with 5 lanes and three states for 5 fingers (with octave shifts) balances portability and a wide range of expression (Figure 4(d)).

First, concerning portability, as you can see in Figure 3(b), ParoTone is hand-size, overwhelmingly smaller than a Piano and a Guitar, can be muted and can be practiced and performed anytime, anywhere. Therefore, portability can be realized (without increasing the number of keys like a Piano) with the new interface with 5 lanes and three states for 5 fingers (with octave shifts).

Second, concerning a wide range of expression, it is possible to play all three basic elements of music (melody, chords, and rhythm) with ParoTone. Although ParoTone does not have as much degree of freedom as a Piano (up to 7.5 octaves), the octave shifts of ParoTone (Supplementary Video 3) allow you to play three octaves with one hand and up to 6 octaves with both hands in melody mode, which is wider than a Guitar (up to four octaves). In chord mode, you can play up to 45 chords (15 chords and three shifts) with one hand. You can play most pop music, easy jazz, and easy classics with ParoTone (Supplementary Video 1). It is true that Piano has an advantage compared to ParoTone in that Piano players have many choices of fingerings and can play fast passages or repeated notes by alternating fingers for greater speed, but even with ParoTone, it is possible for advanced players to change the fingerings. For example, instead of using the ring finger and little finger, you can use the index and middle fingers to trill faster (Supplementary Video 6). Statistically, the pentatonic scale (C, D, E, G, A) is the most often used scale in pop music, easy jazz, easy classics, and so on. With ParoTone, it is possible to fluently and quickly play melodies that make heavy use of the pentatonic scale and melodies that decorate the sound of the pentatonic scale from above and below with a semitone difference. Moreover, it is possible to play chords in melody mode. In this case, any chords can be played because you can freely select the constituent notes of the chords. It is also technically possible to implement touch response (impart sound intensity variation to a music performance) in ParoTone in the future by adjusting the range of the pressure sensor to the range of push strength that can be adjusted with the fingers.

There are also portable simplified Piano products on the market. However, it is impossible to achieve both portability and a wide range of expression at the same time with the Piano interface. If we try to make the Piano portable, it is necessary to reduce the number of keys or make each key smaller. On a Piano with a small number of keys, the number of octaves that can be played is extremely limited, and it becomes difficult to play chords and melodies at the same time. ParoTone can cover one octave with only 5 lanes, but Piano needs at least 12 lanes to cover one octave. If each key of the Piano becomes smaller, it will be difficult to play without miss touch.

From the discussion above, the hypothesis that “the interface with 5 lanes and three states for 5 fingers (with octave shifts) can balance portability and a wide range of expression” (Figure 4(d)) is valid. The ParoTone interface makes it possible to practice and show the performances anytime, anywhere compared to existing approaches.

4.2. Whether ParoTone interface makes learning easier

We tested whether the new ParoTone interface makes it easy for anyone to improve the efficiency of sensorimotor transformation learning compared to the Piano interface. In addition to conventional Piano staff notation, there are music game style notations, such as Piano games, AR assists (Hackl & Anthes, 2017; Sagawa et al., 1999; Trujano et al., 2018), and key lighting Piano keyboards (Mochizuki, 1982) (Figure 2). We consider the AR method to be essentially the same as an extension of the music game style notation because it is no different in that the notes fall from the top. To test the hypotheses, we implemented the prototype of hardware and software, conducted the two experiments, and got results. From the experiments, we got the result that ParoTone was faster to learn than Piano (staff notation) in melody performance (2.3 times faster) and chord performance (3.8 times faster) (Figures 5(b,c), in experiment 1). We also got the result that ParoTone was faster to learn than Piano (music game style notation) in chord performance (4.1 times faster) (Figure 7(d), in experiment 2). From these results, we conclude that the new ParoTone interface improves the efficiency of sensorimotor transformation learning compared to the existing Piano interface, which leads to an improvement of the learning speed. Therefore, we consider that our hypothesis “ParoTone interface makes learning easier” is valid.

4.2.1. Whether music game style notation makes reading easier (a1 & a2 in Figure 4)

In the results of experiment 2 in Figures 7(b,c), [Notation Z] (Piano game style notation) was faster to learn than [Notation X] (the Piano staff notation). Therefore, the hypothesis that “music game style notation that can be read without special knowledge about notation makes it easier to read and learn” has been substantiated.

4.2.2. Whether 5 lanes for 5 fingers interface makes touch typing easier (b1 & b2 in Figure 4)

Subjects looked at their hands less often during learning ParoTone than Piano (Figures 5f, g, and h, in experiment 1). This result means the ParoTone is superior to Piano in touch-typing and supports the hypothesis that “5 lanes for 5 fingers interface makes touch typing easier” (Figure 4(b1)). We interpret this result as thanks to the characteristics of 5 lanes, 5 rows of buttons and 5 fingers in a one-to-one correspondence, radial button arrangement, three states transition buttons in the direction of finger flexion and extension, and being able to feel the boundaries of the three states by touch. Thus, ParoTone makes it possible to effectively utilize the sense of touch and reduce the cognitive load on the visual sense. However, in experiment 2, there was no significant difference in learning speed between the Piano game style sheet music and the ParoTone’s sheet music in playing melody mode (Figure 7(d); The condition of the music game style notation and the number of buttons pressed simultaneously are the same. The only difference is the placement of lanes and buttons). Therefore, we could not substantiate that “easy touch typing with 5 lanes and 5 fingers speeds up learning” (Figure 4(b2)) in this experiment. This may be because we used a relatively simple melody within two octaves in this experiment. As the number of keys used for the Piano was limited to two octaves, there was not so big difference between the number of lanes in Piano sheet music and ParoTone, resulting in no significant difference in the speed of learning the songs. As a future experimental plan, it is necessary to compare the learning speed of Piano and ParoTone in relatively difficult melody using three octaves or more.

4.2.3. Whether playing chords with 1 button makes controlling fingers easier (c1 & c2 in Figure 4)

In experiment 2, ParoTone was faster to learn than the Piano music game style like a notation in playing chords (Figure 7(d)) under the condition that both ParoTone and Piano use music game style notation. This result supports the hypothesis that “Playing chords with one button eliminates the need for simultaneous control of multiple fingers, so finger movement control becomes easier and learning becomes faster.”

4.2.4. To what extent each step in sensorimotor transformation streamlines learning?

As for melody performance, in Figure 7(b), the learning speed of [Notation Y] was 2.7 times faster on average than [Notation X], but there was no significant difference in learning speed between [Notation Z] and [Notation W]. From this result, in learning to play a melody, [Step 1] is effective in improving the learning speed. As for chord performance, in Figure 7(c), the learning speed of [Notation Y] was 13.3 times faster on average than that of [Notation X], and the learning speed of [Notation W] was 4.6 times faster on average than [Notation Z]. From this result, in learning chord performances, [Step 1] contributed mainly, but [Step 2] and [Step 3] also contributed (4.6 out of 13.3) to the improvement of learning speed. In both cases, regardless of melody or chord, the [Step 1] contributes most to the improvement of learning speed.

4.2.5. Is the interface effective for both beginners and experienced people?

From the result of Figures 6(a,b), when comparing the learning speed of ParoTone with the Piano (staff notation) in the beginners’ group, ParoTone is 2.8 times faster on average for melody performance and 4.8 times faster on average for chord performance. In the case of subjects with keyboard instrument experience, there was no significant difference in melody performance and 2.2 times faster on average for chord performance. These results show that the ParoTone interface is more effective for speeding up learning for beginners (both melody and chords) than experienced people (only chords).

4.3. Limitations

We acknowledge there are several limitations to this study. First, in this study, we conducted the experiments using the melodies within a range of two octaves. Although this study holds for many songs because the melodies of many songs fit within two octaves, whether or not it holds true even when playing a song containing a melody of three octaves or more need to be examined in future research.

4.4. Future applications

4.4.1. For music

The ParoTone interface makes it possible to deliver the joy of performing music to more people (beginners of musical instruments, children in developing countries, etc.) in a portable and low-cost manner. Taking advantage of efficient learning of touch-typing, ParoTone also has the potential as a barrier-free musical device for visually impaired people in the future. Compared to existing Piano midi keyboard devices, ParoTone is more portable, octave-movable, minimizes palm and finger movements without adding noise to brain measurements, and is easy for many people to play with. In the future study, the ParoTone interface also has the potential to be useful for solving the problems in measuring brain activity in MRI while playing musical instruments in a more natural and flexible manner.

4.4.2. For other fields than music

Taking advantage of efficient sensorimotor learning based on the readability of operation instruction display and on touch-typing, ParoTone interface may make it easier for learning to adjust multiple commands in a timely manner. For example, in effective video production and distribution, operators need to learn how to switch and adjust multiple camera angles, lighting brightness, sound levels, etc. in a timely manner.

5. Conclusion

From all the discussions above, we conclude that ParoTone interface with music game style notation, one-to-one correspondence between 5 lanes and 5 fingers (with octave shifts), playing chords with one button balances the three elements: easiness to learn, a wide range of expression, and portability compared to existing approaches. The ParoTone interface enables efficient sensorimotor learning, which leads especially for beginners to learn musical performance much faster than Piano. Taking advantage of efficient sensorimotor learning based on the readability of operation instruction display and on touch-typing, ParoTone interface in the future may be useful for learning to operate multiple commands in a timely manner.

Author contributions

Shinjiro Mita, Nayuta Sato, and Taiga Fujimoto invented the new interface. Shinjiro Mita, Hiroki Fukunaga, Zenjiro Mita, and Satoru Iwasaki designed the research. Morio Fujimoto and Makoto Ozawa implemented the hardware.: Shunichi Sugimoto, Taiga Fujimoto, and Nayuta Sato implemented the software. Shinjiro Mita performed the experiments, analyzed the data, and wrote the first draft. All the authors revised the manuscript.

Acknowledgements

We would like to thank all the authors for contributing to this research project, 17 subjects for cooperating with the experiments, and the humanware innovation program office for supporting the research grant.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This work was supported by the “Program for Leading Graduate Schools: Humanware Innovation Program” of Osaka University.

Notes on contributors

Shinjiro Mita

Shinjiro Mita is a PhD student at Osaka University majoring in cognitive neuroscience. He belongs to the Humanware Innovation Program of Osaka University. He is a founder of eMotto Co., Ltd. He has absolute pitch and over 20 years of Piano playing experience.

Nayuta Sato

Nayuta Sato received the BS (Human Science) at Osaka University, in 2020. He is also a co-founder of eMotto Co., Ltd. He also has absolute pitch and his research interest is the relationship between music and emotions.

Morio Fujimoto

Morio Fujimoto is a PhD student at Osaka University. He specializes in mechanical aspects in hardware and he has over 10 years experience as a professional hardware engineer.

Shunichi Sugimoto

Shunichi Sugimoto is a software engineer, PhD student at Osaka University majoring in neuroscience.

Taiga Fujimoto

Taiga Fujimoto is a software engineer. He received the MS (Science) at Osaka University, in 2018.

Makoto Ozawa

Makoto Ozawa is a PhD student at Osaka University majoring in system control engineering. He belongs to the Humanware Innovation Program of Osaka University.

Hiroki Fukunaga

Hiroki Fukunaga is a PhD student at Osaka University. He belongs to the Humanware Innovation Program of Osaka University. His research interest is the molecular biological mechanism of muscle contraction.

Zenjiro Mita

Zenjiro Mita is a PhD student at Osaka University majoring in optical physics. He has absolute pitch and over 20 years of Piano playing experience.

Satoru Iwasaki

Satoru Iwasaki is an Assistant Professor at Osaka University. His research interest is mathematical modelling and analytical study of non-linear differential equations.