Optical laser microphone for human-robot interaction: speech recognition in extremely noisy service environments

Abstract

Domestic robots are often required to understand spoken commands in noisy environments, including service appliances' operating sounds. Most conventional domestic robots use electret condenser microphones (ECMs) to record the sound. However, the ECMs are known to be sensitive to the noise in the direction of sound arrival. The laser Doppler vibrometer (LDV), which has been widely used in the research field of measurement, has the potential to work as a new speech-input device to solve this problem. The aim of this paper is to investigate the effectiveness of using the LDV as an optical laser microphone for human-robot interaction in extremely noisy service environments. Our robot irradiates an object near a speaker with a laser and measures the vibration of the object to record the sound. We conducted three experiments to assess the performance of speech recognition using the optical laser microphone in various settings and showed stable performance in extremely noisy conditions compared with a conventional ECM.

GRAPHICAL ABSTRACT

Keywords:

• Optical laser microphone
• speech recognition
• domestic robot
• laser Doppler vibrometer

1. Introduction

Human-robot interaction applications, which have been in high demand for future services in domestic environments, require the capabilities to accurately receive and understand the user's linguistic instructions. Speech is a natural and effortless way for humans to communicate. The use of spoken commands to operate domestic robots involves the implementation of an automatic speech recognition system. For example, when the user says to the robot ‘Tell me the name of the person in the dining room,’ the robot should search the dining room and identify the person there. In a practical setting, both the user and the robot are not fixed but often move, so the speech recognition performance should be robust to their relative positions. We should also consider the presence of noise around the robot, e.g. sounds of service appliances and conversation voices of people in the same room, as well as the robot operation noise due to electric fans and actuators. Thus, developing techniques for recording clean speech is crucial to achieve high-quality speech recognition of domestic robots.

Most conventional robots [1] use beamforming with microphone arrays to filter interfering noise and reverberation, which estimates the direction of arrival of the speech signal. Electret condenser microphones (ECMs), such as gunshot microphones, provide the same solution at a hardware level. However, in an environment in which a speaker and some noise sources exist in the same direction, as shown in Figure 1(a), these conventional microphones might enhance not only the speech signal but also the noise.

Figure 1. Different relative positions between a microphone and a robot in noisy domestic environments. (a) Conventional microphones might fail to collect speech if noise sources exist between the speaker and the robot or behind the speaker; (b) The robot directly measures the vibration of an object near the speaker. (a) Conventional microphone and (b) Proposed optical laser microphone.

The sound recording quality of the conventional microphones would also be affected when an object emitting the noise exists behind the speaker. The sound wave is attenuated if a microphone is distant from a speech source, which makes the energy of speech low and decreases the signal-to-noise ratio (SNR). Thus, we should develop a microphone robust to these possible conditions in service environments. In the research field of measurement, the laser Doppler vibrometer (LDV) [2] has been developed to directly record the vibration around a target object. The use of the LDV for the purpose of sound recording is called an optical laser microphone, which has recently attracted much attention [3,4]. Using a laser to measure the vibration of objects near a speaker, we can obtain the clean speech uttered by the speaker avoiding the noise around the microphone.

The main objective of this paper is to investigate the effectiveness of an optical laser microphone for human-robot interaction in a noisy service environment. After detecting a speaker using the camera sensor of the robot, the robot uses a LDV to measure the vibration of an object, which is located near the speaker, as a speech signal. Then, the robot sends speech segments extracted from the speech signal to an automatic speech recognition module. Finally, the robot selects the single sentence with the highest confidence of the recognition as a command provided by the speaker. The proposed method enables to understand the speaker's command even in the extremely noisy environment shown in Figure 1(b). Our experiments were conducted following the task defined by RoboCup@Home,¹ which is one of the most famous competitions of domestic robot applications in Japan. The results demonstrated that our optical laser microphone has an advantage that can lead to stable speech recognition performances compared to a conventional ECM.

In summary, the main contributions of this paper are as follows.

• To the best of our knowledge, this is the first study to construct a robot that recognizes the user's spoken commands using an optical laser microphone.

• We provide a comprehensive evaluation of the speech recognition performance, especially in terms of different irradiated targets as well as different relative positions of a robot, speaker, and the irradiated targets. We also tested the performance not only in a clean environment but also in an extremely noisy, real home-like environment.

The remainder of this paper is organized as follows. Section 2 describes conventional studies on robot microphones and related to signal processing technologies. Section 3 presents the robot that uses an optical laser microphone as an input device of spoken commands. Section 4 presents three experiments to assess the performance of the optical laser microphone and validate its effectiveness. Finally, Section 5 summarizes the paper and suggests some future research topics.

2. Related work

A variety of microphones have been used in the field of audio and speech recognition. Air conduction microphones, such as parabolic microphones and shotgun microphones, measure the vibrations of the microphone diaphragm to record speech from a speaker. Some autonomous robots employ a directional microphone [5] or a microphone array [6,7] as an input device for spoken dialog system. In particular, a shotgun microphone which is one of directional microphones was used as a popular input device in the RobotCup@Home league [8]. However, air conduction microphones have the following two limitations: (i) the farther the distance between a robot and a speaker is, the greater the energy of sound wave decay, which produces a distorted speech in an extremely noisy environment; and (ii) the measurement is affected by not only noise around the robot but also noise around the speaker. To tackle these problems, several researchers developed noise reduction techniques, such as microphone-array signal processing [9], filtering-based methods [10], spectral subtraction [11,12], and signal subspace methods [13]. For example, robot audition developed by Nakadai et al. used adaptive filtering based on the human auditory characteristics [14,15] and semi-blind independent component analysis with a microphone array [16]. Model-based signal processing techniques are also popular, e.g. noise reduction based on a maximum a posteriori estimation [17] and speech enhancement based on phase restoration [18]. Recent studies estimated the amplitude spectrum of speech signal based on deep learning [19,20]. However, these post-processing methods often suffer from performance degradation of speech recognition in diverse combinations of noise types, noise stationary, and SNR. Therefore, we need to improve the quality of the speech recording itself as much as possible. As mentioned above, the optical laser microphone has the potential to enhance the noise suppression performance of previous studies because it is less sensitive to the noise around the microphone and can record distant-talking speech that is otherwise difficult to measure with air-conduction microphones.

Optical laser microphones use the LDV to measure the vibrations of the objects near the speakers, thus avoiding the noise around the microphones. The LDV has been widely developed and applied in several fields: for example, lung sound measurement and breast cancer screening in medical research [21,22], or damage detection of bridges and plates in architectural research [23,24]. In acoustics, Oikawa et al. [25] used the LDV to measure the sound field to analyze the physical phenomena of sound, such as propagation, attenuation, reflection, and diffraction. In robotics, Morita et al. [26] used the LDV as a slip sensor to control robot hands. Margerit et al. [27] mounted LDVs on industrial robot arms as sensors for full-field velocity measurements. To the best of our knowledge, there is no previous research that introduces the LDV into robots' speech recognition systems. Recent attempts to downsize the LDVs and reduce their manufacturing cost have a great potential to spread optical laser microphones into domestic applications.

Our study considers a situation in which a user speaks to a robot. However, another approach to give robot speech commands can be to use the microphone of a wireless device, such as a smart speaker or smartphone [28], which requires additional wireless repeaters to transfer the recorded sound from the device to the robot. Since this paper does not focus on such internet-of-things systems but on human-robot interaction, we will discuss the possibility of combination between these techniques in future works.

3. Proposed robot

This section describes our robot that uses an optical laser microphone as an input device of spoken commands. Figure 2 shows an overview of the proposed robot's processing to receive a spoken instruction. Below, we explain the hardware and software aspects of the proposed robot, respectively.

Figure 2. Overview of the proposed method's processing to receive a spoken instruction.

3.1. Hardware description: robot framework

We use a Human Support Robot (HSR) [29] provided by Toyota Motor Corporation with the aim to improve quality of life at home. The HSR is equipped with various sensors, such as a stereo camera, RGB-D camera, and wide-angle camera to visually recognize objects in the environment.

Instead of using the HSR's built-in microphone array of ECMs, our study uses the LDV (Polytec NLV-2500) as an optical laser microphone. The LDV irradiates a vibrating object with a laser and detects frequency change of the laser due to the irradiation position movement according to the vibration. Specifically, when a laser beam is reflected from a vibrating object, the frequency of the laser beam changes according to the object's vibration speed. The LDV converts the measured frequency change into an electrical signal, which corresponds to the sound signal. The laser of our LDV corresponds to Class 2 laser, or lower, as described by Japanese Industrial Standards (JIS) C6802 based on the International Electrotechnical Commission (IEC) standard which requires that the protection of the eyes can be achieved by an antipathetic response that includes blinking.

3.2. Software description: automatic speech recognition

Using the optical laser microphone, the proposed robot receives spoken instruction commands. We present the four steps to achieve it.

• Step 1: Object detection and sound recording

  The LDV mounted on the robot first irradiates an object near to a speaker with a laser beam and measure the object's vibration. The target objects should be chosen from a predefined category list including plastic bottles, glasses, and snack packs, which tend to easily vibrate. Specifically, using an image recognition-based object detector, such as YOLO [30], the robot detects the position of a speaker. Then, the robot searches for objects near the speaker and whose categories belong to the predefined list. In our experiments, a reflective tape was attached to a target object when measuring speech with the LDV. The number of objects detected is denoted by N. The robot obtains the sound by receiving the results of the LDV measurement.

• Step 2: Speech segmentation

  While constantly recording the sound, it is necessary to detect segments in the uttered speech. The use of deep neural networks has become mainstream in recent speech segment detection methods [31–34], and their toolkits are publicly available. We conducted a preliminary experiment using a subset of our corpus and confirmed that both the LDV and the ECM successfully detected the speech segments that are necessary for recognizing spoken commands. Specifically, using the voice activity detection (VAD) toolkit [35], a Python interface of Google WebRTC,² there was almost no difference in VAD performance between the ECM and the LDV. Our future work will further detail results of more domestic robot-related modules, including VAD.

• Step 3: Speech recognition and confidence analysis

  The robot needs to perform synchronous recognition of speech segments. Our robot uses the Google Cloud Speech-to-Text API,³ which have been popular in recent RoboCup competitions. As parameters of the API, we set Encoding to ‘LINEAR16’ (Linear PCM which means 16-bit linear pulse-code modulation encoding), Language to ‘en-US’ (American English), and Model to ‘video’ (the model used for transcribing audio in video clips that includes multiple speakers). Finally, for a given speech segment, we obtain a recognized transcription and its estimation confidence score, which corresponds to the likelihood of the recognition model.

• Step 4: Spoken command decision

  The robot is required to accurately perform users' instructions. Therefore, as shown in Figure 2, our robot repeats Steps 1 to 3 for N objects near to the speaker and recognizes the speech at each object. The robot finds a ‘best object,’ i.e. with the highest confidence score, and selects the transcription at the best object as a user's spoken command.

4. Experiments

This section presents the results of experiments⁴ that followed an early test of our proposed system at RoboCup@Home General Purpose Service Robot (GPSR). Specifically, in a single trial, the participant of the experiments gives a speech command to the robot, and the robot should recognize it to perform an appropriate service task. The commands and scenarios of each trial were prepared using the GPSR Sentence Generator,⁵ producing 15 imperative English sentences (5 sentences × 3 categories), as shown in Table 1. Although some commands have grammatical errors, they were generated identically during RoboCup@Home. For comparison purpose with the robotics community, we left the generated sentences untouched.

Table 1. 15 speech commands used in the experiments.

Category	Script
	Go to the side table, find the coke, and put it on the end table.
	Go to the desk, find the sprite, and place it on the end table.
1	Tell me the name of the person in the dining room.
	Locate the crackers in the kitchen.
	Pick up the potato chips from the storage table and deliver it to me.
	Meet John at the entrance and follow him.
	Locate the cutlery in the corridor.
2	Tell me what's the biggest containers on the side table.
	Bring the paprika from the dining room to the sink.
	Tell me the name of the person at the entrance.
	Robot please put the orange on the counter.
	Could you meet Skyler and take her?
3	Take the sponge to the sink.
	Could you meet Linda at the bed, follow her, and navigate to the kitchen?
	Robot please follow Skyler.

Note: These commands were generated by the GPSR Sentence Generator.

To investigate the effectiveness of the optical laser microphone for this task, we conducted the following three experiments:

• Experiment 1: We first provide a performance comparison of different objects as laser irradiation targets in a recording studio.

• Experiment 2: We compared the performance of the optical laser microphone and a conventional gunshot microphone for different positions of speakers, microphones, and irradiated objects in a recording studio.

• Experiment 3: We compared the performance of the optical laser microphone and a conventional gunshot microphone in a domestic environment.

In all experiments, each participant should speak the same sentence at the same volume for a fair comparison of the microphones. Thus, we decided to first record the participants' voices in Experiment 1 and reproduce the recorded speech from a loudspeaker in Experiments 2 and 3. Figure 3 represents the amplitude spectrum of the directional loudspeaker used in our experiments. The noise was emitted according to this amplitude characteristic. We can find from this figure that the frequency band corresponding to human speech (i.e. frequencies up to 16 kHz) is almost stable, resulting in successful speech reproduction. The sound pressure level was normalized for each experiment across all participants. Note that this study focuses on revealing the difference in the automatic speech recognition performance between the ECM and the LDV. Thus, our experiments started by recognizing the same speech segments to avoid contamination with performance degradation in the voice activity detection module.

Figure 3. Amplitude spectrum of the loudspeaker.

4.1. Experiment 1: comparison of different objects irradiated by the laser

The quality of the sound recorded using LDV depends on the material of the irradiated objects. In Experiment 1, we compared the performance of the following nine objects as irradiation targets: a speaker's throat, an aluminum bottle, a tissue box, a bag of potato chips, an instant noodle cup, a mug, a paper cup, and a plastic bottle, which can be found in domestic environments. All the bottles, cups, and mugs were empty to remove the effect of the contents in this experiment. Table 2 shows details of these irradiated objects. The speech recordings were performed as follows. We asked 10 subjects, whose native languages were not English but Japanese or Chinese, to read out each of the 15 speech commands listed in Table 1. Figure 4 shows the spatial relationships between the participant and the devices used in the recording. The distance between the LDV and the irradiated object was set to 1 m. As shown in Figure 4(a), the speech uttered by the subject was simultaneously measured using the ECM (Sennheiser MKH 416-P48U3) and the LDV (as described in Section 3.1). The LDV targeted the laser to the speaker's throat.⁶ Here, the ECM successfully recorded voices due to the noiseless studio, and we denote the resulting speech data by clean speech. We reproduced the clean speech from the loudspeaker when irradiating a target object, as shown in Figure 4(b), which allows a fair comparison of different objects.

Figure 4. Recording situation: (a) the speech was simultaneously measured using ECM and LDV, in which the ECM recording was regarded as clean speech, and the laser was targeted to the speaker's throat; and (b) the clean speech was played back from the loudspeaker, and the laser of the LDV irradiated a target object (e.g. plastic bottle). (a) Clean speech recording and throat-based measurement of the LDV and (b) Clean speech playback and plastic bottle-based measurement of the LDV.

Table 2. Details of the irradiated objects.

Object	Aluminum	Tissue	Bag of	Glass	Instant		Paper	Plastic
name	bottle	box	potato chips	cup	noodle cup	Mug	cup	bottle
Figure
	W:63	W:230	W:235	W:80	W:109	W:121	W:72	W:52
Size [mm]	D:63	D:110	D:205	D:80	D:109	D:80	D:72	D:52
	H:164	H:45	H:65	H:145	H:119	H:96	H:78	H:165
Weight [g]	25	170	10	240	20	330	5	20
Resonance frequency [kHz]	3.12	1.65	1.44	8.76	1.72	19.22	1.08	0.92

The quality of a microphone's sound recording can be quantified by evaluating the speech recognition results. We calculated the following three types of performance measures: the word error rate (WER), the perceptual evaluation of speech quality (PESQ) [36], and the short time objective intelligibility (STOI) [37]. The WER was calculated using the following equation: (1) $\mathrm{W}\mathrm{E}\mathrm{R}=\frac{I+S+D}{C},$(1) where I, S, D, and C are the numbers of inserted words, substituted words, deleted words, and correct words, respectively. The PESQ evaluates how the speech was distorted from its clean speech, especially in the telephone band (0.3–3.4 kHz). The STOI measures the intelligibility of a given speech on the basis of the correlations between the power spectra of the clean and testing speech at each octave band. Table 3 shows the results of WER, PESQ, and STOI for each irradiated object in Experiment 1. Note that the PESQ and STOI of the clean speech cannot be calculated since their calculation requires both clean and reference speeches. The WER of the clean speech recorded using the ECM was 21.5%. The tissue box achieved the best WER and PESQ in Table 3. When a plastic bottle was irradiated by the LDV, the WER was 35.9% (i.e. a quality decrease of about 15% from clean speech). On the other hand, when the speaker's throat was irradiated by the LDV, the WER significantly degraded to 99%. Figure 5 shows the spectrograms of a single clean speech, a speech measured using the nine target objects. We can observe that the result in Figure 5(c), which was obtained from the LDV that irradiates the throat, drastically changed from the clean speech result shown in Figure 5(a). Based on these results, we decided to use a plastic bottle, which achieved a moderate result in Table 3, as the target object in subsequent experiments.

Figure 5. Spectrograms of speech measured using the ECM and LDV with different irradiation targets. (a) clean speech recorded using the ECM. (b) speech measured using the LDV with an aluminum bottle. (c) speech measured using the LDV with a tissue box. (d) speech measured using the LDV with a bag of potato chips. (e) speech measured using the LDV with a glass cup. (f) speech measured using the LDV with an instant noodle cup. (g) speech measured using the LDV with a mug. (h) speech measured using the LDV with a paper cup. (i) speech measured using the LDV with a plastic bottle and (j) speech measured using the LDV with a speaker's throat.

Table 3. Results of WER, PESQ, and STOI for each irradiated object in Experiment 1.

		LDV
		Aluminum	Box	Bag of	Glass	Instant		Paper	Plastic
ECM		bottle	tissue	potato chips	cup	noodle cup	Mug	cup	bottle	Throat
WER	21.5	38.9	29.8	33.6	48.1	33.0	57.3	35.0	35.9	99.0
PESQ	–	1.81	2.34	1.38	1.43	1.82	1.06	1.60	1.95	1.03
STOI	–	0.89	0.88	0.86	0.78	0.86	0.76	0.87	0.83	0.32

4.2. Experiment 2: comparison of different positions of speakers, microphones, and irradiated objects

In Experiment 2, we changed the relative position between a speaker, the robot, and an irradiated object and analyzed its effect on the speech recognition performance and the sound recording quality. We conducted two experiments whose layouts are shown in Figure 6. The first experiment analyzed the relationship between the speaker and an object (see Section 4.2.1), while the second experiment focused on the relationship between the speaker and the microphone (see Section 4.2.2).

Figure 6. Equipment layout in Experiment 2. (a) Experiment described in Section 4.2.1 and (b) Experiment described in Section 4.2.2.

4.2.1. Performance evaluation of different distances between the speaker and the vibrating object

The purpose of the first evaluation was to determine the distance between a vibrating object and a loudspeaker, in which the speech can be accurately recognized using an optical laser microphone. As shown in Figure 6(a), the distance between the loudspeaker and the LDV was fixed to 5.5 m. The plastic bottles corresponding to vibrating objects were placed at intervals of 1 m between the loudspeaker and the LDV. The loudspeaker emitted a speech signal at about 80 dBA. We also placed another loudspeaker, which emitted white noise and fan noise at three different sound pressure levels of 65, 75, and 85 dBA near the LDV. The white noise has the same power in all frequency bands, while the fan noise has a strong power in lower frequency bands. This procedure made the given speech noisy.

Figure 7 shows the spectrograms of clean and noisy speech for each distance between the vibrating object and the loudspeaker used for reproducing the speech, and Table 4 shows the results of the WER, PESQ, and STOI. We found that as the plastic bottle moved farther from the speech source, the speech recognition rate and sound quality decreased. As shown in Figure 7(a–c), because the sound pressure level of the speech decreased due to distance attenuation, we can consider that the plastic bottle did not sufficiently vibrate. Furthermore, as shown in Figure 7(d–f), as the plastic bottle was farther away from the speech source and closer to the noise source near the LDV, the plastic bottle was more sensitive to the vibration due to the noise, which accelerated the decrease in speech recognition rate and sound quality. Focusing on the type and sound pressure level of noise, there was no significant difference in the results of PESQ and STOI. On the other hand, the fan noise tended to decrease the WER compared with the white noise. The reason is because the frequency components of fan noise were concentrated in the band specific to speech. From the above results, it is necessary for the LDV to measure the vibration of an object close to the speech source and far away from the noise source (at the lowest sound pressure level possible) to receive the speech in high quality.

Figure 7. Spectrograms of clean and noisy speech at different distances between the vibrating object and the loudspeaker for reproducing the speech. (a) Clean speech (0.5 m). (b) Clean speech (2.5 m). (c) Clean speech (4.5 m). (d) Noisy speech (0.5 m). (e) Noisy speech (2.5 m) and (f) Noisy speech (4.5 m).

Table 4. Results of WER, PESQ, and STOI for each distance between the irradiated object and the loudspeaker for reproducing the speech.

		Distance between the irradiated object and
		the loudspeaker
WER		0.5 m	1.5 m	2.5 m	3.5 m	4.5 m
Clean		33.9	46.9	53.7	54.2	58.9
	65 dBA	33.3	49.9	62.8	59.6	56.0
White	75 dBA	32.3	48.3	59.6	62.8	62.0
	85 dBA	32.7	56.3	73.7	80.5	87.6
	65 dBA	32.6	49.3	55.4	63.0	58.2
Fan	75 dBA	29.9	48.3	63.5	70.4	67.3
	85 dBA	33.9	60.5	73.7	91.5	95.0
		Distance between the irradiated object and
		the loudspeaker
PESQ		0.5 m	1.5 m	2.5 m	3.5 m	4.5 m
Clean		3.03	1.46	1.30	1.26	1.31
	65 dBA	2.80	1.35	1.23	1.20	1.25
White	75 dBA	2.80	1.35	1.23	1.19	1.22
	85 dBA	2.72	1.29	1.19	1.14	1.14
	65 dBA	2.81	1.35	1.24	1.17	1.26
Fan	75 dBA	2.79	1.35	1.22	1.17	1.23
	85 dBA	2.62	1.29	1.20	1.17	1.16
		Distance between the irradiated object and
		the loudspeaker
STOI		0.5 m	1.5 m	2.5 m	3.5 m	4.5 m
Clean		0.93	0.77	0.68	0.62	0.65
	65 dBA	0.93	0.76	0.67	0.60	0.64
White	75 dBA	0.93	0.76	0.65	0.60	0.63
	85 dBA	0.93	0.73	0.63	0.56	0.57
	65 dBA	0.93	0.76	0.67	0.60	0.64
Fan	75 dBA	0.93	0.75	0.65	0.59	0.62
	85 dBA	0.92	0.70	0.61	0.54	0.55

4.2.2. Performance evaluation for different distances between the speaker and the microphone

Setting the distance between the vibrating object and the loudspeaker for reproducing the speech to the best distance found in the first evaluation, the purpose of the second evaluation was to investigate the distance between the microphone and the loudspeaker, in which the speech can be accurately recognized using an air-conduction microphone or an optical laser microphone. Figure 6(b) shows the layout of the experimental equipment. The distance between the vibrating object and the loudspeaker for reproducing the speech was fixed at 0.5 m. The microphones ECM and LDV were placed at 2 m intervals from 1.5 to 5.5 m from the loudspeaker. We tested both a clean, noiseless environment and an extremely noisy environment. As shown in Figure 6(b), we prepared the noise source positions 1 to 3 and the microphone positions 1 to 3, and conducted experiments under 9 combinations of these positions (i.e. $3\times 3=9$). The reproduced sound pressure levels of the speech and the noise were the same as in Section 4.2.1.

Figure 8 shows the spectrograms of the noisy speech (white noise, 75 dBA) measured using ECM and LDV for each distance between the microphone and the loudspeaker, and Table 5 shows the results of the WER, PESQ, and STOI. The experiments demonstrated that the performance of the ECM became significantly worse when the noise source was close to the microphone. On the other hand, the LDV achieved better performance than the ECM in any conditions. However, its performance was also affected when the noise source was near the irradiated object. In a clean environment, i.e. a noiseless environment, the WER of speech measured using the LDV was lower than that of the ECM at each distance between the microphone and the loudspeaker. On the other hand, in an extremely noisy environment, all the WER, PESQ, and STOI decreased when the reproduced sound pressure of the noise was high for both microphones. In particular, when the ECM was used, all performance measures decreased significantly. The results of Figure 8 implied that the ECM can accurately measure the speech at a high sound pressure level (i.e. high SNR) when it is close to the loudspeaker. Similarly, when the LDV is close to the loudspeaker, the noise around the LDV make the plastic bottle vibrate strongly, which increases the noise mixed in the speech.

Figure 8. Spectrograms of noisy speeches at different distances between the microphone and the loudspeaker for reproducing the speech. The noise source was fixed near the microphone. (a) ECM (1.5 m). (b) ECM (3.5 m). (c) ECM (5.5 m). (d) LDV (1.5 m). (e) LDV (3.5 m) and (f) LDV (5.5 m).

Table 5. WER, PESQ and STOI results for each distance between the microphone and the loudspeaker for reproducing the speech.

		Distance between the microphone and the speech source
		1.5 m		3.5 m		5.5 m
WER		ECM	LDV	ECM	LDV	ECM	LDV
Clean		29.1	28.5	37.7	36.7	35.0	34.7
	65 dBA	32.5/27.3/28.8	27.5/30.9/31.9	42.3/42.0/41.1	35.3/38.2/37.0	35.9/38.0/49.7	38.1/36.6/37.3
White	75 dBA	36.0/33.2/31.7	33.7/29.9/31.6	46.6/45.8/41.2	37.7/37.2/35.4	43.7/46.7/79.3	38.8/35.3/40.7
	85 dBA	52.9/36.2/34.6	63.8/32.7/29.7	67.1/69.5/49.9	38.6/38.4/37.8	62.5/66.4/97.5	48.8/37.7/38.2
	65 dBA	29.3/27.2/30.4	30.8/28.1/27.0	39.9/42.2/40.8	38.4/36.5/35.9	40.0/37.8/40.5	37.3/35.5/36.4
Fan	75 dBA	35.2/31.9/27.4	37.7/28.5/30.4	46.6/42.3/43.1	36.9/40.5/35.0	41.6/40.0/82.5	35.5/38.7/37.1
	85 dBA	47.0/32.8/31.9	42.7/32.8/30.2	66.7/70.3/45.4	43.6/37.6/37.8	61.7/67.5/100.0	47.5/37.1/37.6
		Distance between the microphone and the speech source
		1.5 m		3.5 m		5.5 m
PESQ		ECM	LDV	ECM	LDV	ECM	LDV
Clean		1.83	2.45	1.34	1.77	1.30	2.31
	65 dBA	1.72/1.77/1.78	1.88/2.07/2.15	1.29/1.29/1.30	1.71/1.73/1.72	1.28/0.71/1.19	2.02/2.18/2.15
White	75 dBA	1.36/1.60/1.67	1.68/2.21/2.21	1.23/1.23/1.28	1.56/1.73/1.73	1.24/0.70/1.07	1.83/2.18/2.19
	85 dBA	1.10/1.27/1.38	1.21/1.99/2.05	1.09/1.08/1.15	1.40/1.69/1.70	1.13/0.64/1.05	1.42/2.09/2.09
	65 dBA	1.71/1.76/1.77	1.71/2.20/2.23	1.30/1.29/1.30	1.69/1.73/1.73	1.27/1.28/1.23	2.02/2.17/2.18
Fan	75 dBA	1.48/1.65/1.67	1.62/2.19/2.16	1.24/1.21/1.27	1.64/1.71/1.72	1.23/1.22/1.09	1.92/2.14/2.15
	85 dBA	1.24/1.46/1.47	1.46/2.08/2.06	1.12/1.09/1.15	1.43/1.70/1.70	1.13/1.11/1.07	1.24/1.97/2.00
		Distance between the microphone and the speech source
		1.5 m		3.5 m		5.5 m
STOI		ECM	LDV	ECM	LDV	ECM	LDV
Clean		0.88	0.91	0.68	0.80	0.72	0.87
	65 dBA	0.87/0.87/0.87	0.90/0.91/0.91	0.67/0.67/0.68	0.80/0.80/0.80	0.72/0.71/0.69	0.87/0.87/0.87
White	75 dBA	0.85/0.87/0.87	0.88/0.91/0.91	0.66/0.66/0.67	0.79/0.80/0.80	0.71/0.70/0.60	0.85/0.87/0.87
	85 dBA	0.76/0.83/0.84	0.73/0.90/0.90	0.59/0.58/0.63	0.76/0.79/0.79	0.66/0.64/0.50	0.79/0.87/0.87
	65 dBA	0.87/0.87/0.87	0.85/0.91/0.91	0.67/0.66/0.67	0.79/0.80/0.80	0.71/0.72/0.69	0.85/0.87/0.87
Fan	75 dBA	0.83/0.86/0.86	0.82/0.90/0.90	0.64/0.64/0.66	0.78/0.80/0.80	0.69/0.69/0.57	0.83/0.86/0.87
	85 dBA	0.76/0.82/0.82	0.77/0.89/0.89	0.57/0.56/0.60	0.75/0.79/0.79	0.62/0.60/0.43	0.69/0.84/0.85

Note: For each combination, we show three results of the following different distances between the loudspeaker and the noise source: 1.5 m, 3.5 m, 5.5 m.

We should emphasize that no post-processing was applied to the ECM results, such as noise and reverberation techniques, which are usually used in conventional speech recognition applications. In general, the better the quality of original recordings is, the further these techniques improve the recognition results. Thus, using sophisticated speech post-processing approaches has the potential to improve not only the ECM's recorded speech but also the LDV's recorded speech. Such a comprehensive analysis should be included in our future work.

4.3. Experiment 3: comparison between the LDV and ECM in a domestic environment

In Experiment 3, we compared the performance of our optical laser microphone with the conventional gunshot microphone in a kitchen room assuming a scenario in which a speaker and a robot interact at home. Figure 9 shows the layout of the experimental equipment. We reproduced the situation in which a speaker, who stood at different positions s1 to s5, spoke to a robot at the position R. We measured the speech using both the LDV and ECM. To test the LDV, we placed nine plastic bottles as vibrating objects at 1 m intervals in all directions and detected the best object, i.e. with the highest confidence score, using the proposed approach in Section 3.2. In this experiment, we prepared the following two noise types: (i) the sound of flowing water in a sink with a sound pressure level of about 70 dBA, which corresponds to usual housework [38], such as washing hands and dishes, and (ii) the noise generated by home appliances around the robot. Specifically, the latter includes white noise and fan noise with three sound pressure levels of 65, 75, and 85 dBA. For example, we denote the case of adding white noise at 85 dBA with flowing water by ‘White(85)+F.’ The loudspeaker emitted the noise toward the center of the room as shown in Figure 10.

Figure 9. Equipment layout.

Table 6 shows the WER, PESQ, and STOI for each position of the loudspeaker that emits the speech. An overall trend was that the speech measurement of the LDV yielded better WER, PESQ, and STOI than that of the ECM. For example, in extremely noisy environments such as ‘White(85)+F’ and ‘Fan(85)+F,’ the LDV obtained about 33% of WER, while the ECM produced over about 90% of WER, implying that the ECM was sensitive to noise. The WER, PESQ, and STOI also got worse as the sound pressure level of the noise increased. Table 6 implied that even though the amount of water flowing was normal and equivalent to usual housework, it disturbed the sound recording of the ECM. On the other hand, when using the LDV, the WER ranged from 30.2 to 34.8% in all noisy environments, indicating that the speech measurement of the LDV was robust to noise compared to the ECM. As shown in Figure 9, the distance between the microphone and the loudspeaker becomes farther in the order of s3 and s5, s2 and s4, and s1. Table 6 demonstrated that the larger the distance between the ECM and the loudspeaker is, the worse the WER, PESQ, and STOI became. This tendency was consistent even when water was flowing. On the other hand, the LDV's results did not depend on a distance change because our method always irradiated an object near the loudspeaker. These results confirmed the effectiveness of using an optical laser microphone for a domestic robot that usually works in a noisy environment.

Figure 10. Recording scene.

Table 6. WER, PESQ and STOI results for each position of the loudspeaker that emits speech.

	s1		s2		s3		s4		s5
WER	ECM	LDV	ECM	LDV	ECM	LDV	ECM	LDV	ECM	LDV
Clean	36.0	33.7	39.2	32.1	37.9	32.1	39.2	32.1	37.9	32.1
White (65)	41.1	34.5	39.8	31.1	38.7	31.2	43.3	30.9	40.5	31.6
White (65) + F	88.1	34.1	90.3	30.2	68.7	31.9	88.3	34.1	67.9	34.0
White (75)	51.0	32.3	51.7	32.3	41.8	30.9	50.6	33.2	43.5	32.3
White (75) + F	89.8	34.7	92.1	32.8	71.9	30.3	92.1	32.4	70.5	31.8
White (85)	76.1	34.3	74.6	32.9	65.5	32.3	76.2	31.8	65.5	32.8
White (85) + F	95.8	33.7	94.7	30.6	84.8	31.3	94.8	31.0	83.2	32.5
Fan (65)	37.7	33.3	37.7	32.1	38.4	32.2	38.4	30.7	36.7	30.2
Fan (65) + F	85.8	33.8	88.8	31.4	67.2	32.3	85.5	32.2	71.6	31.8
Fan (75)	43.4	34.8	44.7	31.3	38.6	31.9	43.1	32.9	43.7	33.2
Fan (75) + F	90.3	32.1	91.0	33.4	70.8	32.2	93.4	32.4	71.0	32.7
Fan (85)	77.5	33.5	81.5	31.0	57.7	31.6	77.6	30.9	59.7	30.2
Fan (85) + F	99.2	33.3	97.9	32.9	86.3	32.2	98.9	33.1	88.2	32.6
	s1		s2		s3		s4		s5
PESQ	ECM	LDV	ECM	LDV	ECM	LDV	ECM	LDV	ECM	LDV
Clean	1.33	2.01	1.28	2.49	1.34	2.49	1.28	2.49	1.34	2.49
White (65)	1.30	2.01	1.26	2.49	1.31	2.49	1.26	2.49	1.32	2.49
White (65) + F	1.09	1.91	1.11	2.41	1.11	2.42	1.08	2.41	1.11	2.46
White (75)	1.22	2.02	1.19	2.49	1.25	2.48	1.19	2.49	1.25	2.49
White (75) + F	1.09	1.94	1.13	2.42	1.11	2.42	1.08	2.42	1.11	2.47
White (85)	1.12	2.01	1.11	2.48	1.15	2.49	1.10	2.48	1.15	2.49
White (85) + F	1.07	1.95	1.07	2.43	1.09	2.43	1.07	2.42	1.09	2.47
Fan (65)	1.31	2.01	1.27	2.49	1.33	2.49	1.27	2.48	1.32	2.49
Fan (65) + F	1.09	1.91	1.08	2.41	1.11	2.42	1.08	2.42	1.11	2.46
Fan (75)	1.25	2.02	1.22	2.49	1.28	2.48	1.22	2.49	1.28	2.49
Fan (75) + F	1.14	1.95	1.08	2.42	1.11	2.42	1.08	2.41	1.10	2.47
Fan (85)	1.11	2.00	1.10	2.48	1.15	2.48	1.10	2.48	1.15	2.48
Fan (85) + F	1.06	1.95	1.07	2.41	1.08	2.42	1.06	2.42	1.08	2.47
	s1		s2		s3		s4		s5
STOI	ECM	LDV	ECM	LDV	ECM	LDV	ECM	LDV	ECM	LDV
Clean	0.68	0.85	0.64	0.91	0.66	0.91	0.64	0.91	0.66	0.91
White (65)	0.68	0.85	0.64	0.91	0.65	0.91	0.64	0.91	0.65	0.91
White (65) + F	0.57	0.85	0.54	0.90	0.58	0.90	0.55	0.90	0.59	0.90
White (75)	0.65	0.85	0.62	0.91	0.64	0.91	0.62	0.90	0.64	0.91
White (75) + F	0.56	0.85	0.54	0.90	0.58	0.90	0.54	0.90	0.58	0.90
White (85)	0.60	0.85	0.58	0.91	0.61	0.91	0.57	0.91	0.61	0.91
White (85) + F	0.54	0.84	0.52	0.90	0.56	0.90	0.52	0.90	0.57	0.90
Fan (65)	0.68	0.85	0.63	0.91	0.65	0.91	0.64	0.91	0.65	0.91
Fan (65) + F	0.57	0.85	0.54	0.90	0.58	0.90	0.55	0.90	0.58	0.90
Fan (75)	0.65	0.85	0.61	0.91	0.64	0.91	0.61	0.91	0.63	0.91
Fan (75) + F	0.56	0.85	0.53	0.90	0.58	0.90	0.54	0.89	0.58	0.90
Fan (85)	0.55	0.85	0.52	0.90	0.57	0.90	0.53	0.90	0.57	0.90
Fan (85) + F	0.51	0.84	0.49	0.90	0.54	0.89	0.49	0.90	0.54	0.89

Note${}^{\ast }$ ( ): Sound pressure level [dBA], F: Flowing water.

5. Conclusion

This paper presented a new speech measurement method using an optical laser microphone to realize spoken dialogue for service robots. The proposed method makes the robots' speech recognition systems robust to the noise around the microphone. The results of the experiments conducted in various noisy environments showed that the recognition of speech measured with a optical laser microphone achieved a higher WER, PESQ, and STOI than that obtained using a conventional air-conduction microphone.

There are several research directions to further develop our robot. Although the evaluations in this paper assumed that the proposed system can accurately detect an object irradiated by the laser, it is necessary to investigate advanced object recognition techniques and analyze the relationship between object recognition performance and speech recording quality. Besides, our experiments used the Google Speech-to-Text API whose recognition model was trained on speech recorded with an air-conduction microphone. We should compare it with a model trained using speech recorded using the optical laser microphone. We will also implement several functions on our robot, such as to help a user to recognize the distant-talking speech and sound events in service environments.

Disclosure statement

No potential conflict of interest was reported by the author(s).

Additional information

Funding

This research was partly supported by Ritsumeikan Global Innovation Research Organization (R-GIRO).

Notes on contributors

Takahiro Fukumori

Takahiro Fukumori received his B.E., M.E., and Ph.D. degrees from Ritsumeikan University in 2010, 2012, and 2015, respectively. From 2012 to 2015, he was a JSPS research fellowship for young scientists (DC1). From 2015 to 2020, he was an assistant professor at Ritsumeikan University. He is currently a lecturer at Ritsumeikan University. His current research interests include speech recognition and speech enhancement. He is a member of IEEE, RSJ, ASJ, IPSJ, and IEICE.

Chengkai Cai

Chengkai Cai received the M.E. from the Chiba Institute of Technology, Japan, in 2018. He is currently a Ph.D. student at the Graduate School of Information Science and Engineering, Ritsumeikan University. His current research interests include speech enhancement for optical laser microphone. He is a member of ASJ.

Yutao Zhang

Yutao Zhang received his M.E. degree from the Ritsumeikan University, Japan, in 2019. He is currently a Ph.D. student at the Graduate School of Information Science and Engineering, Ritsumeikan University. His current research interests include acoustic event detection and speech recognition. He is a member of ASJ.

Lotfi El Hafi

Lotfi El Hafi received his M.S.E. from the Université catholique de Louvain, Belgium, and his Ph.D. from the Nara Institute of Science and Technology, Japan, in 2013 and 2017, respectively. His thesis focused on STARE, a novel eye-tracking approach that leveraged deep learning to extract behavioral information from scenes reflected in the eyes. Today, he works as a Research Assistant Professor at the Ritsumeikan Global Innovation Research Organization, as a Specially Appointed Researcher for the Toyota HSR Community, and as a Research Advisor for Robotics Competitions for the Robotics Hub of Panasonic Corporation. His research interests include service robotics and artificial intelligence. He is also the recipient of multiple research awards and competition prizes, an active member of RSJ, JSME, and IEEE, and the President & Founder of Coarobo GK.

Yoshinobu Hagiwara

Yoshinobu Hagiwara received his Ph.D. degree from Soka University, Japan, in 2010. He was an Assistant Professor at the Department of Information Systems Science, Soka University, from 2010, a Specially Appointed Researcher at the Principles of Informatics Research Division, National Institute of Informatics, from 2013, and an Assistant Professor at the Department of Human & Computer Intelligence, Ritsumeikan University, from 2015. He is currently a Lecturer at the Department of Information Science and Engineering, Ritsumeikan University. His research interests include human-robot interaction, machine learning, intelligent robotics, and symbol emergence in robotics. He is also a member of IEEE, RSJ, IEEJ, JSAI, SICE, and IEICE.

Takanobu Nishiura

Takanobu Nishiura received his M.E. and Ph.D. degrees from Nara Institute of Science and Technology, in 1999 and 2001, respectively. From 2001 to 2004, he was an Assistant Professor at Wakayama University. From 2004 to 2014, he was an Associate Professor at Ritsumeikan University. Since 2014, he is a Professor at Ritsumeikan University. He has been engaged in research on acoustic signal processing, acoustic field reproduction, noise control, parametric loudspeaker, visual microphone, and optical laser microphone. He is also a member of IEEE, IEEJ, IEICE, IPSJ, VRSJ and ASJ.

Tadahiro Taniguchi

Tadahiro Taniguchi received his M.E. and Ph.D. degrees from Kyoto University, in 2003 and 2006, respectively. From 2005 to 2008, he was a Japan Society for the Promotion of Science Research Fellow in the same university. From 2008 to 2010, he was an Assistant Professor at the Department of Human and Computer Intelligence, Ritsumeikan University. From 2010 to 2017, he was an Associate Professor in the same department. From 2015 to 2016, he was a Visiting Associate Professor at the Department of Electrical and Electronic Engineering, Imperial College London. Since 2017, he is a Professor at the Department of Information and Engineering, Ritsumeikan University, and a Visiting General Chief Scientist at the Technology Division of Panasonic Corporation. He has been engaged in research on machine learning, emergent systems, intelligent vehicle, and symbol emergence in robotics.

Notes

1 http://www.robocupathome.org/

2 https://webrtc.org

3 https://cloud.google.com/speech-to-text

4 This study was carried out following Ritsumeikan University Research Ethics Guidelines. The laser corresponds to Class 2, which is safe for humans. An ethical review process was not required for this study because none of the conditions for such a review based on the checklist provided by the Research Ethics Committee of Ritsumeikan University were satisfied. All participants gave written informed consent.

5 https://github.com/RoboCupAtHome/Montreal2018

6 Although we confirmed the safety of the laser beam, the speakers wore light-shielding glasses to protect their eyes during the experiments.