Objective evaluation of mechanical expressiveness in android and human faces

ABSTRACT

Extensive research has been conducted to realize highly expressive android faces. In contrast, not much is being discussed which android faces are mechanically better and how advanced each android is because a fair and meaningful standard index of expressiveness does not exist. Therefore, in this study, we propose a numerical index and three types of visualization methods for facial deformation expressiveness to investigate the mechanical performance of android faces compared to humans. We used a data set of three-dimensional displacement vectors of over 100 facial points on the faces of two androids and three human males to calculate the expressiveness that reflects how diversely each facial point could move. As a result, we found that the expressiveness of the androids was significantly lower than humans, especially in the lower faces. The expressiveness of the androids was approximately only 20$\mathrm{\%}$ that of humans even in the most expressive axis and two orders of magnitude lower than humans in three-dimensional evaluation. Thus, our method revealed the lackness of mechanical performance in recently developed androids to move facial skin widely. This method enabled us to reasonably discuss future directions to realize next generation androids with expressiveness similar to that in humans.

GRAPHICAL ABSTRACT

KEYWORDS:

• Android robot
• facial expression
• facial deformation
• evaluation index
• motion capture

1. Introduction

Improving expressiveness on a face is crucial in android robotics because the deformable facial surface for communication distinguishes android robots from other humanoid robots. Multiple research teams have attempted to generate improved expressive android faces using ingenious facial mechanisms [1–19] considering the social effects of androids can be enhanced by improving the motion performance on a face, a remarkable deformable display device of emotional and social cues. For example, androids exhibit stronger emotional states if they move their face more widely. Furthermore, they can distinctly communicate several emotions and social signals by clearly generating different facial movements.

However, not much research has been conducted to determine which android faces are mechanically better, and how advanced each android is in terms of expressiveness on the basis of a fair and meaningful standard index. Traditionally, the number of actuators installed in an android face and their locations are often highlighted and compared among different androids [10,13,18–21]. This comparison is helpful to discuss which facial deformation patterns, such as action units defined in the Facial Action Coding System [22], android designers intend to install. However, this is insufficient to evaluate how much these patterns are realized on the facial surface because skin deformations are determined not only by the number and locations of actuators but also by the soft skin properties, such as the thickness, material, and elasticity, as pointed out by Tadesse and Priya [15]. Expressiveness should be evaluated based on the realized surface deformations because humans observe the patterns of the surface deformations produced by the hidden actuators and spread on the facial surface. Moreover, even when the realized deformation patterns are the same with different androids, the android with fewer actuators should be evaluated more than the others from an efficiency point of view when the motion performance is the same, as suggested by Lin et al. [18]. Although impression evaluation has also been adopted for androids [3,7,8,10,11,17,18,23,24], it is insufficient to determine the index of expressiveness given that the variations in the evaluation value are considerable among human evaluators, making it difficult to specify the problematic facial regions where expressiveness is low. In addition, numerically comparing the motions and shape changes of facial surface among android and human faces [9,10,16,17,25–27] is considered effective in discussing the similarity, or human likeness, in each facial expression, but not the expressiveness that each face equips.

To effectively improve facial expressiveness in androids, a numerical index of expressiveness would be required to fairly and strictly examine the amount of expressiveness each face and facial region equips. Based on this expressiveness index, we can investigate the achievement level of expressiveness in recently developed androids compared to the expressiveness of human faces (e.g. we can say ‘the facial expressiveness of this android is 80$\mathrm{\%}$ that of human on average’). Furthermore, we can determine ways to improve the facial deformation mechanism of each android and examine their insufficiency of expressiveness (e.g. we can say ‘an additional facial mechanism as this is necessary to improve the expressiveness by 10$\mathrm{\%}$ near the mouth region’).

However, it is difficult to define expressiveness for each region or point on every face. Considering the facial expressiveness for humans and androids is often mentioned as ‘this face is very expressive’ or ‘this new android is more expressive than the others,’ facial expressiveness is a familiar perspective with respect to the motion performance of faces. Additionally, there is no clear and practical definition of the expressiveness of faces in the form of display devices that convey a wide variety of information through surface deformation. Although the intensity of facial motions, which has been examined extensively [11,14,15,26], should be considered as a component of facial expressiveness, it does not reflect the repertoire of facial motions. Similarly, the realisticity of facial motions as human facial expressions are essential features of androids but do not reflect the repertoire of facial motions. The number of typical facial motions an android face can produce appears as an appropriate representation of the repertoire of facial motions [1,4,7,8,10,12,28]. However, the expressiveness should be low when several motions in the repertoire are similar.

Therefore, in this study, we define facial expressiveness based on the intensity and similarity of the repertoire of facial motions. Then, we propose a calculation method to obtain a numerical index value for the expressiveness of facial deformation, which can be adopted by both androids and humans. Furthermore, we developed three types of visualization methods, i.e. expressiveness map, expressiveness plot, and representative expressiveness octahedron, to investigate the property of expressiveness for each face. We evaluated a total of five faces, which includes two android faces and three human faces, in the proposed index to verify the effectiveness of the index by comparing their mechanical performance and efficiency, specifying the expressive/inexpressive facial regions, and determining ways to improve android faces.

2. Method

2.1. Requirements and concept for determining the expressiveness index

In this section, we discuss the six requirements for determining the expressiveness index of a face.

1. The index should not be influenced by the static properties of the facial surface: Static properties such as size, shape, and skin color should not influence the evaluation of expressiveness in order to precisely evaluate the performance of the facial movement.

2. The index should be fairly applicable to any facial mechanism: This is necessary to compare the expressiveness among androids having different facial mechanisms, including actuators, transmissions, and skin materials.

3. The index should be applicable both to androids and humans: This is necessary to evaluate the absolute achievement level of android technologies.

4. The index should be independent of human likeness: This is crucial in order to accept androids whose facial motions are not human-like, but expressive.

5. The index should be applicable to all facial regions: This is necessary to specify the expressive/inexpressive regions on faces.

6. The index should be useful for investigating the expressiveness from different perspectives: This is necessary to determine the improvement direction of the expressiveness in androids.

Based on the above requirements, we propose the expressiveness index, which is the expressiveness is evaluated at all facial points as the maximum length, area, and volume of the approximate octahedron of movable regions of the facial point. This index satisfies the first four requirements mentioned above considering it focuses on the variations of facial skin movements produced on facial surfaces of androids and humans. Furthermore, the fifth requirement was satisfied by measuring the movements of multiple skin points on android and human faces. Moreover, to satisfy the sixth requirement, we evaluated the movement variations of a skin point in three-dimensional orthogonal evaluation axes corresponding to the three orthogonal diagonals of the approximate octahedron. Details of the expressiveness index are explained in Section 2.3.

2.2. Data

The proposed expressiveness was calculated according to the variations in the three-dimensional movements of each facial skin point using the data set of three-dimensional displacement vectors of over 100 skin points in the right-half faces of two androids (child- and female-type) and three human male adults, measured by using an optical motion capture system in a previous study [27]. Figure 1 shows the appearance of the androids and locations of the measurement points (silver markers) on faces of child-type android (left: Affetto [21]), female-type android (middle: A-lab Co., Ltd., A-lab Female Android Standard Model), and a human male adult (right). The child- and female-type androids are different in the number of degrees of freedom (DoFs) for facial expression, especially in the lower face: the child-type android has 15 DoFs (5 in the upper face and 10 in the lower face) for testing various facial expressions in exploratory research, whereas the female-type android has 8 DoFs (5 in the upper face and 3 in the lower face) for reducing its price in the market. The previous study [27] measured the displacements of the facial points by tracking the three-dimensional positions of hemispherical infrared reflection markers with a 3 mm diameter with six infrared cameras (OptiTrack Flex13). These markers were attached to the right halves of the faces at intervals of approximately 10 mm, which was the minimum length with which the motion capture system could track the markers correctly.

Figure 1. Locations of the measurement points, namely infrared reflective markers for the motion capturing, on faces of child-type android (left), female-type android (middle), and a human (right) (modified from [27]).

The differences in the shape and size of three adult males and two androids were compensated using thin-plate spline warping [29], a non-linear smooth transformation of multivariate data, to appropriately compare the expressiveness of each facial region among these faces. More precisely, the geometric arrangements of facial parts were normalized so that the nine representative facial points (the nose root, outer and inner corners of the eyes, top of the nose, earlobe root, corners of the mouth, tops of the upper and lower lips, and top of the chin) matched among these five faces. Hereafter, we refer to the child- and female-type androids as A1 and A2, and the three humans as H1, H2, and H3.

The displacement vectors were measured for almost every independent movement unit of the androids and humans. The displacement vectors of A1 and A2 were measured for 15 and 8 types of independent facial movements, each produced to their limits by an individual actuator, respectively. Although the android A1 and A2 have 16 and 9 movement units (degrees of freedom) produced by 16 and 9 actuators in total, we excluded one facial movement used to open the jaw intensively, considering the skin movements associated with these movements were exceptionally extensive. On the other hand, 43 independent facial movements were examined for H1, H2, and H3, who were instructed to exhibit each facial movement with their maximum effort. We chose these movements from 44 independent movements that have been defined in the Facial Action Coding System [22]. The movement excluded here was the one to open the jaw. Therefore, we obtained a set of 15, 8, and 43 displacement vectors for each facial point in the child-type android (A1), female-type android (A2), and three humans (H1, H2, and H3) faces, respectively. Tables 1, 2, and 3 summarized the measured movements in A1, A2, and H1, H2, and H3, respectively. The numbers for the facial movements listed in Table 3 correspond to the numbers defined in the Facial Action Coding System as action units.

Table 1. Measured movements in the child-type android (A1).

Number	Description
1	Lowering the upper eyelid
2	Turning the eye pit upwards
3	Raising the lower eyelid
4	Raising an eyebrow
5	Centering the eyebrows
6	Raising the middle cheek
7	Pulling the middle cheek to the side
8	Pulling the lower cheek to the side
9	Lowering the corner of the mouth
10	Centering the upper lip
11	Centering the lower lip
12	Lowering the nose
13	Pulling the corner of the mouth
14	Lowering the lower lip
15	Raising the corner of the mouth

Table 2. Measured movements in the female-type android (A2).

Number	Description
1	Lowering the upper eyelid
2	Turning the eye pit upwards
3	Raising the lower eyelid
4	Raising the outer eyebrow
5	Raising the inner eyebrow
6	Raising the cheek
7	Centering the upper lip
8	Pulling the corner of the mouth

Table 3. Measured facial movements in humans (H1, H2, and H3).

Number	Description	Number	Description
1	Inner brow raiser	24	Lip pressor
2	Outer brow raiser	25	Lip part
4	Brow lowerer	26	Jaw drop
5	Upper lid raiser	28	Lips suck
6	Cheek raiser	29	Jaw thrust
7	Lid tightener	30	Jaw sideways
8	Lips toward each other	31	Jaw clencher
9	Nose wrinkler	32	Lip bite
10	Upper lip raiser	33	Cheek blow
11	Nasolabial furrow deepener	34	Cheek puff
12	Lip corner puller	35	Cheek suck
13	Cheek puffer	36	Tongue bulge
14	Dimpler	37	Lip wipe
15	Lip corner depressor	38	Nostril dilator
16	Lower lip depressor	39	Nostril compressor
17	Chin raiser	41	Lid droop
18	Lip puckerer	42	Slit
19	Tongue out	43	Eyes closed
20	Lip stretcher	44	Squint
21	Neck tightener	45	Blink
22	Lip funneler	46	Wink
23	Lip tightener

This set of displacement vectors on each point reflect the diversity in the movement of each point, that is, they reflect the expressiveness of each point on a face. We defined the jth displacement vector of the ith skin point as ${\mathit{d}}_{i,j}(i=1,\dots ,N;j=1,\dots ,M)$, where N and M are the number of skin points and facial movements measured for individual androids and humans, respectively.

2.3. Calculation of the expressiveness

2.3.1. Evaluation axis

We defined a peculiar three-dimensional coordinate system for each facial point (hereafter called an ‘expressiveness coordinate system’) to calculate its expressiveness to analyze the peculiar variations in its displacement vectors. As shown in Figure 2, the expressiveness coordinate system was defined as the orthogonal coordinate system whose first, second, and third axes are consistent with the first, second, and third principal components of the end points of the displacement vectors, respectively. The initial positions of a skin point and its displacement vectors are represented as a black dot and black allows in this figure.

Figure 2. Expressiveness coordinate system defined according to the displacement vectors on a skin point.

Here, the principal components are the orthogonal axes that pass through the average point of the displacement vector endpoints. The first principal component is the axis along which the variance of the vector endpoints is maximized, whereas the second principal component is the axis orthogonal to the first principal component. The plane defined with the first and second principal components maximize the variance of the vector endpoints. The third principal component is the axis normal to the plane.

The first, second, and third axes of the expressiveness coordinate system are referred to as the first, second, and third expressiveness axes, respectively, which can be used to investigate the variations in facial movements at each facial point. For example, different types of facial movements are differentiated most prominently along the direction of the first expressiveness axis. Additionally, based on the contribution ratio of the first principal component, we investigated whether the distribution vectors were distributed one-dimensionally or multi-dimensionally.

2.3.2. Expressiveness octahedron

The existence region of the displacement vectors was approximated with an octahedron, referred to as the ‘expressiveness octahedron,’ whose three diagonals corresponded to the expressiveness axes. We defined the diagonal lengths along with the first, second, and third expressiveness axes of the ith skin point as ${}^1{l}_i$, ${}^2{l}_i$, and ${}^3{l}_i$, respectively. Furthermore, we determined these lengths as the distribution widths of the displacement vectors in each expressiveness axis, as shown in Figure 3.

Figure 3. The expressiveness octahedron.

The distribution widths were calculated based on the principal component scores of the displacement vectors, defined as the value of each displacement vector projected onto each expressive axis, namely, the principal component. We defined the principal component score of the displacement vector ${\mathit{d}}_{i,j}$ against the first, second, and third principal components as ${}^1{p}_{i,j}$, ${}^2{p}_{i,j}$, and ${}^3{p}_{i,j}$, respectively. Then, we calculated ${}^1{l}_i$, ${}^2{l}_i$, and ${}^3{l}_i$ as the differences between the maximum and minimum principal component scores as: $\{\begin{array}{l}{\displaystyle {}^1{l}_i=\underset{j}{max}{}^1{p}_{i,j}-\underset{j}{min}{}^1{p}_{i,j}}\\ {\displaystyle {}^2{l}_i=\underset{j}{max}{}^2{p}_{i,j}-\underset{j}{min}{}^2{p}_{i,j}}\\ {\displaystyle {}^3{l}_i=\underset{j}{max}{}^3{p}_{i,j}-\underset{j}{min}{}^3{p}_{i,j},\mathrm{r}\mathrm{e}\mathrm{s}\mathrm{p}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{v}\mathrm{e}\mathrm{l}\mathrm{y}.}\end{array}$ Figure 4 shows examples of the expressiveness octahedron for five facial skin points (black dots) with different variations of displacement vectors (black arrows). It can be seen that the shape and size of the expressive octahedron are different depending on the existing regions of the displacement vectors, that is, the octahedron size is small when all displacement vector are small (Figure 4(a)), or when every displacement vector is similar (Figure 4(b)). Conversely, the octahedron size is large when the displacements are large and different in one, two, and three directions (Figure 4(c–e), respectively). Note that the octahedrons in Figure 4(a–d) were depicted as almost flat.

Figure 4. Examples of expressiveness octahedrons.

2.3.3. 1D, 2D, and 3D expressiveness

We calculated three different types of expressiveness, one-dimensional (1D), two-dimensional (2D), and three-dimensional (3D) expressiveness, for each point depending on their expressiveness octahedron. The expressiveness can be calculated from the lengths of the three expressiveness octahedron diagonals. The 1D expressiveness of ith skin point was calculated as the diagonal length along with the first expressiveness axis, namely ${}^1{l}_i$. The 2D expressiveness was calculated as the area of the expressiveness octahedron in the plane defined by the first and second expressiveness axes, namely ${(}^1{l}_i{}^2{l}_i)/2$. Lastly, the 3D expressiveness was calculated as the volume of the expressiveness octahedron, namely ${(}^1{l}_i{}^2{l}_i{}^3{l}_i)/6$.

2.4. Visualization methods for expressiveness

We proposed three different types of visualization methods of 1D, 2D, and 3D expressiveness: the ‘expressiveness map,’ ‘expressiveness profile,’ and the ‘representative expressiveness octahedron.’ The expressiveness map is a spatial distribution plot of the expressiveness on a face, which is illustrated as a two-dimensional color map. The expressiveness profile, which is presented as a line plot, is a compressed representation of the spatial distribution of expressiveness along with the facial midline. The representative expressiveness octahedron, drawn as a single octahedron, is a more compressed representation of the expressiveness in a facial region.

First, the spatial resolution of expressiveness was enhanced to obtain an expressiveness map having a higher resolution on a face. Furthermore, we performed Natural neighbor interpolation [30] on the displacement vectors to calculate lattice point data at intervals of 0.5 mm on the coronal plane, which is a two-dimensional plane that cuts the head into the front and back halves. We defined the coronal plane as the x–y plane, where the origin is the chin top and the y-axis is the axis direction towards the top of the head. As a result, the y-axis corresponds to the facial midline in the expressiveness map.

Second, we calculated the averaged values of the expressiveness in each y position to obtain the expressiveness profile. Because the averaged values of the expressiveness are different along the facial midline, the line plot of the averaged values along with the y-axis showed which facial regions having higher expressiveness, such as the forehead, eye, cheek, mouth, and jaw regions.

Lastly, we calculated the average expressiveness in facial regions to obtain a representative expressiveness octahedron. By drawing a single octahedron for each facial region based on the region's averaged expressiveness, we were able to obtain the regional characteristics of expressiveness intuitively.

3. Results

3.1. Displacement vectors

Figure 5 shows the variations of the displacement vectors on each facial point for A1 and A2 and H1, H2, and H3 on the x–y plane. The displacement vectors for measured facial movement patterns were superimposed in these figures. Each arrow represents each displacement vector, and the colors indicate that the displacements were measured for the same facial movement. At the jaw, mouth, cheek, nose, eye, and forehead regions, y = 10, 30, 50, 70, 100, and 120 mm, respectively. The vertical line x = 0 was approximately the facial midline.

Figure 5. Variations of displacement vectors in the two androids (A1 and A2) and three humans (H1, H2, and H3).

It can be seen how the displacement vectors vary at each facial point. For example, the vector lengths were observed to be longer in the forehead region (around y = 120) compared to the cheek regions (around y = 50), whereas the vector orientations were similar in the forehead regions than the mouth regions (around y = 30). Additionally, we can determine the difference in the variations of the displacement vectors among androids and humans based on the fact that the vector lengths were prominently longer in humans than androids.

3.2. Expressiveness axes

Figure 6 shows the directions of the expressiveness axes on each facial point of A1 and A2 and H1, H2, and H3 on the x–y plane. The first, second, and third expressiveness axes are indicated as black, red, and blue line segments, respectively. The lengths of these line segments were determined based on the three diagonal lengths of the expressiveness octahedrons at each facial point. The heat maps show the contribution ratio of the first principal components. The red and yellow regions indicate that the ratios were approximately 1 and below 0.7, respectively, in these regions.

Figure 6. Spatial distributions of the contribution ratio of the first principal components in two androids (A1 and A2) and three humans (H1, H2, and H3).

We evaluated whether the ratio was approximately one in the forehead region in both androids and humans. Results showed that the endpoints of the displacement vectors were distributed on a straight line at each facial point in this region. In other words, different movements of the skin point were expressed almost one-dimensionally. Conversely, the ratio was below 0.7 in several regions, especially around the mouth, thereby indicating that different movements were expressed two- or three-dimensionally in these regions.

3.3. Expressiveness map

Figures 7, 8, and 9 show the 1D, 2D, and 3D expressiveness maps, respectively. The line segments depicted on the facial points represent the diagonals of expressiveness octahedrons used to calculate the expressiveness at each facial point. For example, the diagonals of the first expressiveness axes were depicted in the 1D expressiveness map, whereas that of the first, second, and third expressiveness axes were in the 3D expressiveness map.

Figure 7. 1D expressiveness maps of androids (A1 and A2) and humans (H1, H2, and H3).

Figure 8. 2D expressiveness maps of androids (A1 and A2) and humans (H1, H2, and H3).

Figure 9. 3D expressiveness maps of androids (A1 and A2) and humans (H1, H2, and H3).

The spatial distributions of the calculated expressiveness were depicted as color maps, where the orange and blue regions represent the higher and lower expressiveness regions in a face, respectively. The ranges of these color maps were different for different faces, that is, the maximum value of each color bar was determined based on the maximum expressiveness score in each face, which is summarized in Table 4.

Table 4. Maximum expressiveness score in each face.

	Maximum Expressiveness
Face	1D [mm]	2D [mm${}^2$]	3D [mm${}^3$]
A1	16.3	25.1	4.3
A2	12.1	4.8	1.1
H1	21.7	152.3	383.7
H2	21.1	208.5	565.7
H3	26.4	194.8	640.2

From the figures, we can see which facial regions are more expressive in each face. For example, the upper eyelid region (around y = 110) was most expressive in the androids (A1 and A2) in the 1D expressiveness map. Conversely, the mouth regions (around y = 30) were the most expressive regions in humans (H1, H2, and H3) in the 1D expressiveness map. Additionally, it is noteworthy that a common blue region with a relatively low 1D expressiveness was observed in the central region of the human faces (H1, H2, and H3), which was identified as the continuous region comprising the nose, lower eye, and outer eye corner regions, located at approximately $70<y<100$.

Furthermore, we observed that the most expressive regions were not always the same among the 1D, 2D, and 3D expressiveness maps, despite in the same face. For example, the upper eye region was most expressive in A2 in the 1D expressiveness map, whereas the upper forehead region was the most expressive in the 3D expressive map.

3.4. Expressiveness profile

Figures 10, 11, and 12 show the 1D, 2D, and 3D expressiveness profiles, respectively. The horizontal axis indicates the position along the y-axis, whereas the vertical axis indicates the average expressiveness scores in each y position. These graphs also depict the standard deviations of expressiveness. Figures 11 and 12 are semilogarithmic graphs.

Figure 10. 1D expressiveness profiles of androids (A1 and A2) and humans (H1, H2, and H3).

Figure 11. 2D expressiveness profiles of androids (A1 and A2) and humans (H1, H2, and H3).

Figure 12. 3D expressiveness profiles of androids (A1 and A2) and humans (H1, H2, and H3).

While the expressiveness profiles among androids and humans were similar, they were different between humans and androids, especially the lower face regions, whose y positions were approximately below 60. The 1D, 2D, and 3D expressiveness profiles of humans exhibited higher peaks in the lower face regions compared to those in the upper face regions. In contrast, the expressiveness profiles of androids were relatively flat and exhibited the highest peaks in the upper face regions.

Table 5 summarizes the averages of expressiveness in lower and upper faces for each face. The standard deviations are listed in parentheses. Herein, we determined the boundaries of the upper and lower faces on the line of y = 60 mm, which passes through the nose top.

Table 5. Average and standard deviations of expressiveness in the lower and upper faces.

	Lower face expressiveness			Upper face expressiveness
Face	1D [mm]	2D [mm${}^2$]	3D [mm${}^3$]	1D [mm]	2D [mm${}^2$]	3D [mm${}^3$]
A1	3.5 (1.3)	3.3 (2.0)	1.1 (0.9)	5.5 (2.7)	3.6 (3.5)	0.5 (0.7)
A2	2.4 (1.1)	0.9 (0.7)	0.1 (0.2)	4.2 (2.3)	1.9 (1.2)	0.2 (0.2)
H1	14.1 (3.9)	62.2 (32.7)	122.3 (94.8)	7.4 (1.9)	10.6 (4.6)	5.3 (3.1)
H2	14.1 (3.1)	66.5 (30.0)	143.3 (95.8)	9.4 (2.7)	15.8 (7.6)	7.7 (4.5)
H3	14.5 (4.3)	79.5 (45.0)	148.7 (126.8)	6.1 (1.7)	9.0 (4.3)	4.0 (2.9)

We also determined the difference in the expressiveness between androids and humans to be prominent in the lower face and this prominence increased as the number of expressiveness dimensions increased. In other words, on average, the expressiveness scores of A1 and A2 were approximately 25$\mathrm{\%}$ and 17$\mathrm{\%}$ of those of humans in 1D expressiveness, respectively. Additionally, these scores were one and two orders of magnitude lower in androids than in humans in 2D and 3D expressiveness, respectively. In contrast, the difference in the upper face between androids and humans was relatively small, especially in 1D expressiveness.

Furthermore, we observed a difference in expressiveness between A1 and A2. The 1D, 2D, and 3D expressiveness of A1 was higher than that of A2 in the upper and lower faces. Particularly, on average, the 3D expressiveness of A1 was eleven times larger than that of A2 in the lower face as shown in Table 5.

3.5. Representative expressiveness octahedron

Figure 13 shows the representative expressiveness octahedrons of the lower and upper faces for A1, A2, and H1. Table 6 summarizes the diagonal lengths of each octahedron, calculated according to the averaged expressiveness shown in Table 5. Because the diagonal lengths of H1, H2, and H3 were similar, only the representative expressiveness octahedron of H1 was depicted in Figure 13.

Figure 13. Comparison of the shape and size of the representative expressiveness octahedrons of the two androids (A1 and A2) and a human (H1).

Table 6. Diagonal lengths of the representative expressiveness octahedrons.

	Lower face			Upper face
Face	1D [mm]	2D [mm]	3D [mm]	1D [mm]	2D [mm]	3D [mm]
A1	3.5	1.9	1.0	5.5	1.3	0.4
A2	2.4	0.8	0.3	4.2	0.9	0.3
H1	14.1	8.8	5.9	7.4	2.9	1.5
H2	14.1	9.4	6.5	9.4	3.4	1.5
H3	14.5	11.0	5.6	6.1	3.0	1.3

We observed that the representative expressiveness octahedron of the lower face (H1) was remarkably thick and large. In contrast, the octahedron of the upper face (H1) was small and elongated. Its diagonal lengths along with the first, second, and third expressiveness axes were a half, one third, and one-quarter of the octahedron of the lower face, respectively, as shown in Table 6.

Although the H1's upper face octahedron was smaller than that of the lower face, the upper face octahedron was larger than that of A1 and A2. The diagonal length of the first expressiveness axis was approximately 1.5 times longer in the upper face than the octahedrons of A1 and A2. Furthermore, it was observed that the upper face octahedrons of the androids were elongated in one direction and longer than their lower face octahedrons.

4. Discussion

4.1. Usefulness of the proposed index

The proposed evaluation index of facial deformation expressiveness depicted how and where the variations in the displacements of the facial points were quantitatively inferior in androids compared to humans. Considering the scores of 1D, 2D, and 3D expressiveness corresponded approximately to the maximum length, area, and volume of the existence region of displacement vectors, respectively, we could easily determine the amount of insufficiency in the displacements for androids to achieve expressiveness similar to humans. For example, according to Table 4, the maximum displacement length should be increased approximately by 5–10 mm for androids to have a level of 1D expressiveness similar to humans.

Therefore, we proposed three visualization methods of expressiveness, which were useful in investigating the characteristics of facial expressiveness. First, the expressiveness map was effective in investigating which facial regions having higher or lower expressiveness on each face. Second, the expressiveness profile was effective in comparing the expressiveness contour among different faces. A semilogarithmic graph was used for 2D and 3D expressiveness profiles considering the 2D and 3D expressiveness of humans exhibited a different order of magnitude, which was higher in the lower face than the androids. Third, the size and shape of the representative expressiveness octahedrons were used to compare the expressiveness among different faces.

The proposed method can be applied to any android for evaluating and comparing their mechanical performance based on the proposed expressiveness index. Furthermore, we can calculate the efficiency of the mechanical design for facial expressions by dividing the expressiveness in any facial region by the number of actuators installed in the region. This design efficiency indicates how efficiently the installed actuators realize the expressiveness on average. Here, we know that the lower face 1D expressiveness were 3.5 and 2.4 as shown in Table 5 while the number of actuators in A1 and A2 were 10 (numbers 6–15 in Table 1) and 3 (numbers 6–8 in Table 2) for the facial movements in the lower face, respectively. In this case, the design efficiency of the 1D expressiveness of A1 and A2 in the lower face is 0.35 and 0.8, respectively. Consequently, we can say that the mechanical design of A2 is 2.3 times more efficient than A1 in the lower face. This lower efficiency of A1 in the lower face is considered because the skin displacements produced by several actuators are similar or relatively small, and therefore, several actuators do not significantly increase the variations of skin movements. To date, several types of android robots have been created with different kinds of facial mechanisms. Still, it has been difficult to clearly say how a newly developed android is novel and advanced to the others in the mechanical performance. The proposed expressiveness index can create a favorable situation where android engineers claim that their new design can definitely increase the mechanical performance of android faces effectively and efficiently. This contributes to promoting effective competition in android robotics.

The most significant advantage of the proposed index is that android robot researchers can obtain a stable and objective evaluation score for their android face. Since the proposed index depends on the ‘mechanical’ property of a face and not on ‘psychological’ factors of human evaluators, the researchers can have a rigorous discussion on the mechanical design of the android faces. The hardware performance of android faces will be increased with this new evaluation index as the performance of display monitors has been increased with the physical evaluation index, such as color gamut and screen luminance.

4.2. Characteristics of expressiveness determined in androids and humans

Herein, we summarize the characteristics of expressiveness determined in androids and humans in this study. First, we observed that human faces exhibit three-dimensional expressiveness in both upper and lower faces. In particular, the expressiveness was prominent in the lower face, indicating that on average, the expressiveness in the lower face was 26 times higher than that in the upper face. This higher expressiveness was attributed to the particular subcutaneous structure in the lower face, that is, the skin surface in the lower face can move widely in any direction owing to the presence of a thick deformable fat layer that supports it and is not fixed to the bone.

Second, the maximum 1D, 2D, and 3D expressiveness of H1, H2, and H3 were found to be 26.4 mm, 208.5 mm${}^2$, and 640.2 mm${}^3$, respectively. These values of expressiveness are considered as one of the numerical targets for determining future androids that have similar expressiveness to humans. The skin materials and structures should be improved to accept these levels of skin surface movements.

Third, we found that the expressiveness of the upper faces in androids tends to be one-dimensional. On average, the 1D expressiveness was 1.7 times higher in upper faces than in lower faces. According to Figure 7, this high 1D expressiveness can be attributed to the fact that the upper eyelids can move widely.

Fourth, the 1D, 2D, and 3D expressiveness in the lower faces was found to be 4–6 times, one order of magnitude, and two orders of magnitude higher in humans than in androids, respectively. This indicates that android designers should preferentially improve the 1D expressiveness 4- to 6-fold, despite being insufficient for achieving 2D and 3D expressiveness like humans.

Fifth, we found that the 1D expressiveness of the upper face, whose expressiveness axes were almost along with the y-axis, was comparable for androids and humans. In contrast, the 2D and 3D expressiveness was one order of magnitude lower in androids than in humans. This indicates that improving the facial movements that move skin points along the x-axis in the upper face can enhance the 2D expressiveness. For example, facial motions that bring the eyebrows or lower eyelids inward are candidates.

Sixth, human faces were found to have common regions with a relatively low expressiveness between the upper and lower faces. These regions are candidate regions where a soft android skin is fixed to a hard inner shell representing a human skull. Given that android skin is too soft to maintain its default shape, it is necessary to prepare several supporting points on the inner shell. However, we could lose the expressiveness around the points where the skin is fixed. Therefore, having low expressiveness regions in human faces is desirable for android designers.

4.3. Limitation

One of the limitations of the proposed method is the high time and costs for developing the measurement environment of skin displacement vectors, a 3D motion capture system. This system requires a set of expensive infrared cameras and reflective markers instead of accurately obtaining 3D motion data of facial points. Moreover, we should carefully determine the positions and optical parameters of cameras, intervals of reflective markers, and lighting conditions for the motion capture system.

We can utilize the displacement vectors obtained by using the other measurement environments to calculate the proposed expressiveness because the calculation method of the expressiveness is independent of the measurement method. One of the other measurement methods is to calculate the displacement vectors called ‘optical flows’ based on the estimation of displacement of each in consecutive frames of an image sequence [31]. This method is more casual than 3D motion capture systems because it requires only one standard camera and no facial markers. On the other hand, it is estimated that the calculated expressiveness will be significantly lower than actual in several facial regions because of the inaccuracy of motion estimation based on the optical flow as explained in a survey of optical flow methods [31]. For example, the optical flows will be smaller than the actual displacements around the lateral facial regions because the displacements are nearly vertical to the camera image plane. Also, the optical flows around the cheek and forehead regions might be zero even when they are moving because the appearance is uniform in these regions, especially in androids. Therefore, when we utilize the optical flows to calculate the expressiveness, we should carefully evaluate the accuracy of the estimated displacement vectors in each facial region. The displacement vector data used in this study, namely 3D motion capture data with facial markers, is appropriate ground truth for the accuracy evaluation.

The other limitation is on three aspects of the proposed expressiveness. First, we cannot discuss the asymmetricity of the expressiveness because this study focused only on the half faces. Since it is well known that human faces are asymmetric, the expressiveness plot may be asymmetric as well. Suppose we reveal how the asymmetricity of the expressiveness in human faces contributes to providing human-likeness or friendliness to others. In that case, the asymmetricity will be effectively adapted to the android face design. Second, we cannot discuss how the proposed mechanical expressiveness corresponds to the perceived expressiveness by human observers. If we know the relationship between mechanical and perceived expressiveness, we can determine the design target of the mechanical expressiveness that satisfies the design requirement of the perceived expressiveness. We will investigate the relationship in future works by examining how humans evaluated android faces' expressiveness with different mechanical expressiveness. Third, the proposed expressiveness evaluates the variations of each facial point's movements independently, and therefore, it does not evaluate the variations of producible deformation patterns on entire facial surface. Even when androids or humans have the same evaluation scores in the proposed expressiveness, the ones who can move several facial regions more independently can exhibit more numbers of facial deformations. Considering humans cannot express some of the facial motions listed in Table 3 completely independently, we should propose the other type of expressiveness index to evaluate the variations of deformation patterns that any combinations of actuators or muscles can produce.

5. Conclusion

In this study, we proposed a numerical index for the facial deformation expressiveness in androids and humans. Depending on the proposed index, we investigated the difference in facial expressiveness among faces and facial regions. As a result, we determined that the expressiveness of two recently developed androids were approximately 25$\mathrm{\%}$ and 17$\mathrm{\%}$ that of humans in the lower faces, even in one-dimensional evaluation, wherein the variation of facial point movements was calculated along the axis with maximized range. The proposed method revealed how recent androids lack mechanical performance in terms of moving the facial skin widely. Moreover, we were able to numerically discuss the future directions of hardware improvement of android faces to realize the next generation androids with expressiveness similar to that of humans.

Author Contributions

H. I. designed the study, analyzed the data, and prepared the manuscript.

Acknowledgements

The author thanks the members of the Emergent Robotics Laboratory from the Department of Adaptive Machine Systems at the Graduate School of Engineering at Osaka University for their helpful comments.

Disclosure statement

The author declares that this study received funding from A-Lab, Co., Ltd. The funder was not involved in the study design, collection, analysis, interpretation of data, writing of this article, and the decision of submission for publication.

Additional information

Funding

This work was partially supported by JSPS KAKENHI Grant Number JP20K14697, JP20H05617, and A-Lab Co., Ltd.

Notes on contributors

Hisashi Ishihara

Hisashi Ishihara received M.S. and Ph.D. degrees in engineering from Osaka University in 2009 and 2014 respectively. He has been an associate professor at Osaka University, Osaka, Japan, from 2019. He had been a PRESTO/Sakigake researcher at Japan Science and Technology Agency from 2016 to 2020. His research field is android robotics.