Measuring Aesthetic Preferences of Neural Style Transfer: More Precision With the Two-Alternative-Forced-Choice Task

Abstract

The present work compares the two-alternative forced choice (2AFC) task to rating scales for measuring aesthetic perception of neural style transfer-generated images and investigates whether and to what extent the 2AFC task extracts clearer and more differentiated patterns of aesthetic preferences. To this aim, 8250 pairwise comparisons of 75 neural style transfer-generated images, varied in five parameter configurations, were measured by the 2AFC task and compared with rating scales. Statistical and qualitative results demonstrated higher precision of the 2AFC task over rating scales in detecting three different aesthetic preference patterns: (a) convergence (number of iterations), (b) an inverted U-shape (learning rate), and (c) a double peak (content-style ratio). Important for practitioners, finding such aesthetically optimal parameter configurations with the 2AFC task enables the reproducibility of aesthetic outcomes by the neural style transfer algorithm, which saves time and computational cost, and yields new insights about parameter-dependent aesthetic preferences.

1. Introduction

The rising public and academic awareness of computer-generated arts over the past decade (DiPaola & Gabora, 2009; McCormack et al., 2019) has increased the acceptance of creativity generated by computers (McCormack & D’Inverno, 2012). The debate over its legitimacy has been dominated by the question whether its outcome can be deemed aesthetic (Chamberlain et al., 2018). The relevance of this question is further evidenced by the emerging new academic field of computational aesthetics (Bo et al., 2018) which aims at the generation and the evaluation of visually aesthetic expressions. As artificial intelligence advances in the generative models for visual artefacts, a new point of interest has become the aesthetic value of deep learning generated images.

1.1. Measurement of aesthetics

Measuring the aesthetic value appears particularly interesting in the context of HCI for several reasons. The perceived aesthetics of visual artefacts are known to induce pleasurable experiences (Blijlevens et al., 2017; Graf & Landwehr, 2017; Hekkert, 2006) and thereby increase the usability and likeability of products (Hassenzahl, 2001; Jordan, 2002; Norman, 2004). For commercial applications, it would be particularly interesting to find an “optimal” configuration of the image-generating algorithm to enable the reproducibility of desired outcomes, where “optimal” refers to the most aesthetically perceived result among the compared configurations. This should be possible under the assumption that the perceptual responses to the aesthetic outcome of different parameter configurations are measurable. Measuring such perceptual responses would allow us not only to find the configuration of maximum achievable aesthetics but also to detect patterns that reveal potentially new insights on aesthetic preferences. Therefore, the assessment of aesthetic perception of computer-generated arts is highly relevant for HCI.

1.2. Problematic of rating scales measuring aesthetics of neural style transfer

The computer-generated arts have risen in popularity partly due to a particular generative deep learning algorithm, neural style transfer, that caught wide public attention at the DeepArt exhibition at the 2017 Neurips conference. Gatys et al. (2015) introduced this algorithm to show how the artistic style of painters can be transferred to another image. This algorithm extracts the structural information of the input image (e.g., the facial structure and body contour), learns the color and texture information from the style image, and then renders the semantic structure of the input image in the color and texture of the style image.

Since its introduction in 2015, neural style transfer has been developed further in many variants to improve specific visual challenges (T. Q. Chen & Schmidt, 2016; Choi, 2018; H. Huang et al., 2017; X.-C. Liu et al., 2017; Zhu et al., 2017). Despite this rapid technical evolution, the majority of studies are methodologically flawed in the measurement of aesthetics. In a comparison of neural style transfer algorithms in 2018, Li (2018) noted that the most common measurement method was the self-evaluation by the authors (e.g., Cui et al., 2018; Ghiasi et al., 2017; Ulyanov et al., 2016, 2017; Wang & Tan, 2018). This constitutes a severe selection bias because the assessment by the authors is subjective and positively biased towards their own results (“cherry picking”).

Only a few studies on neural style transfer avoided subjective self-evaluations, and assessed their outcomes by user surveys (D. Chen et al., 2018; H. Huang et al., 2017; Luan et al., 2017; Risser et al., 2017). In these surveys, the most frequently used measurement instruments are rating scales (e.g., Likert scales) which are known to be prone to the acquiescence bias (Friborg et al., 2006; Knowles & Nathan, 1997). The acquiescence bias is a response bias that tends to regress to the previously selected answers in a survey, thus producing the same or similar responses on subsequent ratings (Ray, 1983). An additional difficulty for rating scales is to differentiate the visual output of neighboring parameter configurations which can be too subtle to distinguish by normal eye sight.

The same problematic applies for ranking scales which are occasionally used as alternative to rating scales. The inherent problem of ranking is that it neglects how close or far two assessments are in reality, losing measurement precision. Furthermore, the ranking scale forces the viewer to create a relative order, which imposes a strong cognitive demand because the subject has to assess all presented stimuli at the same time (Palmer et al., 2013). In contrast, the 2AFC task restricts the cognitive load to one comparison at a time (Mantiuk et al., 2012).

1.3. The advantages of the two-alternative choice task

Because rating scales lack differentiation of subtle visual differences and are prone to the acquiescence bias, the present work suggests to use the two-alternative forced choice (2AFC) task (Thurstone, 1927) instead. The 2AFC task specifies the assessment of two simultaneously presented samples on either of two available choices, e.g., aesthetic vs. non-aesthetic (Macmillan & Creelman, 2004). It was originally used in psychophysical research to detect perception thresholds (Fechner et al., 1966), and has since been extended to analyze complex stimuli such as decision making (Bogacz et al., 2006) or auditory stimuli (Delplanque et al., 2019).

In addition to the lack of acquiescence bias, further advantages of the 2AFC task over rating scales are the absence of memory load and response bias (Palmer et al., 2013), as well as significantly reduced fatigue and carryover effects (Yang & Ng, 2017).

1.4. Research question

Based on the previous considerations, the present work aims to close two gaps in the literature. The first gap is that despite its proven qualities, the 2AFC task has not been applied yet to measure the aesthetic perception of computer-generated arts in general, and of neural style transfer-generated images in particular. The second gap is the lack of direct comparisons in the literature between the 2AFC task and rating scales. The present work aims to close these gaps by applying the 2AFC on neural style generated images, and comparing the aesthetic preference patterns with those extracted by rating scales. Notwithstanding, the goal of this study is not to find generalizable aesthetic patterns but to test the 2AFC task in its psychometric quality. The focus is therefore on investigating whether the 2AFC task responds more sensitively than the commonly used rating scales, and whether this results in more differentiated, clearer or more pronounced patterns. In light of the above considerations, the research question can be formulated as follows.

RQ: Can the 2AFC task detect more differentiated patterns than rating scales when measuring aesthetic preferences of computer-generated images?

With this research question, the present work addresses the methodological problem in previous studies on neural style transfer of qualitative subjective assessment by authors and acquiescence bias of rating scales. These problems are expected to not occur with the 2AFC task because of the way stimuli are presented (pairwise comparisons) and selected (forced choice).

The 2AFC task could be used to analyze differences between any kind of images. However, applying it to images generated by a configurable image-generating algorithm, such as neural style transfer, makes it particularly interesting. As the 2AFC task empirically identifies the optimal configuration among the given stimuli with more precision than the rating scale (as will be shown in this study), the results become reproducible, given similar stimuli and sample. The higher precision of the 2AFC task derives from the fact that it can detect perceptual differences even when people have difficulty to make conscious decisions, e.g., for very similarly looking stimuli like in the present study. This effect seems even more pronounced with judgments for which no objective criteria exist, such as for aesthetic evaluation. Taken together, the application of the 2AFC task on various parameter configurations allows to find the configuration that maximizes aesthetics. This knowledge is practically relevant as it allows to create a degree of reproducibility on the very subjective perception of aesthetics.

The practical relevance of this endeavor is further elevated by the emerging trend in the advertising industry of targeted advertising (S. Chen, 2020), i.e., the customization of promotional material to more differentiated target audiences. This trend, together with the shift from print to online media (Sridhar & Sriram, 2013), might facilitate that advertising will be increasingly shaped by new technologies like deep learning in the near future (Gharibshah et al., 2020). Important content domains where deep learning could be applied are promotional commercials and banners in TV and social media that feature celebrities known from acting, singing, sports or other entertainment industries. In these content domains, a neural style transfer algorithm would allow designers to customize the promotional material easily and quickly to specific color and texture preferences of several differentiated target audiences.

In summary, the present work exemplifies how to respond to the recently emerging HCI challenges of augmenting human cognition and applying AI for personalization to human needs (Stephanidis et al., 2019).

2. Related works

2.1. Neural style transfer

Since its introduction in 2015 by Gatys et al. (2015), neural style transfer has gained widespread popularity both academically and industrially (e.g., http://deepart.io). An abundance of neural style transfer algorithms has emerged that widely differ in model architectures and purposes (Jing et al., 2020). Originally, neural style transfer was demonstrated with common photo motifs like houses or landscapes (Gatys et al., 2016) or doodles (Champandard, 2016). Neural style transfer was later applied to more complex topics such as fashion (Jiang & Fu, 2017), videos (H. Huang et al., 2017; Zhu et al., 2017) and artistic improvisation (Choi, 2018).

For high structural information motifs, like the portrait images in the present study, computational time is higher and mainly determined by the number of iterations. This is a concern that research already addressed by developing so-called feedforward networks that aimed to reduce computational time (T. Q. Chen & Schmidt, 2016). Such feedforward networks are faster than optimization algorithms by applying pretrained models for stylization which only requires a single forward propagation pass. However, they require substantial training time for generating these pretrained models on many variations of a specific style image.

Although the multitude of available neural style transfer algorithms is difficult to categorize without going into very technical details, a characteristic conceptual distinction between all algorithm variants can be made on the loss function, i.e., the function which the deep learning models aims to minimize by gradient descent. The loss function in neural style transfer is commonly composed of two loss terms (for the content and the style loss), and often contains a purpose-specific additional loss term and/or a regularization term. Examples for purpose-specific loss terms are perceptual loss terms to replace per-pixel losses by higher-level features (J. Johnson et al., 2016), histogram loss terms (Risser et al., 2017) to enable texture synthesis, a depth loss term to increase depth awareness (X.-C. Liu et al., 2017), or a temporal consistency loss term for neural style transfer of videos that reduces the arbitrary changes between consecutive video frames (M. Ruder et al., 2016). Applications that require a purpose-specific regularization term include photorealism (Luan et al., 2017), or an input noise factor k (Ulyanov et al., 2016). Another strategy for noise reduction is to use the parameter total variation which had occasionally been used (T. Q. Chen & Schmidt, 2016). The present study disregarded any of the above parameters because they were specific to the model architecture and thus impeding the study’s goal of reproducibility.

2.2. Aesthetic evaluation

Visual aesthetics has been taught not only in fine arts, design, and computer vision, but also in psychology from a perceptual lens as aesthetic preferences. According to Palmer et al. (2013), aesthetics is “the study of mental processes that underlie disinterested evaluative experiences,” and such evaluation can be characterized along a continuum of emotional valence. Several studies using neuroimaging (Cela-Conde et al., 2011) or fMRI (Vartanian & Goel, 2004) have largely confirmed the existence of neural correlates for subjective aesthetic judgments, based primarily on the assumption that no objective standard for evaluating “beauty” exists.

Nethertheless, the literature on aesthetic preferences has identified several relevant predictors in the last decades. For example, aesthetic preferences correlate with pupil size (M. G. Johnson et al., 2010), and with different brain regions for men (right hemisphere) and women (bilateral) (Cela-Conde et al., 2009). For digital artworks, they can be predicted by visual simplicity, visual symmetry, visual contrast, and visual self-similarity (Mayer & Landwehr, 2018). For simple pictures, overall preference (i.e., liking) is predicted by the perception of balance that includes homogeneity (Hübner & Fillinger, 2016). The preference of abstract paintings over figurative paintings is predicted by a higher need for cognitive closure, i.e., the behavioral avoidance of uncertainty (Wiersema et al., 2012). Similarly, abstract arts is preferred over renaissance art by people with high scores on the personality trait openness for experience (Pelowski et al., 2017).

Several studies found distinct aesthetic preferences related to geometry. People prefer a larger (smaller) size of a displayed object relative to the frame for larger (smaller) objects, and a displayed object size that is proportional to the logarithm of its physical size (Linsen et al., 2011). Despite high inter-individual variation, a general preference for curvature over non-curvature in object contours could be found, and a high sensitivity to curvature is consistent for real objects and abstract designs (Corradi et al., 2019). Whereas individuals’ preferences of rectangle shapes strongly differ, they are not related to any other inter-individual differences commonly investigated in psychology including personality traits and need for cognition (McManus et al., 2010).

Aesthetic preferences can be predicted by complexity of the visual stimulus following an inverted U-shape, and this curvilinear relationship has also been confirmed for audial stimuli (Delplanque et al., 2019). Contrary to common perception, aesthetic preferences are not stable across the human lifespan but follow an inverted U-shape peaking around early to middle adulthood (Pugach et al., 2017).

Aesthetic preferences of colors have been explained by ecological valence theory which postulates that people prefer colors associated with objects they like (blue from sky), and dislike colors associated with objects they dislike (brown from rotten food) (Palmer & Schloss, 2010).

Based on the conception of the dynamic interaction between an aesthetic interface and an aesthetically experiencing user (Udsen & Jørgensen, 2005), human computer interaction (HCI) research has increasingly investigated aesthetic preferences in the last two decades. For example, a website’s perceived expressive aesthetics, along with usability, predicts the overall website preference (De Angeli et al., 2006). The design aesthetics of a mobile phone increases this product’s usability rating whereas the actual phone use does not (Sonderegger & Sauer, 2010). Aesthetic preferences of product design are strongly predicted by novelty in an inverted U-shape (Hung & Chen, 2012).

The aesthetics literature has further provided insights about aesthetic preferences on traditional paintings. For example, some evidence appeared about the existence of two antagonistic preferences for realistic versus abstract paintings. This preference for abstract over figurative paintings could be predicted by a higher need for cognitive closure, i.e., the need for a predictable and unambiguous world (Wiersema et al., 2012), and by the personality trait openness (Furnham & Walker, 2001) which also predicted the preference for abstract arts over renaissance art (Pelowski et al., 2017).

2.3. The 2AFC task

The 2AFC task is an experimental method of psychology introduced by Thurstone (1927). In cognitive psychology, it is used to detect perception thresholds or evaluate the psychological differentiation of stimulus variation. Applying signal detection theory (McNicol, 2005), the underlying model identifies the detection thresholds of perceived stimuli in a 2AFC task by fitting the so-called psychometric function, a maximum-likelihood estimator, to the averaged 2AFC task responses. In such applications, the presented image pairs contain a baseline image (image without any effect) to which the other image is compared.

The 2AFC task specifies the assessment of two samples, thus called a paired comparison test. Both samples must be presented at the same time [p. 148] (Macmillan & Creelman, 2004). If only one stimulus is shown, it represents a yes/no task (often used for lexical decisions) which is sometimes confused with the 2AFC task.

The methodological advantage of the 2AFC design is that it significantly reduces fatigue, carryover and memory effects encountered when assessing more samples (Yang & Ng, 2017). Moreover, it reduces the required sample size of participants because each individual image is assessed several times due to the pairwise comparisons.

Palmer et al. (2013) compare three methods for the direct measurement of aesthetic preferences: rank ordering, rating, and the 2AFC task. According to the authors, the optimal measurement can be achieved by the 2AFC task. The disadvantage of rank ordering is that it overwhelms the participant with a strong cognitive demand to identify a relative ordering of all images. Rating scales, such as Likert scales, are prone to the acquiescence bias which substantially decreases the validity of results (Friborg et al., 2006; Knowles & Nathan, 1997).

Psychological research has applied the 2AFC task to investigate aesthetic preferences, e.g., classifying images as art vs. non-art (Pelowski et al., 2017), the spatial composition in multi-object pictures (Leyssen et al., 2012), or the size of images for real-world objects (Linsen et al., 2011). For testing aesthetic preferences, the common 2AFC task procedure is to present the participant with all possible pairs of stimuli instead of all comparisons with a baseline image (Palmer et al., 2013). The 2AFC task’s main measure for an image’s relative preference is calculated by the actual frequency of selecting it over all other images, which represents the probability chosen.

Recently, the 2AFC task has been increasingly used for evaluating images in aesthetic dimensions, e.g., for analyzing the color effectiveness in macromolecular structures (Harrison et al., 2014), or the hue, saturation, brightness and texture in protein structures (Vuong et al., 2018).

Furthermore, the 2AFC task revealed smallest measurement variance, highest accuracy, and most time-efficiency given a moderate number of conditions when compared with the other most widely used methods (single-stimulus, double-stimulus, similarity judgments) for assessing subjective image quality (Mantiuk et al., 2012).

3. Parameter selection for the neural style transfer algorithm

The literature shows an abundance of neural style algorithms that vary in purpose, e.g., hand-created doodles (Champandard, 2016), houses and landscapes (Gatys et al., 2016), fashion (Jiang & Fu, 2017; Zhu et al., 2017), or artistic improvisation (Choi, 2018). Because algorithms differ widely in parameters¹ depending on these different purposes, it is not surprising that the literature of neural style transfer has not provided a systematic analysis of parameter variation yet. Therefore, the present work made a selection among the parameters most frequently cited in publications and related code implementations to facilitate comparability, including (a) the content and style loss (directly configurable or indirectly as content and style weight), (b) the learning rate, (c) the batch size, and (d) the number of epochs or number of iterations. The latter two parameters are interdependent, i.e., fixing one determines the other parameter—e.g., with a small batch size of 2, 80,000 iterations only amount to two epochs of the training set containing 40,000 video frames (H. Huang et al., 2017). Similarly, the present study also fixed the batch size to the default value, so selecting the number of iterations as parameter was sufficient.

While the outcome differs among neural style algorithms due to many different influence factors including content and style extraction mechanisms, we can expect a visible impact of the selected parameters in the sense that the content structure will be rendered more clearly both with increasing number of iterations and higher content-style ratio. However, no such prediction can be made for the learning rate because of algorithm-specific reasons: Whereas locations higher on the cost curve require higher learning rates, proximity near the cost optimum favors lower learning rates. However, as the content and style loss values are unknown before the neural style transfer, the visual impact cannot be predicted and thus remains subject to the empirical investigation in the present study.

3.1. Number of iterations

For high structural information motifs, like the portrait images in the present study, computational time is higher and mainly determined by the number of iterations. The implementation in the present study is an optimization algorithm (provided by the script slow_lua_style.lua) (J. Johnson, 2016) that requires separate training on each style image. The training duration thus depends on the number of iterations.

For practitioners, it would be beneficial to know a minimum cutoff value because it would save both time and the computational cost which a higher number of iterations would entail. However, the literature rarely provides recommendations about the number of iterations (X. Huang & Belongie, 2017; Risser et al., 2017; M. Ruder et al., 2016). In general, it is expected that a higher number of iterations produces better results because the content and style loss are increasingly reduced. This reduction is gradually decreasing and thus converging to a certain level. This convergence allows us to locate an approximate cutoff value that indicates the minimum number of iterations for acceptable aesthetics.

3.2. Learning rate

For parameters other than the number of iterations, simple rules like “the more/less, the better” are not applicable for neural style transfer. Generally, smaller learning rates are advantageous because they can approach the cost minimum closer than higher learning rates which can fluctuate around the cost minimum and thus hinder convergence. However, as S. Ruder (2016) pointed out, a too small learning rate “painfully” slows down the convergence to the cost minimum. Even more importantly, reaching the minimum of the content cost or style cost merely means that the content or style image is regenerated well by the neural style transfer algorithm. This does not necessarily translate into a more aesthetic outcome.

3.3. Content-style ratio

The present study verifies this fact for neural style transfer by varying the content-style ratio, with higher content weight resulting in realistic paintings and higher style weight resulting in abstract paintings. The same applies to the content-style ratio where increasing the weights for the content image or style image cannot be predicted in its impact on aesthetics. It is therefore not surprising that many researchers had previously remarked that the parameter configuration for neural style transfer was “tedious” and required “careful” manual tuning (Jing et al., 2017; Li, 2018), and even noted that the parameters’ impact on the visual outcome was “unpredictable” (Wang & Tan, 2018).

3.4. Algorithm selection for reproducibility

Recently, the lack of reproducibility for many findings in the AI literature, mainly due to unpublished code but also lack of documentation, has been increasingly criticized (Gundersen & Kjensmo, 2018; Hutson, 2018). The present work addresses this problem by selecting the algorithm depending on how well it facilitates reproducibility and comparability of results. Therefore, the algorithm was not chosen from the latest advances but rather by the highest ranking in popularity and citations, namely the implementation of Justin Johnson (2016) in the Github repository jcjohnson/neural-style (J. Johnson, 2015). It gained the highest popularity among all neural style algorithms on Github (starred over 18.000 times on Github in June 2021). Contributing to its popularity was its excellent documentation and the parameters available for configuration. This explains why it was used in numerous studies for comparison (X. Huang & Belongie, 2017; S. Liu et al., 2016; Ulyanov et al., 2016) and for extension (Luan et al., 2017; M. Ruder et al., 2016). It is important to note that although the conducted literature review guided the parameter selection, it could not guide the range of parameter values in the study because network architectures and pretrained models for image classification networks widely differ, resulting in different parameter specifications.

4. Method

The present study aimed at identifying the impact of neural style transfer-specific parameters on aesthetic preferences. As stimuli, neural style transfer images were generated from three input images and five style images, varying one parameter in five configurations for each input image. The aesthetic perception of the same input image-style image combinations was analyzed both by the 2AFC task and by rating scales. These assessment scores were analyzed by ANOVA (analysis of variance) and the retrieved aesthetic patterns were compared between the 2AFC task and the rating scales.

4.1. Participants

The present study recruited a sample of 55 participants undergraduate and graduate students of interaction design, as well as professional UX designers. The students were recruited through the mailing list of the university’s design department, and the professional designers via Amazon Mturk filtering for Mturk workers in the “design & arts” occupation category with at least a Bachelor’s degree.

Participants were 38.2% male and 61.8% female, and on average 30.73 years old (SD = 10.84). The academic status was 50.9% Bachelor student, followed by 14.5% Master student, 18.2% professional with Bachelor’s degree, 14.5% professional with Master’s degree, and 1.8% professional with Ph.D. degree. Participants’ nationality was South Korean (49.1%), United States (32.7%), Canada (3.6%), Germany (3.6%), and the remaining 9.1% distributed over Singapore, United Kingdom, Côte d’Ivoire, Ghana, Kazakhstan and Venezuela.

4.2. Stimuli

4.2.1. Stimuli—Input images

Generally, the visual structure is partially obscured by the neural style transfer so viewers feel the need to orient themselves towards a recognizable depiction of structural information. Hence, it was important to select input images which are easily recognized in their semantic content.

Celebrity faces, rendered in different colors and texture, are a frequent theme in advertising (e.g., product commercials) and product marketing (e.g., album covers). Therefore, this study chose the faces of celebrities from both genders and with a global reach as featured content, namely Charlie Puth, Jessica Alba, and Ellie Goulding (see Figure 1).

Figure 1. Input images.

4.2.2. Stimuli-Style images

To control for the effect of textural granularity by style images, the present study chose style images with a textural granularity ranging from very fine (thin bush plant), medium (black and white patterns, cloth design) to very coarse (toys and abstract arts) as shown in Figure 2.

Figure 2. Style images.

Preliminary experiments tested whether certain background characteristics could interfere with rendering the content image showing the head and facial structure. The result was that the generated image can be confounded with a background containing structure as the algorithm would render the background of the style image as well.

Therefore, only style images with empty and mono-colored backgrounds were chosen. This restriction did not apply to the input images as long as their background did not contain distinct structural information similar to facial structures.

4.3. Neural style transfer algorithm

The present study used the implementation of neural style transfer provided by the Github repository jcjohnson/fast-neural-style (J. Johnson, 2016) which is implemented in torch (Collobert et al., 2018). The original optimization-based algorithm (Gatys et al., 2015) was implemented by Justin Johnson in the Github repository jcjohnson/neural-style (J. Johnson, 2015) but did not provide sufficient configurability. The jcjohnson/fast-neural-style repository provides an improved version of the same optimization algorithm in the script slow_neural_style.lua. This version was selected for the present study because it provided the configuration of all relevant parameters, i.e., number of iterations, learning rate and content-style ratio.

The loss network was set to the default VGG-16 (Simonyan & Zisserman, 2015), a pretrained model trained on the Imagenet dataset (Deng et al., 2010). As an alternative, the present work initially considered using Resnet networks (He et al., 2016) but discarded them because they did not render any stylization effect in preliminary experiments, which the author Justin Johnson confirmed on email request. The present study chose Adam over L-BFGS as optimizer² following the recommendation by the author, Justin Johnson, for significantly reduced memory usage and because Adam is more commonly used as optimizer in general (X. Huang & Belongie, 2017; Ulyanov et al., 2016). All results of this study were generated following the process shown in Figure 3 on an Ubuntu 16.04 LTS virtual machine in the Google Cloud using a Nvidia Tesla P100 GPU with CUDA 9.1 and CUDNN 8.0 libraries.

Figure 3. Image generation process of neural style transfer.

4.4. Parameter configuration

4.4.1. Number of iterations

Johnson et al. (2016) found that the optimization is successful within 500 iterations in most cases. The term iterations does not refer to model training (because a pretrained model of image recognition is used) but to the optimization-based style transfer (Gatys et al., 2016). The default setting was 500, yet preliminary experiments showed that satisfactory results can be achieved with a lower number of iterations. Therefore, this study compared 100, 200, 300, 400, and 500 iterations (Figure 4).

Figure 4. Neural style transfer-variations of parameter: number of iterations.

4.4.2. Learning rate

The learning rate is a parameter that specifies the step size with which the pixel values are updated by the gradient of the combined content and style loss (Gatys et al., 2016). Although the default learning rate is 1.0, learning rates above this value revealed more structure in output images in preliminary experiments. This is understandable as this study aimed for a lower number of iterations, which require higher learning rates to approach the cost minimum. Therefore, this study explored the learning rates 0.5e1, 1e1, 2e1, 4e1, and 6e1 (Figure 5).

Figure 5. Neural style transfer-variations of parameter: learning rate.

As noted before, the chosen learning rates cannot be compared with other learning rates cited in the literature (Hicsonmez et al., 2017; Ulyanov et al., 2016) because the underlying algorithms differ and thus the corresponding loss functions differ as well. For example, in feed-forward models, the learning rate refers to model training (X. Huang & Belongie, 2017; Ulyanov et al., 2016) whereas in our algorithm the learning rate refers to image optimization.

4.4.3. Content-style ratio

The content-style ratio determines the degree of importance given to the input image vs. style image for rendering the output image. The original default ratio is 1:5, i.e., the style is weighted five times higher than the content. Preliminary experiments showed that this yielded too little image structure in many cases, so a lower range was chosen. Hence, this study compared neural style transfer images generated by the content-style ratios 10:100, 50:100, 100:100, 200:100, and 300:100 (Figure 6).

Figure 6. Neural style transfer-variations of parameter: content-style-ratio.

4.5. Two-alternative forced choice (2AFC) task

When the 2AFC task is used for detecting perception thresholds, it shows the stimuli in all pairwise comparisons with a baseline image. The comparison is different when the 2AFC task measures general sensitivity, as in the present study for aesthetic preferences. Here, the 2AFC task presents the participant with all pairwise permutations of stimuli. For each pairwise combination, participants are asked which “is more aesthetic” or “looks better” which corresponds to their aesthetic preference (Palmer et al., 2013).

Each parameter was analyzed by the 2AFC task for every unique combination of parameter (three), parameter configuration (five), input image (three) and style image (five). Figure 7 visualizes this process and how it contrasts to the stimulus creation process for the rating choice/rating scale task. As can be seen, this procedure amounts to an effective sample size of n_final = 2750 for each parameter in the 2AFC task. The 2AFC design hence holds a statistical advantage of effective sample size because each individual image is assessed multiple times in the pairwise comparisons whereas with rating scales, each stimulus is assessed only once.

Figure 7. Selection and Assessment Process with 2AFC Task and Rating Choice/Rating Scale. The 2AFC task was performed for each parameter by 55 participants × 5 unique input image-style image combinations × 5 parameter configurations × 4 comparisons/2 two-alternative-forced-choice, amounting to 2750 assessments for each of the three parameters. The rating scale was performed for each parameter by 55 participants × 5 unique input image-style image combinations × 1 sequence (of all 5 parameter configurations), amounting to 275 assessments for each of the three parameters.

The 2AFC task’s main measure for an image’s relative preference is calculated by the actual count of selecting it over all other images. These absolute counts are aggregated as relative frequencies of choosing a unique combination over another, and are expressed after normalization as probability chosen (ranging between 0 and 1) for easy comparison and interpretation.

4.6. Comparison with analysis by rating scales

The acquiescence bias is induced by the repeated row design in the rating scales. This means that the participants see all images to compare next to each other; in addition, they see the same rating scale in multiple rows to assess the aesthetic perception of these images. As the resulting effect of acquiescence bias is well known and therefore expected, the present study tested an improved measurement design of rating scales which aimed to reduce the acquiescence bias. This design asks participants to pick one of the five configurations as their favorite, and rate the degree of aesthetic preference specified as a percentage (0–100%). The acquiescence bias is reduced by this design by replacing the multiple assessment by the favorite choice. This alternative measurement will be used for comparison with the 2AFC task.

For this comparison, the present study analyzed whether both measurement methods could detect significant differences on aesthetic preference scores with ANOVA tests. ANOVA reports the F-statistic which is defined as the ratio of between-group variance divided by the within-group variance: $$F=\frac{\mathit{between}-\mathit{group}\mathrm{ }\mathit{variance}\mathrm{ }{s_b^2}_{}^{}}{\mathit{within}-\mathit{group}\mathrm{ }\mathit{variance}\mathrm{ }{s}_w^2}$$ With $$s_b^2=\frac{{\sum }_{j=1}^m{n}_j{({\stackrel{-}{x}}_j^{}-\stackrel{-}{X})}^2}{m-1},$$ $$s_w^2=\frac{{\sum }_{j=1}^m{\sum }_{i=1}^n{{(x}_i-{\stackrel{-}{x}}_j^{})}^2}{n-m}$$

The ANOVA is performed as a hypothesis test on the null hypothesis of equal means, and yields the F statistic and the significance level p. The closer p is to zero, the higher the probability that the null hypothesis can be rejected, i.e., that the mean scores differ from each other.

The ANOVA was performed on the global level (on configuration, style, and interaction between configuration and style) as two-way within-subjects ANOVA (dependent variable: aesthetic preference, independent variables: configuration, style) for each parameter. On the detail level, the ANOVA was performed as a one-way within-subjects ANOVA (dependent variable: aesthetic preference, independent variable: configuration or style).

Like any least-squares model, ANOVA is based on the assumptions that the residuals are normal distributed and the variances are homogeneous (homoscedasticity). The normal distribution of residuals was verified by QQ normality plots and histograms for both 2AFC task (Appendix A, Figures A1 and A2) and rating scale (Appendix A, Figures A3 and A4). Most observations are located near the reference line of normal distribution, yet a few violations at the scale extremities. These were deemed acceptable as ANOVA is known for being robust against violations of the normal error assumption (Hocking, 2013) and insensitive to unequal distributions (Blanca et al., 2017).

The violations to homoscedasticity were verified by the variance ratio, i.e., the ratio between the maximum and minimum variance, for which the literature recommends cutoff values between 3 (Kirk, 2013) and 10 (Tabachnick & Fidell, 2019). The variances in the data grouped by style revealed that the variance ratio was below these thresholds both for the 2AFC task (1.247) and the rating scale (2.731), thus testifying homogeneity of variance.

4.7. Procedure

Before starting the survey, participants were asked to not use a smartphone but a laptop, desktop or tablet for larger image display. They were instructed to take a close look at each of the two images for at least 2–3 s, and to make the choice between the two images intuitively rather than by objective criteria. They were told that the differences between some images were very subtle and that they didn’t have to bother to find them.

The first part of the survey contained the 2AFC task. It presented each image pair on a separate page with the title question “Which image is more aesthetic?,” and the participants had to select a radio button below the left or right image (see Figure 8). Each participant assessed five configurations of three parameters for five styles.

Figure 8. Participant perspective of 2AFC task.

The position of the two images in the 2AFC task (either left or right) was counterbalanced within participants so that each unique image (unique denoting a specific combination of input image, style image, parameter, and configuration) was evaluated in the same frequency in the left and right position. As example, for the input image “Charlie Puth,” image pairs with the following combinations of style image (S1 = godami style), parameter (A = number of iterations), and configuration (1–5 = 100–500 iterations) were evaluated: S1A1–S1A2, S1A2–S1A3, S1A3–S1A4, S1A4–S1A5, S1A3–S1A1, S1A1–S1A4, S1A5–S1A1, S1A4–S1A2, S1A2–S1A5, S1A5–S1A3. This permutation scheme was applied for each of the parameters, input images and style images.

The second part of the survey contained rating scales. Participants had to select their favorite image among the five configurations for each unique combination of parameter, input image, style image, and configuration. In a previous study on computer-generated visual stimuli (in review), we corroborated the acquiescence bias of rating scales found by other research (Palmer et al., 2013). We thus aimed in this study to remove the acquiescence bias by a new task design that preempted the rating with a “rating choice” task, i.e., the choice of the most aesthetic image among five parameter configurations (see Figure 9). This allowed estimating the differential effect of favorite image choice among five compared to two images (rating choice task vs. 2AFC task). After each choice of favorite image, participants rated the degree of aesthetic preference (“How aesthetic is your favorite?”).

Figure 9. Participant perspective of rating choice/rating scale task.

5. Results

The goal of this study was to explore whether and to what degree aesthetic preferences can be measured and with which measurement method most precisely. To answer this question, this study analyzed the aesthetic preference scores on a global level, i.e., two-way ANOVA on configuration and style, and on the detail level, i.e., in one-way ANOVAs, on configuration differences for each style, and on style differences for each configuration.

Figure 10 visualizes the preference scores measured by the 2AFC task and by the rating scale. The 2AFC task preference scores depict the means (dots) and standard errors (error bars) for 2750 pairwise comparisons for each of the three parameters, amounting to a total of 8250 aesthetic assessments. Analogously, the preference scores measured by rating scales were visualized in their means and standard errors with varying sample sizes dependent on the favorite choice frequency.

Figure 10. Probability chosen, 2AFC task vs. rating choice vs. rating scale aggregation across configurations and styles.

Because the participants used the rating scale subsequently after selecting their favorite choice (amongst the five configurations for each unique content image-style image combination), the frequency of making this choice (probability chosen) was calculated as well. This distribution is shown in Figure 10 in the middle column labeled “rating choice,” denoting the favorite choice made for the subsequent assessment by the rating scale.

5.1. Global level analysis

A two-way ANOVA was conducted with configuration and style as repeated measures for the aesthetic preference scores measured by the 2AFC task and rating scales. This ANOVA represents a global level of analysis as the stimuli were either collapsed across styles (for configuration differences) or across configurations (for style differences).

Testing configuration differences, the 2AFC task showed a significant main effect of configuration for number of iterations (F = 86.786, p = 0.000), learning rate (F = 27.891, p = 0.000) and content-style ratio (F = 16.205, p = 0.000). In contrast, the rating scale yielded consistently higher significance values than the 2AFC task for all parameters, demonstrating an overall lower precision than the 2AFC task to detect aesthetic preferences. The rating scale found a significant main effect of configuration (F_config = 2.549, p_config = 0.040) and of style (F_style = 2.817, p_style = 0.026) for number of iterations, but neither for learning rate (F_config = 1.138, p_config = 0.332; F_style = 2.267, p_style = 0.063) nor for content-style ratio (F_config = 2.256, p_config = 0.064; F_style = 1.566, p_style = 0.184).

Testing style differences, contrary to configurations, expects to find non-significant differences among style images because this allows us to generalize across different style images. Here, the ANOVA results show that among styles, the 2AFC task found no differences for any parameter (F(4, 1350) = 0, p = 1) whereas the rating scale found no significant style either except for number of iterations (F(4, 250), p = 0.026). With the exception of the latter, we thus can generalize the aesthetic preference patterns across style images.

Lastly, the interaction between configuration and style was analyzed. Even though the 2AFC task found two significant differences (p = 0.000 for learning rate and content-style ratio), these results must be discarded because one of the terms (style) had a p-value of 1. This shows the significant results of the interaction terms derive mathematically only from the zero results of the configuration terms. The rating scale yielded consistently high p-values (F(16, 250), p = 0.509–0.873) for all interaction terms, revealing no significant differences. Taken together, neither measurement method supports that an interaction effect between configuration and style exists.

5.2. Distribution comparison 2AFC task vs. rating choice

The distribution of rating choice tremendously differs from the 2AFC task. It is apparent that the rating choice for the number of iterations and content-style ratio shows not only a ceiling effect but also a floor effect, as evidenced by distinct and significant differences to the neighboring configurations. For both these parameters, the scores on both extremities overlap, so there is no identifiable maximum along the scale range. For the learning rate, a peak can be identified at the middle configuration, consistent to the 2AFC task. However, as the standard error of this peak also overlaps with that of the minimum configuration, no significant maximum can be identified here either. The identified floor effect is thus a unique characteristic of the rating choice distribution that differentiates it from the rating scale.

A potential explanation for the observed distribution is that the rating scale scores could merely reflect the rating choice distribution. In other words, a lower (higher) probability chosen in the rating choice could result into a lower (higher) aesthetic preference score in the rating scale. However, the high scores on the left extremities in the rating choice are not reflected in high scores in the rating scale for any of the three parameters. This proves that the left extremity high scores in the rating choice represent indeed floor effects. In contrast, the lower rating choices between the extremities do not consistently reflect the aesthetic perception either—for example, the 4th configuration of parameter number of iterations is lowest score in the rating choice, but scores second highest in the rating scale. The patterns of rating choice and rating scale seem consistent for the learning rate, but the overlapping standard errors disconfirm the presence of a significant rating scale pattern.

Finally, we aggregated the favorite choices across styles to calculate the correlation between the rating choice frequency (number of favorite choices per participant and per parameter configuration) and the rating scale score. This correlation turned out to be weak (r = 0.09), further corroborating quantitatively the preceding qualitative observation.

Taken together, the aesthetic preference patterns found by the 2AFC task were not detected by the rating choice task or rating scale. The reason is that even after removing the acquiescence bias, both rating choice and rating scale tasks showed a distinct response bias towards the scale extremities which scored not significantly different from other peaks.

5.3. Configuration differences analysis

To compare the 2AFC task with the rating scale on detecting configuration differences, a one-way ANOVA on configuration, per style, was conducted as shown in Table 1.³

Table 1. ANOVA aesthetic preference ratings on configuration differences, per style, 2AFC task vs. rating scale.

Parameter	2AFC task						Rating scale
Style	Mean	SE	lwr.ci	upr.ci	F	p	Mean	SE	lwr.ci	upr.ci	F	p
Nr. iterations
Godami	2	0.072	1.859	2.141	26.316	0.000***	56.000	2.873	50.239	61.761	1.626	0.182
Feathers	2	0.076	1.850	2.150	14.916	0.000***	68.182	2.685	62.798	73.566	1.099	0.368
Toys	2	0.073	1.855	2.145	7.266	0.000***	61.273	2.351	56.560	65.985	1.112	0.362
Mosaic	2	0.074	1.855	2.145	27.884	0.000***	59.818	2.728	54.349	65.287	1.569	0.197
Houses	2	0.077	1.848	2.152	20.101	0.000***	57.818	3.186	51.432	64.205	0.945	0.446
Learning rate
Godami	2	0.078	1.846	2.154	8.907	0.000***	51.455	3.348	44.743	58.166	1.018	0.407
Feathers	2	0.077	1.849	2.151	19.554	0.000***	62.000	3.495	54.993	69.007	0.939	0.449
Toys	2	0.080	1.843	2.157	3.358	0.011**	55.273	3.301	48.654	61.892	1.424	0.240
Mosaic	2	0.079	1.844	2.156	19.469	0.000***	53.091	2.877	47.324	58.858	0.751	0.562
Houses	2	0.078	1.846	2.154	7.294	0.000***	51.636	3.479	44.661	58.612	1.089	0.372
Content-style
Godami	2	0.080	1.843	2.157	6.361	0.000***	59.636	3.670	52.279	66.993	1.463	0.227
Feathers	2	0.077	1.848	2.152	12.943	0.000***	69.455	3.020	63.400	75.509	0.209	0.932
Toys	2	0.080	1.842	2.158	23.554	0.000***	65.636	3.565	58.488	72.784	1.283	0.289
Mosaic	2	0.078	1.846	2.154	10.082	0.000***	63.455	3.493	56.451	70.458	1.228	0.311
Houses	2	0.074	1.854	2.146	0.774	0.543	58.182	3.885	50.394	65.970	0.406	0.803

***p < 0.001, **p < 0.01, *p < 0.05.

SE: Standard error of the mean; lwr.ci/upr.ci: lower/upper bound of 95% confidence interval; F: ANOVA test statistic.

5.3.1. Configuration differences-Number of iterations

The 2AFC task results show that people prefer neural style transfer generated images with more iterations. This can be evidenced by the small and non-overlapping error bars (which had been averaged over the five style images) in the visual depiction in Figure 10, and by the ANOVA analysis revealing statistically significant configurations differences for each style (Table 1, 2AFC Task, Nr. iterations), whereas the rating scales found none (Table 1, Rating Scale, Nr. iterations). Noteworthy is that the shape of the 2AFC task preference curve can be characterized by an “elbow” at 300 iterations, with a strong decline of the gradient thereafter. This suggests a convergence to a climax point near 500 iterations, further indicated by the decreased standard error.

In contrast, the rating scale shows a decline at 300 iterations, with higher probabilities both at 200 and 400 iterations. However, it is crucial to note that all the standard errors between 100 and 400 iterations are clearly overlapping (Figure 10), indicating the absence of significant differences (Cumming & Finch, 2005). Hence, the visible pattern is random and not subject to further interpretation. The only significant difference can be found at 500 iterations, which represents the scale extremity and therefore corroborates the presence of an acquiescence bias.

5.3.2. Configuration differences-Learning rate

The 2AFC task’s aggregated preference curve for the learning rate (blue) in Figure 10 marks an inverted U-shape with the peak in the middle configuration 2e1 (M = 2.560, SE = 0.079). Pairwise Wilcox Tests confirm that the peak at the middle configuration (2e1) is statistically different from the smallest (0.5e1: M = 2.015, SE = 0.0.078) and larger (4e1: M = 1.604, SE = 0.080, 6e1: M = 1.636, SE = 0.077) learning rates, but not for the second smallest learning rate (1e1: M = 2.185, SE = 0.062). Further, the difference among the small learning rates (0.5e1 or 1e1) and among large learning rates (4e1, 6e1) are not significantly different, and thus can be merged together into a small and large group. In conclusion, the middle configuration, in relation to this small and large learning rate group, forms an inverted U-shape pattern.

5.3.3. Configuration differences-Content-style ratio

Similar to the learning rate, the 2AFC task highlights a clearer pattern than the rating scale for the content-style ratio. Whereas the 2AFC task shows a double peak pattern at content-style ratios 100:100 (M = 2.262, SE = 0.055) and 300:100 (M = 2.371, SE = 0.077), the rating scale shows a strong peak at 300:100 (M = 69.412, SE = 2.824) but only slight elevation at the 100:100 (M = 60.976, SE = 3.709) configuration. The ANOVA on the global had revealed the between-group variance as 16.2 times higher than the within-group variance, thus supporting the 2AFC curve as a clear preference pattern for the content-style ratio.

5.3.4. Configuration differences-Visual inspection summary

The rating scale curves and the 2AFC task curves share a pattern of peaks found roughly at the same location (500 iterations, learning rate 2e1, content-style ratio 300:100). However, as indicated by the error bars, the rating scales show consistently larger standard errors than the 2AFC task. Furthermore, the rating scale distributions reveal a ceiling effect (Everitt, 2002), a bias incurred when people tend to prefer choices at the right (ceiling) or left side (floor) of a rating scale. This ceiling effect is indicated by the peaks for the number of iterations and content-style ratios.

Importantly, the differences between the remaining configurations are not significant, as indicated by the substantial overlap between neighboring configurations of the standard error intervals. Notably, this overlap is consistently higher than the corresponding overlap in the 2AFC task, further evidenced by the consistently higher p-values of the rating scales compared to the 2AFC task (Table 1) with the only exception for the houses style. Taken together, the rating scale does not identify an aesthetic preference pattern as clearly as the 2AFC task did.

5.3.5. Configuration differences - Summary

Overall, these findings provide evidence that whereas the 2AFC task can detect aesthetic preference patterns for each parameter that differ distinctly among configurations, the rating scales cannot. Furthermore, we can generalize the found patterns across style images with statistical evidence (Table 2, 2AFC task) for all style images (p = 0.000) except the houses style (p = 0.543).

Table 2. ANOVA aesthetic preference ratings on style differences, per configuration, 2AFC task vs. rating scale.

Parameter	2AFC task						Rating scale
Configuration	Mean	SE	lwr.ci	upr.ci	F	p	Mean	SE	lwr.ci	upr.ci	F	p
Nr. iterations
100	0.975	0.084	0.809	1.140	1.793	0.130	57.143	2.693	51.745	62.540	0.776	0.546
200	1.873	0.062	1.750	1.995	0.841	0.500	60.800	2.600	55.574	66.026	0.968	0.434
300	2.273	0.050	2.174	2.372	0.874	0.480	56.500	2.890	50.717	62.283	3.042	0.025**
400	2.404	0.057	2.291	2.517	1.635	0.166	61.163	3.397	54.307	68.018	0.309	0.870
500	2.476	0.073	2.333	2.620	0.440	0.779	66.818	2.454	61.917	71.719	1.585	0.190
Learning rate
0.5e1	2.015	0.078	1.861	2.168	5.834	0.000***	54.032	3.336	47.361	60.703	0.402	0.806
1e1	2.185	0.062	2.063	2.307	8.920	0.000***	51.579	3.686	44.110	59.048	1.317	0.284
2e1	2.560	0.079	2.404	2.716	3.077	0.017**	58.588	2.517	53.583	63.593	2.338	0.062
4e1	1.604	0.080	1.447	1.761	2.446	0.047**	50.200	3.664	42.836	57.564	0.348	0.844
6e1	1.636	0.077	1.485	1.788	7.571	0.000***	56.000	3.968	47.974	64.026	1.987	0.118
Content-style
10:100	1.644	0.102	1.444	1.844	3.097	0.016**	59.718	3.035	53.664	65.772	1.284	0.285
50:100	1.916	0.068	1.783	2.050	12.393	0.000***	58.421	4.141	50.031	66.811	0.469	0.758
100:100	2.262	0.055	2.153	2.370	8.389	0.000***	60.976	3.709	53.479	68.472	2.085	0.103
200:100	1.807	0.071	1.667	1.948	4.220	0.002***	59.130	4.352	50.105	68.155	0.662	0.627
300:100	2.371	0.077	2.219	2.523	11.279	0.000***	69.412	2.824	63.809	75.014	0.312	0.869

***p < 0.001, **p < 0.01, *p < 0.05.

SE: Standard error of the mean, lwr.ci/upr.ci: lower/upper bound of 95% confidence interval; F: ANOVA test statistic.

5.4. Style differences analysis

The previous analysis revealed that by collapsing the styles, the 2AFC task measures differences between configurations with more precision compared to the rating scales. In contrast, the style differences analysis collapses the configurations and thereby elaborates which preference patterns can be found between styles. To this end, a one-way ANOVA was performed for aesthetic preference on styles for each configuration (Table 2).

5.4.1. Style differences-Number of iterations

When comparing the aesthetic preferences between styles, the number of iterations delineated a common pattern as shown in Figure 11. Aesthetic preference increases with rising number of iterations, and this observation is consistent across the five style images with minor deviations for the toys style. The qualitative visual inspection of the outcome images (Figure 4) hints that the underlying reason might be the increasing detail in rendering the structural information of the face. Important from a practical perspective is that the aesthetic preference converges around 500 iterations. This means for practitioners that it is not necessary to run the algorithm for more iterations which saves unnecessary time and importantly, also saves considerable cost if cloud computing is used.

Figure 11. Probability chosen in 2AFC task for number of iterations 100, 200, 300, 400, 500.

5.4.2. Style differences-Learning rate

The analysis of the learning rate on style differences showed a differentiated pattern in the 2AFC task as displayed in Figure 12. Whereas the majority of style images had an aesthetic preference peak for a moderate learning rate (2e1), the feathers (blue) style image showed the peak clearly at a smaller learning rate (1e1). For this exception, the visual inspection (Figure 5) provided a plausible reason: The higher learning rate (2e1) rendered the right eye with a blue stain whereas the lower learning rate (1e1) marked the face with a blue feather on the cheek, leaving the eye unmarked. The different aesthetic assessments might be explained by people being more familiar with eye spots (similar to eye makeup) than cheek spots that could be more perceived as skin impurities.

Figure 12. Probability chosen in 2AFC task for learning rate 0.5e1, 1e1, 2e1, 4e1, 6e1.

5.4.3. Style differences-Content-style ratio

The style differences analysis of the content-style ratio revealed the highest variation across the five style images in the 2AFC task, and did not seem to show a common pattern visually (Figure 13).

Figure 13. Probability chosen, 2AFC task, content-style ratios 10:100, 50:100, 100:100, 200:100, 300:100.

We can speculate on some underlying reasons for the distinct patterns by qualitatively analyzing the peaks of the aesthetic preference curves. For example, the toys style (green) and mosaic (purple) show the peak at the highest configuration (300:100) because it is the only configuration that clearly renders the face with natural eye renderings and without any deformation. Further, the low aesthetic ratings for content-style ratios 50:100, 100:100, and 200:100 can be explained by their facial deformations.

Contrasting this are the styles godami (red) and feathers (blue) that both peak in the middle configuration (100:100) which, despite representing a less realistic rendition than the highest configuration (300:100), nonetheless shows a well recognizable facial structure.

The houses style (yellow) differs from this pattern with a roughly flat curve, further evidenced by the ANOVA revealing no significant difference between configurations on p = 0.543. This result can be explained by the clearly recognizable face and hair structure in all renderings. Somewhat unique is the style mosaic (purple) because it shows a double peak pattern. Whereas the highest configuration (300:100) shows the most realistic rendering, the smaller one (50:100) is more abstract than the configurations in-between (100:100, 200:100) while still recognizable as a face, unlike the lowest configuration (10:100).

5.4.4. Style differences-Summary

In summary, analyzing the style differences with the 2AFC task revealed a double peak pattern for the content-style ratio at the middle (100:100) and high configuration (300:100). The confirmed double peak pattern indicates that two different preference reasons exist for the content-style ratio: The middle configuration allows us to easily identify the person but in a strongly stylized form which could correspond to a preference for abstract arts, whereas the high configuration allows us to recognize the subtleties of a person's face close to a photograph, which could correspond to a preference for realistic renderings.

6. Discussion

The present work provides an initial exploration on the question whether and to which degree computer-generated arts can achieve optimal aesthetic results. In this sense, it is the first empirical application of the 2AFC task on computer generated arts in general, and neural style transfer in particular. It addressed the two identified problems in the literature of measuring the outcome of a neural style algorithm (the subjective assessment by the authors and the acquiescence bias associated with rating scales) by applying a previously not applied measurement instrument, the 2AFC task.

The research question did not aim for a generalization of aesthetic preferences across input images, but instead for the comparison between the 2AFC task and rating scales in their ability to extract aesthetic preference patterns ceteris paribus, i.e., given the same conditions. This explains why each of the three parameters was applied to only one input image each, and consequently why several face-related attributes (e.g., gender, ethnicity, facial dimensions) were not varied. Although the present work acknowledges these face-related attributes as relevant variables in aesthetic measurement, it focused on a different goal, namely to isolate each parameter’s aesthetic impact, and assess whether the results allow generalizing the relationship between perceived aesthetics and the parameter configuration for a particular combination of input image and style image.

In answer to the research question, the main contribution of this work is threefold.

First, the present study corroborated that the 2AFC task is a more effective method for detecting aesthetic preferences than the rating scale, even in absence of the acquiescence bias. It provided evidence that the 2AFC task detects patterns that are more statistically different between groups. Importantly, these patterns were not revealed by the rating scales even though they had been designed to avoid the acquiescence bias by the preceding rating choice task. This provides the empirical proof that the 2AFC is de facto a better measurement instrument for aesthetic preferences than the commonly used rating scales. This finding corroborates previous research declaring the 2AFC task the preferred measurement method for aesthetic preferences (Mantiuk et al., 2012; Palmer et al., 2013) to neural style transfer-images.

As a theoretical contribution, this finding is important because it demonstrates that seemingly similar aesthetic outcomes go unrecognized by the rating scale but can be measured precisely in their subtle differences by the 2AFC task. This can be understood further by the fact that the 2AFC task was initially applied in cognitive psychology to detect perception thresholds.

Second, the present work demonstrated the higher sensitivity of the 2AFC task by extracting three distinctly different patterns of aesthetic preferences. More specifically, these three distinct patterns showed

1. a convergence pattern (a continuously positive curve) for the number of iterations

2. a single peak pattern (an inverted U-shape) for the learning rate

3. a double peak pattern for the content-style ratio

Moreover, beyond the scope of the present work, these extracted patterns could contribute to theory building or corroboration of existing theories. As an example, the found double peak pattern may allow us to postulate that people favor either the proximity of the expected, i.e., a less artistic impression, or a more stylized and thus abstract image rendition. This theory is aligned with research on arts paintings that also found antagonist preferences for abstract vs. realistic arts and related them to trait differences (Pelowski et al., 2017; Wiersema et al., 2012).

Third, the implementation of the present study shows commercial applications a pathway to leverage the precise measurement of aesthetic preferences in the business context. For companies, it is crucial to know that the 2AFC task allows them to find parameter configurations that achieve optimal aesthetic perception. Companies can use these optimal configurations to achieve reproducible aesthetic outcomes that are customized for target groups with their specific aesthetic preferences. Moreover, the concrete recommendations for configurations effectively eliminate the time-consuming tuning of hyperparameters.

6.1. Limitations and future research

The findings of this study regarding optimal parameter values are specific to the combination of the algorithm implementation, the parameter configuration of this algorithm, and the selection of input and style images.

The found aesthetic patterns must be further limited to the sample which constitutes a selection bias. For example, the selection of design professionals and design students might have a higher level of visual sensitivity compared to the general population. The selection bias also applies to the used algorithm, so the findings are not generalizable to other neural style transfer algorithms. Furthermore, controlling for variables such as gender, age or education could allow future research to detect individual differences. Likewise, future research could test other explanatory variables for interaction effects such as a general preference for abstract vs. realistic arts, celebrity preference, personality trait preference, or any specific aesthetic preference including color and geometric shape of style images.

In addition, future research could yield new insights by testing whether specific visual aspects are related to neural style transfer parameters, such as perceived balance (Hübner & Fillinger, 2016) and learning rate, or visual simplicity (Mayer & Landwehr, 2018) and content-style ratio. Furthermore, future research could measure the perception of the style image and systematically vary its textural granularity. This approach will be particularly interesting when analyzing content motifs that differ from the present study to investigate whether similar preference patterns emerge. An underlying factor, subject to empirical investigation, could be the familiarity of the content image to the observer because knowing the structural information well may reward its higher recognizability in the image. Therefore, it can be conjectured that if the observer is well acquainted with the content structure, such as with famous sight seeing scenes or paintings, the aesthetic preference could favor realistic renderings for such motifs.

Lastly, it would be interesting if future research could replicate the findings with other neural style transfer implementations. A good candidate for replication is the preference for a higher number of iterations as it seems to be associated with a preference for a more finely rendered style image’s texture in the output image. Whether the content-style ratio pattern can be replicated depends on whether there is an underlying binary preference for realistic vs. abstract arts. The results for the learning rate might not replicate unless the cost function is shaped by similar parameters as the algorithm used in this study.

7. Conclusion

The findings of this work demonstrate that aesthetic preferences of computer-generated arts, such as neural style transfer, underlie distinct preference patterns. Varying the parameter configuration results in different aesthetic perception which is only distinguishable if measured by the 2AFC task. The practical insight for commercial contexts is that optimal aesthetic outcomes can indeed be parametrized by testing the aesthetic perception with the 2AFC task and extracting the optimal parameter configuration. This configuration allows the neural style algorithm to generate visual artefacts which could be customized for a specific target audience if their preferred style images are used as input. Such customization would effectively implement the guiding HCI principle of user-centered design (Vredenburg et al., 2002). In response to Harper's (2019) call for putting AI in the service of HCI, this work demonstrated how to achieve this goal by systematic stimulus variation and capturing corresponding aesthetic patterns.

Acknowledgments

The author wants to express deepest gratitude to Prof. Karen Muckenhirn for proofreading subsequent versions of this manuscript, and to ML engineer Lars Sjoesund for feedback on the deep learning parts.

Additional information

Notes on contributors

Chaehan So

Chaehan So is an Assistant Professor of Information and Interaction Design at Yonsei University. His research focuses on the intersection of psychology, artificial intelligence and design to create psychologically-grounded ways of user experience. Dr. So has a Ph.D. in Social Psychology and Masters' degrees in electrical engineering and business.

Notes

1 For simplicity, this paper refers to the hyperparameters of a deep learning model as parameters, knowing that hyperparameters (which are used to configure the training process) shall not be confused with model parameters (which are learned from the dataset).

2 The role of the optimizer in the used algorithm implementation (J. Johnson, 2016) is for optimizing the image, and not for training the full network.

3 Due to the same number of pairwise comparisons (10) for the five configurations tested, the mean for the 2AFC task shows the identical value of 2 for all one-way ANOVAs on configuration.

Appendix A

 

Figure A1. QQ plots configuration differences by_style, 2AFC task.

Figure A2. QQ plots style differences by configuration, 2AFC task.

Figure A3. QQ plots configuration differences, rating scale.

Figure A4. QQ plots differences by configuration, rating scale.