A user-centered approach to evolutionary algorithms and their use in industry

Abstract

The key contribution of this article to the domain of engineering optimisation is establishing best practices with respect to the development of evolutionary algorithms in the context of design engineering. Despite their various uses, the uptake of evolutionary algorithms in industry remains limited. In order to understand why uptake is low a survey of engineers was undertaken, the results of which are presented here. A total of 23 participants (N = 23) took part in the 3-section mixed methods survey. Reflexive thematic analysis was conducted on the open-ended questions. A common thread throughout participants responses is that there is a question of trust towards evolutionary algorithms within industry. Perhaps surprising is that the key to gaining this trust is not producing good results, but creating algorithms which explain the process they take in reaching a result. Based on this, recommendations have been made to increase their use in industry.

Keywords:

• reflexive thematic analysis
• human in the loop
• optimisation algorithms
• design optimisation

PUBLIC INTEREST STATEMENT

A survey was carried out to find out what engineers and students want from optimization algorithms and engineering design tools. The answers come from a variety of experience levels and proficiency levels. Based on their answers, we put together some recommendations to make these algorithms more accessible to engineers and students. This work is important because it provides data from those who use optimization algorithms and design tools every day. For academics, it provides a base from which further work can be done to explore the effectiveness of the recommendations. This publications gives developers a citable source to justify their decisions when it comes to making optimization algorithms.

1. Introduction

Evolutionary algorithms (EAs) are used to optimise parameters for which an exhaustive search of the parameter space would not be possible, either because of a resource constraint or a time constraint. This is important because it allows for larger and more complex optimisation problems to be solved. They have been used in a variety of applications industrial applications, such as aerospace design (Berrezzoug et al., 2019). Despite this, the uptake of EAs in industry remains low.

Table 1. A breakdown of roles of participants

Role	Count	Percentage
Student	9	32%
Researcher	1	3.6%
Design Engineer	8	29%
Other Engineer	7	25%
Manager	3	11%
Retired	0	0%

Table 2. A breakdown of education levels of participants

Education	Count	Percentage
Secondary School	1	4.3%
Undergraduate	12	52%
Postgraduate	10	44%

Table 3. A breakdown of experience levels of participants

Experience	Count	Percentage
Less than 1 year	2	8.7%
1 to 2 years	7	30%
3 to 5 years	6	26%
6 to 10 years	1	4.3%
More than 10 years	7	30%

Table 4. A breakdown of proficiency levels of participants

Proficiency	Count	Percentage
Not Proficient Whatsoever	0	0%
Minimal Proficiency	2	8.7%
Some Proficiency	2	8.7%
Proficient	8	35%
Very Proficient	10	44%
Extremely Proficient	1	4.3%

Table 5. A breakdown of how participants heard about the survey

Source	Count
Reddit	1
Twitter	6
LinkedIn	5
Facebook	4
Email	5
Referal	2

Table 6. A list of terms that were presented to participants and the frequency with which they had heard of the terms

Terminology	Count
Genetic algorithm	15
Heuristic search algorithm	10
Evolutionary programming	15
Selective search	4
Gradient descent	11
Simulated annealing	9
Adjoint state method	3

Academic literature has shown that the use of EAs can add value in industry (Berrezzoug et al., 2019; Kaveh et al., 2016). The integration of designers in the design loop has been shown to be possible (Guo et al., 2019; Umetani & Bickel, 2018). The results, however, have been focused on the quantitative aspect of the application rather than the user experience. Furthermore, the best way to integrate these algorithms in the design loop remains unclear.

To try and understand why the uptake of EAs in industry is so low, the following research questions were posed:

(1) To what extent is there a pre-existing sentiment (negative or positive) among practitioners towards evolutionary algorithm-based design?

(2) What are the requirements of students, engineers and managers with regard to design optimisation and the design optimisation process?

The recommendations in this article are based on user-centred design principles (Norman, 1988) and on the data gathered in the survey. The analysis of certain data was done using reflexive thematic analysis (Braun & Clarke, 2006). This form of analysis is used to identify important words or phrases within an open-ended answer. It is then used to group similar answers together. The groupings are referred to as themes. The steps of reflexive thematic analysis are outlined in Section 3.3.

As a result of answering the research questions posed above, the contributions of this article are new insights into practitioner attitudes towards EAs and proposed best practices for developing EAs. This article is the first time, as far as we are aware, that quantitative and qualitative data concerning practitioner requirements for EAs have been published. Results presented here are beneficial to the EA community since they provide empirical grounding and justification for future EA development. In addition, it is beneficial to understand how practitioners want EAs to be designed in order to increase adoption in industry.

The rest of the article is structured as follows. Section 2 provides the reader with important information and background literature. Section 3 gives an overview of the survey, an analysis of the participants, and a review of the method used to analyse results. Results from the survey are presented in Section 4. In Section 5, a discussion of the results is given. Recommendations based on the results of the survey are presented in Section 6. An evaluation of our approach is given in Section 7, and finally concluding remarks are made in Section 8.

2. Background

2.1. Evolutionary algorithms and their applications

Evolutionary algorithms (EAs) have proved to be a powerful tool for optimisation (McGookin et al., 2000; Oduguwa & Roy, 2002; Wang, 1997). The performance of EAs is often measured using engineering benchmarks such as the design of a simple cantilever beam (Mirafzal et al., 2016) or a pressurised vessel (Zhang et al., 2018). Standard benchmark functions such as the Ackley Function (Ballester & Carter, 2004) are also used to demonstrate computational efficiency. More elaborate applications like aerodynamic design (Evans et al., 2019) showcase the range of applications of EAs.

EAs are particularly well suited to Multi-objective optimisation (MO) problems. Many engineering problems can be classified as MO; these designs involve many different trade-offs. It is often challenging for human designers to determine an optimal solution without the aid of a computer-based system. One example of an MO problem where a EA was used successfully is presented by Tawhid and Savsani Tawhid and Savsani (2018) who evaluated their Artificial Algae Algorithm on 20 different benchmark problems to show its suitability for engineering design optimisation problems. Zi-Min and Wang (2020) extended the use of their algorithm to large-scale MO problems. An industrial application of MO engineering can be found in aerospace, specifically the design of geostationary satellites. Customer demands for higher bandwidth capabilities and longer lifespans (Airbus, 2020, Airbus) are at odds with the many constraints in place. These constraints include

• Higher power requirements;

• Thermal considerations, including hot and cold zones;

• The ability to fit in a rocket fairing; and

• The ability to withstand vibrations during launch.

Each of these variables need to be considered throughout the design lifecycle of the satellite. To illustrate the usefulness of EAs in this context, Berrezzoug et al. Berrezzoug et al. (2019) proposed a method which applies a Gravitational Search Algorithm to the design of a satellite by considering the many design variables. The approach proved useful for determining optimal trade-offs between the variables.

The optimisation problems presented above are bounded by the constraints put on them by external factors, such as launch conditions for a satellite. In the following section, the constraints that designers face, such as resources and time, will be explored. These factors present barriers for the designers when attempting to optimise their designs.

An important caveat to algorithms named after nature metaphors is that they are not necessarily unique (Sörensen, 2015). The mixing of metaphors has been criticized as a deterrent to furthering the research area (Weyland, 2015). The algorithms mentioned in this article are not exempt from this criticism.

2.2. The relationship between humans and algorithms

Rather than acting as autonomous agents, EAs can be integrated into the design process to enhance the capabilities of human designers. In recent years, a growing trend of combining the expertise of designers with algorithms has been taking place. For example, Ant Colony Optimisation was applied to the design process of the layout of train tracks (Flurl et al., 2016), where the expertise of the engineer was combined with energy simulations to determine optimal solutions. The layout can be optimised to reduce the energy consumption of the trains as they travel the tracks. Furthermore, Flurl et al. (2016) proposed a method to allow design engineers to see the effects of their changes on energy consumption in real-time. This method takes advantage of the algorithm’s ability to find an optimal solution while allowing the designer to stay in control of the track layout.

The push for user-centric design is evident in engineering as well as other disciplines such as architecture. In a 2017 survey of 165 architects, 91% of respondents would like to see an inclusion of a “human-in-the-loop” approach with regard to the software and technologies they use (Cichocka et al., 2017). Furthermore, 93% of respondents wanted to be able to understand the underlying principles of the algorithm while just over half (54%) would like full control over the process. In an attempt to bring this to fruition, Berseth et al., 2019) proposed an interactive Computer-Aided Design (CAD) optimisation program. It engages with architects by analysing their designs and optimising for a number of variables such as “open space in passages, aesthetic relationships, or building codes.” The study presented here investigates similar concepts to the study done by Cichocka et al. (2017). A human-first approach has also been used in engineering. In fact, a user-centric approach was adopted by engineers at NASA (Burnell et al., 2020) and allowed engineers to evaluate the interdependencies that different systems had to each other with respect to the mission objective.

By combining human expertise with algorithms, designs that are both optimal and adhere to all design constraints can be produced. This is demonstrated by Guo et al. (2019) for bolt supporting networks. Users could select from a range of solutions which were fed back to the algorithm to develop a solution that was strong, cost-effective and manufacturable. Oulasvirta Oulasvirta (2017) considered user-centric design when applied to User Interface (UI). One of the key points raised with the approach was that the “designer can steer and redefine the tasks intermittently as results stream in.”

Thus, an optimum between designer and computer also exists; a region in which the algorithm is able to perform the necessary calculations while the human designer can focus on producing a result in line with their vision. The real-time approach is adopted by Umetani and Bickel (2018) and is applied to automobile aerodynamics. The changes made by users and the effects those have on the drag characteristics of the vehicle can immediately be fed back to the designer.

Incorporating humans in the optimisation process indirectly involves designers from outside of the design process. This is because engineers do not work in isolation, but rather as a team. Adjoul et al. (2019) observed that optimisations in the production phase required experts from both production and design to work together to create an optimal product.

The entire life cycle of a product must be considered when dealing with optimisations. An instance of this is probed by Kang et al. (2019) while considering the trade-offs faced by customers when purchasing a new car. Their techniques “could readily lead to crowdsourced, real-time, manufacturer-feasible design optimisation,” which includes customers, designers and engineers.

2.3. Human-centred design

The research, findings, and recommendations presented in this article are based on the related works in user-centred design. Human-computer interaction (HCI) investigates how users behave with digital technology and a component of HCI is user-centred design (UCD). One of the earliest works in this field which put forth user-focused design principles was by Norman (1988). Shneiderman (1986) listed 8 ’Golden Rules’ for interaction design. Both works contributed heavily in establishing best practices within HCI and UCD. This also helped establish UCD as a prominent field in Computer Science. Others, such as Gulliksen et al. (2003), have built upon these design principles and developed principles which detail the way in which the UCD process should be undertaken. Authors such as Rogers have also helped develop the field with books which explore the theory behind HCI concepts and designs (Rogers, 2012).

A design cycle is used to structure ideas from infancy to maturity. There are 4 main stages in a UCD cycle:

1. Identify: Who is going to be using the product?

2. Specify: What are the requirements of the product?

3. Design: Create potential solutions.

4. Evaluate: Do the solutions address the needs of the users?

In this article, the first three stages are done. Participants who use optimisation algorithms and processes in their work have been recruited. A survey was designed to capture the requirements for these algorithms. Finally, recommendations are presented as potential solutions.

There are a range of tools to put users at the centre of a design. These include usability testing and prototyping. In this article, the UCD tool used is a survey. This is a preliminary tool also for initial data gathering. Some benefits of surveys include their accessibility and low cost.

3. Materials and methods

A survey was designed to gather data to help answer the research questions posed. Details of the survey are available in the following Section. In the Materials Section, an analysis of the participants who partook in the survey is given which includes their experience, jobs, education, experience and proficiency levels. Lastly, an overview of reflexive thematic analysis is provided.

3.1. Survey

The survey consisted of 3 Survey Sections. Survey Section 1 was used to gather non-identifying traits about the respondent such as their experience in the field and their roles. Survey Section 2 gauged the respondents prior knowledge of the domain as well as their preferred characteristics for design tools. Survey Section 3 of the survey had 6 optional, open-ended questions aimed at determining the current state of their design process and the respondents attitudes towards algorithmic design aids. There was no time limit for responses, however, 20 minutes was suggested as an estimate of time for completion.

The analysis of the results have been broken down into three parts to reflect the structure of the survey. Thematic analysis has been used for Survey Section 3, while Survey Sections 1 and 2 can be analysed quantitatively. A total of 23 participants (N = 23) answered the survey over the course of 9 days. All participants answered all questions in Survey Sections 1 and 2. Some respondents chose not to answer some or all questions in Survey Section 3. The total number of responses per question in Survey Section 3 are given along with the result.

The survey was created electronically using Google Forms. A link to a survey was posted on various social media sites (Twitter, LinkedIn, Facebook, and Reddit) to gather responses; emails were also sent directly to those with known prior domain experience. Ethics approval for this study was granted by the Swansea University College of Science Ethics Committee (SU-Ethics-Student-110,620/2921).

3.2. Materials

Participants were asked to categorise themselves based on their current roles with a summary available in Table 1. The sum of the responses exceeds 23 as some participants may feel that more than one category applies to them. More than half (N = 13; 57%) of the respondents considered themselves as engineers, with roughly a third (N = 8; 34%) specifically selecting “Design Engineer.” Some participants (N = 5; 22%) selected more than one descriptor.

Participants were asked to state their highest level of education, summarised in Table 2. All participants but one (N = 22; 96%) hold either an undergraduate or postgraduate degree. Slightly under half (N = 10; 43%) of participants hold a postgraduate degree. The specific degree was not probed.

In addition to their education, respondents were asked to state their length of experience in the field of engineering design (Table 3) as well as their self-declared proficiency. The proficiency was declared on a 6-point Likert scale (from “Not Proficient Whatsoever” to “Extremely Proficient”). Of all participants, just under a third (N = 7; 30%) indicated they had more than 10 years of experience in this field as seen in Table 4. Moreover, most (N = 19; 83%) participants indicated they are at least “proficient” in engineering design. Over half (N = 10; 43%) of those 19 would say that they are very proficient while one (N = 1; 4%) participant indicated extreme proficiency in engineering design.

Respondents were asked how they had heard about the survey with a breakdown available in Table 5. This questions served as a means to determine which platform attracted the most participants. Five (N = 5; 22%) of the participants recruited via email have an existing professional relationship with one the authors. The potential bias and mitigating actions are discussed in Section 7.

An additional question to determine how familiar respondents were with the field prior to answering was posed. To do this, some common and other, more niche terms were selected by the authors to gauge that prior knowledge. The question listed terminology associated with evolutionary algorithms and are listed in Table 6. The most frequent terms that participants had previously encountered were “genetic algorithm” and “evolutionary programming.” Both had been selected 15 times. The least frequent terms were “selective search” and “adjoint state method.” These terms were selected 4 times and 3 times, respectively. One (N = 1; 4%) participant did not select any terms and none of the participants selected all the terms.

3.3. Thematic analysis

The technique used to evaluate the results of the open-ended questions in this survey is reflexive thematic analysis. The process behind the evaluation is based on the work of Braun and Clarke (2006). The steps outlined have become standard practice to conduct similar research in areas such as psychology, sports science, and engineering (Braun et al., 2016; Memarian & McCahan, 2018; Terry et al., 2017). A brief overview of each step is given.

(1) Initially, a general familiarisation of the data is conducted to get a general overview of the contents. A rough idea of codes is written down and used as a starting point for the next step.

(2) Once the data has been familiarised, the process of coding begins, which is the labelling of the data that appears interesting. This can be one or two words within the data or a short segment that conveys a narrow idea.

(3) Once the data has been coded, the codes themselves are grouped together in general themes. The relationship between themes is also explored at this stage.

(4) Themes are reviewed at this point. Some themes may need to be split, combined or otherwise adjusted. Importantly, a pattern must be evident within the theme among the codes. A complete review of the data is also done at this point to bridge any gaps that may have been missed initially.

(5) Themes are concretely defined at this step. Again, themes may be restructured to ensure that each theme has one central idea.

(6) Lastly, the analysis of the data is written up. The analysis is broken into individual themes using codes and data to support the arguments.

Three criteria were followed to ensure the highest quality of analysis as outlined by Nowell et al. (2017). The criteria are credibility, transferability and dependability. Through the realisation of the three metrics, then a fourth metric is fulfilled according to Guba and Lincoln (1989): conformability. To guarantee that these metrics are being met, Nowell et al. (2017) suggested an audit trail is set up so that any researcher could follow the logic used by researchers. Also suggested is that researchers remain critical of their own findings. Analysis of the results was done independently and electronic records of the analysis were shared among the authors.

Acknowledging internal bias is key to maintaining credibility. In an update on their 2006 article, Braun and Clarke (2019a) describe researcher bias as an inherent component to thematic analysis and point out that it is the researchers who generate the themes rather than the data itself. Knowledge of one’s own bias is 1 of 9 recommendations provided by Castleberry and Nolen (2018). As per the recommendation, bias is discussed in Section 7.

4. Results

4.1. Rating Criteria for Design Tools

Survey Section 2 contained five questions which assessed the respondents’ preferences with regard to engineering design tools. The criteria are user interface, versatility, robustness, frequency of use in industry, and supporting documentation. The questions were evaluated on a 3-point Likert scale; the participants had the option to choose from “no importance,” “some importance” and “high importance.” An additional option of “no opinion” was provided for respondents.

Figure 1. User preferences of 5 unique design tool criteria, ranked on a 3-point Likert scale with an additional option of “no opinion.”

As seen in Figure 1, the two most important criteria according to participants was user interface (UI) and robustness (Ro). Both of these criteria had a majority (UI: N = 15; 65%, Ro: N = 14; 61%) of respondents mark them as highly important. With respect to UI, the participants declared that it is at least somewhat important when considering design tools. From the answers given by participants, the criteria can be ranked: user interface, robustness, supporting documentation, versatility, and frequency of use in industry. This ranking however does not negate the need for any of the criteria as all had been ranked as somewhat important by the majority of participants.

4.2. Open-ended questions

4.2.1. Question 1: describe your current process for optimising designs; include proportion of time spent on each stage if possible

A total of 16 participants answered the first question. Coding could be done on 12 of the answers. The remaining 4 answers contained insufficient detail to accurately assess the meaning and intention of the respondent, or did not answer the question in a coherent manner. The incoherency of answers in this question and following questions had to do with a lack of any sentence structure. As an example, one answer contained only “System generative design.” The first open-ended question was geared towards answering the second research question.

When discussing the current state of their design processes, participants approached the question from different angles. 4 included a human element, 7 from a product perspective and 3 made reference to technology. 2 answers had two themes present. Many of the answers that fell under the product theme were heavily focused on the design itself. This included talking about the requirements of the design, refinement and optimisation. Participants were also asked to included proportions of their time spent at each stage of the design process. 3 participants gave a breakdown while 1 participant only noted the longest stage of the process. For these 4 participants, iteration/optimisation/redesign takes a significant portion of their time. Some participants also included other considerations into the design process such as designer experience, designer preferences and customer/supplier relationships. With regard to customer requirements, participants said that those requirements are used as a starting point for high-level, initial designs. Optimisation and iterative design are significant portions of the design process. When discussing their design process and the time spent at each stage, one participant said that in their experience “the detailed subsystem design is usually the longest stage” and that “an iterative evolutionary algorithm would speed up this process.” This is in conjunction with the previous statement that this stage takes up 25% to 40% of the total design process time.

One participant, a manager, also noted that finding an optimal solution is “highly unlikely” although admitted that solutions could be found that are quite close to the optimum. This is in line with another participant, an engineer, who mentioned Multidisciplinary Optimisation (MDO) which invariably results in compromises. These in turn lead to less-than-optimal designs. Of course, this is acceptable as long as the design requirements are met.

4.2.2. Question 2: what barriers do you have to overcome during the design process?

This questions was answered by 16 participants. Again, this question was related to the second research question.

Most (N = 11; 69%) of the participants’ answers in the second question fell under the product theme. 5 participants explicitly stated that cost was a barrier to their design process. Other resources, such as time and processing power were also mentioned. Previously, it was noted that some participants use the customer requirements as a starting point for their initial design. For one participant, this was incidentally their biggest obstacle. This lends itself to another problem, where the customer does not have clear requirements. Both optimisation algorithms and engineers rely on a clearly defined objective around which to construct their solutions.

One engineer relayed that selecting weights for a fitness function was a barrier to their design process. This is indicative of an over-reliance on the expertise of the engineer and a lack of technology meeting its potential. Spending extra time determining weights takes time away from engineers adding value in their area of expertise.

Both a student and an engineer mentioned competing design requirements as a barrier. Incorporating conflicting requirements can be challenging, especially when, as mentioned earlier, requirements can be unclear or ill-defined.

Another two participants, a student and a design engineer, mentioned their own knowledge as a limitation to the design process. For the student, being confident enough to be able to get their design from a mental model to a CAD model was a hurdle. For the engineer, the “learning curve of tool sets” was one of the obstacles. These two answers indicate that part of the design process could be improved by a better learning experience.

Two instances of trust were brought up: one from an engineer and another from a manager. The first had to do with trust in the technology itself, specifically design simulations. The results from simulations must be accurate so that engineers can make informed decisions about the design and ultimately the final product. The second instance of trust was related to the participant’s peers. Getting buy-in from new colleagues was mentioned as a barrier. For this participant, trust was instilled as a result of the outcome of the project. This is one manner in which EAs can gain an engineer’s trust, although it would still require a high level of trust from key early adopters.

4.2.3. Question 3: what comes to mind when you think of evolutionary algorithms?

This question was answered by 16 participants. The intention of this question was to gauge whether respondents have any prejudices or preferences towards algorithms as a part of the design process. This question was in line with the first research question.

The answers in this question came under two themes: technology and human. The count for each was 10 and 2 respectively. 4 answers were not coded: of these, 3 pertained to biology and did not fall in among the themes while 1 answer was too short to accurately determine the respondents intention. The answers which related to biology were not put into a theme because the creation of an additional theme did not seem appropriate for 3 out of 101 total answers.

Two managers expressed a negative sentiment towards the term used in the question: evolutionary algorithm. The first participant stated that it is “over-rated in practice” and preferred more rudimentary optimisation methods; pattern search was given as an example. The second participant questioned the ability for evolutionary algorithms to add value. Both participants point to a hurdle that new technology faces: convincing stakeholders that learning a new technology or system is worth the time investment. This sentiment was echoed by an engineer when responding to the last open-ended question. By noting the shortcomings of the current state of EAs, a clear pathway to wider adoption can be determined by addressing their concerns.

Other participants related the evolutionary algorithm to an automated processes or global optimisation. Three different participants mentioned Artificial Intelligence (AI) and Machine Learning (ML). Seven participants used a variation of selective process, biology and survival of the fittest in their answers. These answers indicate that developers of evolutionary algorithms should include a succinct explanation to let engineers have a better understanding of their mechanisms.

4.2.4. Question 4: do you trust the designs produced by automated optimisation algorithms? explain your answer

This question was answered by 18 participants. As with the previous question, it was aimed at answering the first research question.

Participants could very easily be categorised in this question. 13 participants expressed varying sentiment of doubt regarding the result produced by an algorithm. As an example, participants noted that they prefer to check the results of an algorithm by hand. One design engineer extended this by stating that the results would have to be validated by a real world test. This is an example of maintaining the human element in the design process. Keeping the human element in the process is noted in nearly every answer. Another design engineer noted that “there are some processes in optimisation that require experience and intuition,” which is a direct call to designer expertise. Likewise, one student incorporated the entire lifecycle of the product in their answer; cost, manufacturing and feasibility for human use. A similar sentiment was shared by another student in that some results may be outside of the constraints of the design space.

Three participants stated an outright trust in results generated by algorithms. One of these respondents, an engineer, states that the results are “usually overchecked.” This is contrary to another engineer’s answer, who stated “a ‘blind’ trust [in automated optimisation algorithms] is a bad approach.” Both sides point to the need for a proper explanation of the potential applications of these algorithms as well as the limitations.

This question also revealed some feelings that designers have towards optimisation algorithms. Designers will not trust the final design unless they can understand the process the algorithm took to yield that result. 76% of respondents expressed a lack of trust to some degree. 8 participants stated that they would only trust the results of an algorithm if the results were verified by a person; 4 of those participants explicitly used the word “experience” as a designated requirement for trust.

4.2.5. Question 5: do you think a computer-based algorithm could help your design process? Explain your answer

This question was answered by 18 participants, although one answer can be discarded as the respondent simply stated that the question was not applicable. This question was related to both research questions.

Two thirds (N = 12; 67%) of participants expressed a positive sentiment towards working with algorithms in their design process. Much like the previous question, the degree to which participants wanted to incorporate this technology varied. One manager stated that they have already incorporated optimisations into their process. A student declared that they would use it only for the initial design stage. The development of this technology should be able to cater to those who want to use it for one or all parts of during their design processes.

One design engineer gave requirements in anticipation of such technology being incorporated in their process. The most important requirements for this participant was a properly designed UI and a link to other engineering tools. A seamless experience allows engineers to focus on designing rather than debugging. It also avoids user frustration.

4.2.6. Question 6: do you have any reservations about implementing more computer-based assistance in the design process? explain your answer

This question asked participants if they have any reservations with regard to implementing more optimisation algorithms in their design processes. It was related to the first research question and was answered by 17 participants.

Each answer could be attributed to the human theme, though through various degrees. One student drew comparisons between other software such as CAD and Computational Fluid Dynamics, noting that the design process is already heavily computer based.

An engineer and a student are concerned about the reliance that future engineers and designers will have on systems like EAs. The engineer mentioned that these algorithms have “disengaged the brains of engineers” which threatens a long-term consequence of engineers that have poor design experience.

Two participants, a student and an engineer, cited job security as a reason why they would not want this technology implemented. This is a very real concern that needs to be addressed by properly explaining the intent of developing these tools.

Another engineer was very supportive of the idea, mentioning that they have been advocating for this kind of technology for the last 20 years. It is an accurate reflection of the challenge of getting new technology to be adopted by individuals and by organisations alike.

4.2.7. Open-ended question summary

From the thematic analysis, 3 themes were generated and were given broad names to serve as an overarching link: human, product, and technology. The associated codes are visualised in Figure 2.

Figure 2. A mind map of the themes developed based on the survey data. the nodes emerging from the three main themes (human, product, technology) are the codes which were generated from participants’ answers. the size of the bubbles has no correlation to the number of mentions and indicates hierarchy only

The amount of mentions per theme are summarised in Figures 3 and 4. Discrepancies in theme mentions can be explained by a varying number of participants in each category. The number of mentions also exceeds the total number of answers as some answers contained more than one theme.

Figure 3. The raw count of mentions per theme by each different role in all questions

Figure 4. The raw count of mentions per theme by each different experience level in all questions

5. Discussion

A common thread throughout participants responses is that there is a question of trust towards EAs within industry. Perhaps surprising is that the key to gaining this trust is not producing good results, but creating algorithms which explain the process they take in reaching a result. Participants have expressed a desire to remain in the design loop. This is at odds with the motivation of a portion of the EA community of removing humans from the loop (Shahriari et al., 2015). It is clear a different approach is needed to increase industrial uptake.

The participants in this study have demonstrated through their answers that there is a general distrust towards EA-based design in industry. 72% of answers expressed doubt when discussing whether designs produced by an optimisation algorithm could be trusted. Their reservations have to do with the unproven state of these algorithms in their own personal experiences. However, respondents also recognise the potential of this technology, with 76% of respondents saying they could see the value of integrating EAs somewhere in their process. Of the respondents that expressed doubt, 69% of them noted that these algorithms could also help their design process. Separately, participants noted that the inclusion of human expertise is vital for adoption.

Much to the surprise of the authors, the answers between the different groups in this study (students, engineers, managers) and experience levels did not vary significantly. This can be seen in Figure 3 and 4. The different groups in this study shared the same thoughts across a variety of questions, especially when considering the two research questions. The most notable example of this is that both a student and an engineer were concerned about their jobs being replaced or made redundant by automation.

Recognising that trust is important in the development of new algorithms and the integration of such algorithms in industry is key to their adoption. Participants noted that trust plays a big role in the adoption of new technologies. One participant had a clear mistrust of new algorithms. Others mention that the need to be able to review the results and the decisions that an algorithm takes is a priority. When asked whether they had any reservations towards optimisation algorithms, three participants explicitly said yes. One of these participants went on to ask a series of questions about the ethics of the decisions made by an algorithm. Questions included “Who would be liable for a death or damage caused by a design made by this algorithm?” and “Are we going to write a decisions rules book?” which clearly demonstrates that designers are not only aware of the decisions they make with respect to their design, but also to the wider consequences.

The sophistication of the design process was also different across the range of participants. Two participants had a more traditional approach when refining their design, opting for “trial and error” and “one factor at a time.” These approaches are useful for exploring the entire design space and are a mix between designer experience and technology. However, they can be very time-consuming processes, especially on complex designs. While a Design of Experiments (DoE) is generally suitable for proof that a global optimum has been achieved, a trial and error method is not.

6. Recommendations

The participants in this study have shown with their answers that there is one key element which will determine the scale of adoption of EAs in industry—trust. This principle is used as a foundation for the following recommendations.

6.1. Algorithm developers should trust designers and engineers

By trusting designers to know what’s best for them, developers of new EAs should engage with experts in creating solutions aimed at solving their optimisation problems. While the development of new algorithms is important in addressing different optimisation challenges, testing the performance of algorithms must go beyond benchmark tests and include industrial problems for algorithms to realise their potential. Moreover, the reservations that participants had towards these algorithms point to a general need for algorithms to have suitable explainability and transparency. These criteria correspond closely to the requirements set out by those researching the ethics and law of AI (European Commission). The domain of evolutionary algorithms could co-opt the proposed regulatory framework of AI as a starting point as these technologies become more prevalent in industry.

6.2. Algorithm developers should incorporate human-computer interaction research

Throughout the survey, the need for human involvement in the development process of EAs is brought up; it demonstrates a need for a user-centric approach. Human-Computer Interaction (HCI) is the field of study which concerns itself with researching the relationship between users and digital technology. This can first be observed in Survey Section 2, where respondents identified a good UI as the most important factor when considering a design tool. The expectations from designers is that any additional tools that are made must integrate into their existing design process. This is articulated by a participant who needs an “intuitive set-up and an interface to existing tools.” This can be seen as a direct answer to the second research question which enquires about the design requirements of domain experts. These tools need to also cater to novice users such as students. A user-friendly interface can lower the barrier of entry to EAs. An engagement between HCI researchers and EA researchers can further these algorithms towards a higher adoption rate in industry.

6.3. Algorithm developers should be accountable for their algorithms

The need for solutions to be developed in conjunction with the designers who will eventually be using these tools is evident in the answers of participants. Integrating constraints in the design optimisation process was a recurrent subject among participants’ answers. Simplifying the problem down to something that could be modelled and optimised while maintaining a feasible enough design is a challenge one of the participants cited. The human aspect of design was also brought up by the participants by referencing the necessity for designer expertise in more complex designs. One of the participants also talked about aesthetic constraints. This is a constraint that cannot be modelled and relies solely on the experience and preferences of the designer. Moreover, it should also serve as a reminder that any engineering design tool developed is not meant to cover every single aspect of the design process. The expectations of engineers needs to be set by researchers who develop these tools to determine what constitutes “state-of-the-art” with respect to this technology and what the limitations are of these techniques. Analogously, EAs are not the Swiss Army Knife of design tools.

6.4. Algorithm developers should include visualisation as a part of their development process

Engaging with designers and building their trust in EAs can be done with the help of visualisations. A key aspect of building trust is to develop a level of understanding of the algorithms. Visualisations allow engineers to understand the behaviour of these black-box algorithms. A simple manner in demonstrating the performance of an algorithm is to visualise the resulting solution after each generation. This is especially useful when comparing two or more algorithms. If the algorithms themselves are considered as tools, then this would give designers an easy manner to comprehend which algorithm is better for their application. For some designers, it would also be beneficial to see the different solutions that a EA can produce. This can be especially useful for designers who also need to consider aesthetic constraints. It also leads to another consideration in the development of EAs.

7. Evaluation of our approach

As recommended by Castleberry and Nolen (2018), personal bias is addressed. Every effort has been made to mitigate the effects of bias when analysing, presenting, and discussing the data. with regard to pre-survey bias, the design of the study along with the questions were reviewed by the authors and amended to keep the study as objective as possible. As this survey was done entirely online, interview bias was not present. During the entire duration of the survey being available to respondents, the answers were not gleaned to prevent any bias during the analysis stage. Of the 23 respondents that took part in this study, 7 could definitively be attributed as a professional connection to one of the authors (through answers provided to the question “How did you hear about this survey?”) which introduces some level of bias. This was minimised by not asking respondents for any identifiable and traceable details. Moreover, not a single answer contained any words or phrases which could have been used to identify a participant.

Under a restriction-free scenario, participants for this study would have been interviewed in-person; due to COVID-19, this could not be done. This did mean that the participants could not be prompted further when discussing their answers. As an example, some of the answers could not be coded due to their brevity so further detail from respondents would have been beneficial. In the future, video conferencing could be used instead of face-to-face interviews.

The number of participants recruited for this study is in line with other studies which used thematic analysis to analyse qualitative data (Huerta et al., ; Bradford et al., 2019; Fielden et al., 2011; Salleh et al., 2017; Schinke et al., 2013). More participants in this study might have led to the generation of more or different codes. However, the objective of thematic analysis is not to determine all possible themes, but to generate themes based on the available data. The subjectivity of the size of a dataset is addressed by Braun and Clarke (2019b).

8. Summary

A total of 23 participants were recruited through digital means to answer a survey. Thematic analysis was performed on 6 open-ended questions, which yielded 3 major themes: human, technology, product. With regard to their current design processes, respondents noted that optimisation takes a significant portion of their time. Respondents were generally aware of the source of inspiration from which evolutionary algorithms take their names, but lacked an in-depth knowledge of the subject. A majority of participants stated that they would implement these algorithms in their design process, although some still held reservations about their implementation.

With respect to the two research questions posed, participants have demonstrated through their answers that there is a general distrust towards EA-based design, yet there is also an acknowledgement among participants that these algorithms could benefit the engineering design process. Trust in these algorithms was one of the barriers to adoption. Cost, manufacturability, and knowledge limitations were also mentioned as hurdles in the design engineering process.

Practitioners of EA tools want to understand how the algorithm made its decision. Making tools for designers that are intuitive and whose functionality can be accessed easily by engineers is paramount; future EA interfaces should maximise engagement with designers. A clear communication channel between researchers and their end users can increase the adoption rate. Trust is an important factor for designers. By allowing engineers to be in control of the process, engineers will be able to build a level of trust with the design tools. Likewise, a human-centred approach will demonstrate to designers that these tools are not there to replace them.

Disclosure statement

The authors confirm that no financial interest or benefit has arisen from the direct application of this research.

Data availability statement

The data that support the findings of this study are openly available in Mendeley at http://doi.org/10.17632/y83cp6tgmm.1 (Vincalek, 2021).

Additional information

Funding

This work was supported by EPSRC under Grant Engineering and Physical Sciences Research Council EPSRC:EP/S021892/1.

Notes on contributors

Jakub Vincalek

Jakub Vincalek is a PhD researcher and is a candidate on the EPSRC CDT in Enhancing Human Interactions and Collaborations with Data and Intelligence Driven Systems. Jakub has Master’s degrees in Aerospace Engineering and Computer Science, and has previously worked in the space and defence industry.

Sean Walton

Dr Sean Walton is a Computer Science Lecturer at Swansea University, Sêr Cymru II Fellow, and Founding Director of Pill Bug Interactive (https://www.pillbug.zone/), a BAFTA Cymru nominated games development studio. Sean’s research background is primarily in using evolutionary optimisation techniques in engineering. Since joining the Computational Foundry at Swansea he has expanded his research interests into educational technology and the procedural generation of video game content.

Ben Evans

Dr Ben Evans is an Associate Professor in Aerospace Engineering and sits on the design team for the Bloodhound Land Speed Record project with responsibilities for aerodynamic modelling. Dr Evans’s research interests range from computational shape optimisation and high speed aerodynamic modelling to molecular gas dynamics simulation and has worked with companies including Rolls-Royce, Airbus and Reaction Engines. He has interests in engineering public engagement and engineering education.