Modelling a hormone-inspired controller for individual- and multi-modular robotic systems

Abstract

For all living organisms, the ability to regulate internal homeostasis is a crucial feature. This ability to control variables around a set point is found frequently in the physiological networks of single cells and of higher organisms. Also, nutrient allocation and task selection in social insect colonies can be interpreted as homeostatic processes of a super-organism. And finally, behaviour can also represent such a control scheme. We show how a simple model of hormone regulation, inspired by simple biological organisms, can be used as a novel method to control the behaviour of autonomous robots. We demonstrate the formulation of such an artificial homeostatic hormone system (AHHS) by a set of linked difference equations and explain how the homeostatic control of behaviour is achieved by homeostatic control of the internal ‘hormonal’ state of the robot. The first task that we used to check the quality of our AHHS controllers was a very simple one, which is often a core functionality in controller programmes that are used in autonomous robots: obstacle avoidance. We demonstrate two implementations of such an AHHS controller that performs this task in differing levels of quality. Both controllers use the concept of homeostatic control of internal variables (hormones) and they extend this concept to also include the outside world of the robots into the controlling feedback loops: As they try to regulate internal hormone levels, they are forced to keep a homeostatic control of sensor values in a way that the desired goal ‘obstacle avoidance’ is achieved. Thus, the created behaviour is also a manifestation of the acts of homeostatic control. The controllers were evaluated using a stock-and-flow model that allowed sensitivity analysis and stability tests. Afterwards, we have also tested both controllers in a multi-agent simulation tool, which allowed us to predict the robots' behaviours in various habitats and group sizes. Finally, we demonstrate how this novel AHHS controller is suitable to control a multi-cellular robotic organism in an evolutionary robotics approach, which is used for self-programming in a gait-learning task. These examples shown in this article represent the first step in our research towards autonomous aggregation and coordination of robots to higher-level modular robotic organisms that consist of several joined autonomous robotic units. Finally, we plan to achieve such aggregation patterns and to control complex-shaped robotic organisms using AHHS controllers, as they are described here.

Keywords:

• modular robotics
• stock-and-flow modelling
• hormone controller
• cybernetics
• homeostasis
• system dynamics

1. Introduction

In recent years, bio-inspired robotics frequently applied bio-inspired mechanisms to improve the group performance of physically embodied agents (robots). Starting from controlling the behaviour of a single robotic unit, the next step in complexity is often the control of groups or swarms of robots. In our interpretation, multi-modular robotics, which are physically coupled and interacting autonomous modules, are also not different from a self-organizing decentralized controlled swarm. Thus, we create a robotic controller, which inherits the principles of self-organizing, self-regulating and swarm-intelligent systems, as they are described in [1–4].

Such principles have been applied to decentralized-controlled robotic systems several times before: For the swarm-robotic project I-SWARM [5–7], we developed algorithms that allow a swarm of autonomous robots to aggregate at a target without a central unit of control. Some of these swarm-robotic algorithms are based on bio-inspired communication principles among swarm members. For example, the trophallaxis-inspired control algorithm [8–10] emulates the flow of nectar within a honeybee colony by allowing swarm robots to share their individual loads of virtual nectar among each other (i.e. nearest neighbour communication of integer numbers). Such virtual nectar amounts are shifted from one robot to neighbouring robots. Robots consume fractions of the nectar as they move and they can replenish their nectar loads at specific target sites, this way establishing a gradient of individual nectar loads within the swarm, always pointing uphill to the target areas. Robots can query the nectar loads of their local neighbours and decide on their motion direction based on this locally available information. For example, they can form trails that head towards the target zones.

Although the information needed by individual robots is available locally, the diffusion, addition and consumption of nectar in the swarm provides a sort of collective and spatially resolved process of computation that finally enforces the swarm to converge to the desired collective foraging pattern. We showed that this system is stable and adaptive by means of artificial evolution [11–14] and adaptive by runtime adaptation on the swarm and on the individual level, as it was demonstrated for the trophallaxis-inspired algorithm [9,10,15,16].

In the projects SYMBRION [17] and REPLICATOR [18], we have the aim to create a swarm of small autonomous robots able to generate autonomously moving higher-level robotic organisms by joining together many individual robots to one structure (see Figure 1). The process how these robotic modules navigate throughout the environment and how they interact (communicate) with each other will not be pre-programmed. In contrast, at the end it will be a self-organized process shaped by artificial evolution. Our robotic modules will reside in three behavioural states in our SYMBRION and REPLICATOR systems: They will move independently (alone) as a member of a group (swarm) or as an organism (several coupled modules, physically connected). We claim that such a robotic organism can be seen again as a swarm, with the only difference being that units are more tightly coupled as they are physically connected. As a result of this coupling, neighbours stay close to each other longer as it might be the case in a usual ‘swarm’. However, we think that an appropriate adaptation of swarm algorithms can produce algorithms that are also able to control an organism in a decentralized and self-organized way.

Figure 1. (a) A swarm of autonomous JASMINE robots [19], partially aggregated to a multi-modular robot organism. The artificial homeostatic hormone system (AHHS) controller presented here aims to control the swarm of individually driving robot units. (b) Example configuration of several joined robotic units that form a higher-level organism. For such systems, we use a network of joint AHHS controllers to regulate internal homeostasis and motion of such aggregated robotic organisms. Photos courtesy of Serge Kernbach, University of Stuttgart, Germany.

1.1. Modelling aspects

In the research presented here, modelling techniques were used twofold:

First, we develop a very simple model that depicts fluid-carried hormone-based communication that can achieve self-regulation (homeostasis) and self-organization within several compartments that communicate by diffusion processes. This model becomes the ‘core’ of a robotic controller, as it is ‘fed’ with sensor inputs and as it ‘spits out’ motor power values. The main ‘intelligence’ of the system resides within this physiological model, mainly determined by the model structure, that is how sensors, hormones and actuators interfere with each other. Thus, this structure is crucial for generating a well-functioning robotic controller. Therefore, the model is analysed by the tools offered in ‘system dynamics’ [20]: analysis of feedback loops and sensitivity analysis.

On the other hand, not only the model itself has to be analysed. As this physiological model is used to control the behaviour of the robot, one has to also predict/observe these behaviours. We used several different levels of prediction here: After having analysed the proposed physiological models in mathematical terms and in stock-and-flow models, we implement the model into a simple, almost physics-less multi-agents simulation [10], into a physics-enhanced robot simulator for multi-modular robotics [21] and finally also into a real robot [22]. The above-mentioned models and simulators are already described in other publications, thus we use them only for showing a ‘proof-of-concept’ here and do not analyse them in detail.

We aim for a control system that is able to govern the behaviour in all three behavioural states, which allows similar potential of collective computation as the trophallaxis algorithm described above. From our point of view, a robotic organism is nothing more than a swarm, only that the robotics units are coupled strictly by physical joints. We decided to use biological hormone systems and cell-signalling cascades as the principal source of inspiration. This system has many similarities to the trophallaxis algorithm: It mimics the flow of a fluid substance among swarm members, this way distributing information within the swarm. On the one hand, amounts of this virtual fluid can be locally increased based on environmental conditions, for example, when a robot has found the target spot. On the other hand, this virtual fluid decays over time, removing outdated information from the system.

However, the new control algorithm also has significant differences with the trophallaxis-inspired algorithm: The controller usually operates with a whole set of different virtual fluids. A set of rules describe how these virtual fluids affect the robots' actuators, this way allowing a much greater variety of behaviours. Substances (hormones) affect each other, this way allowing a whole chemical reaction network to be emulated within each single robot. The rule sets are coded in a table that we call artificial genome [23,24], allowing a genetic algorithm [25] to adapt the behaviour of the robots to specific engineering goals, to specific environmental conditions and/or to specific organism shapes.

2. The physiological model that controls the robot

In the following, we describe here how a novel bio-inspired robot controller allows the robot to develop reaction patterns that result from the received sensory input. In our approach presented here, the robot is driven by an abstract top-down model of specific aspects of the physiology of biological organisms: This model depicts the physiological processes found in living organisms (hormones) or within single cells (second messengers). In recent years, hormone-inspired approaches have been suggested for robot controllers for multiple purposes: navigation of single robots [26–30], coordination of multi-modular robots [31] or sometimes also called pheromone-based [32] or trophallaxis-based controllers, for the coordination of robot swarms [10].

We claim that hormone controllers mimicking the dynamics of fluid concentrations have three main advantages: First, they allow internal homeostasis resulting from interwoven positive and negative feedback loops; second, when equilibrium is disturbed by a sudden event, they automatically converge to a new homeostatic set point; and third, evolutionary algorithms operate on a smooth-fitness landscape that is advantageous for numerical optimization. Our suggested novel algorithm simulates such flowing hormones and uses an internal compartmentalization of the robot by also incorporating a diffusion process of virtual hormones. Our studies on real robotic hardware [33] and on robot simulators [23,34,35] showed that such a bio-inspired artificial homeostatic hormone system (AHHS) is well suited to navigate robots and to coordinate (and aggregate) robot swarms to specific (body-) shapes. In this article we demonstrate the basic abilities of such an AHHS with a simple mathematical representation, which can be either a set of linked ordinary differential equations or a set of difference equations that fit well to the stepwise execution of robot programmes.

The basic idea of our AHHS-based robot controller is that sensory input to the robot triggers the excretion of specific hormones, which are internally represented by numeric variables. The virtual internal space of the robot can consist of several compartments, thus an array (or matrix) is internally used to hold the hormone levels for all compartments.

Each hormone has a base production rate and a fixed decay rate. These two processes of production and decay have led already to a hormone-specific equilibrium that is approached by the system. These homeostatic set points can be shifted by additional hormone input that is triggered by sensor inputs. Sensors only affect the hormone secretion in the compartment they are associated with, but all hormones spread through the system by diffusing to neighbouring compartments with a fixed hormone-specific diffusion coefficient. See Figure 2 for a schematic picture that describes how the robot is assumed to be (virtually) compartmentalized and how sensors and actuators are linked to specific compartments. In Figure 2 we show how a simple robot, which is driven by two wheels (differential drive) and which has two distance sensors pointing in a lateral-frontal direction, can be represented by an AHHS controller: We assume that the internal space of the robot is best reflected by three compartments. The left one holds the left sensor and the left motor and the right compartment holds the right sensor and the right motor. The central compartment acts as a time-delaying compartment for the diffusion processes. The secreted hormones decay over time (destruction) and diffuse through the barriers. Thus, hormone levels will always be higher in the compartment at the side that is closer to the nearest obstacle, as these sensors are frequently triggering hormone secretion.

Figure 2. Schematic representation of the way our hormone controller assumes an internal compartmentalization of the robot's body. Sensors excrete hormones into one compartment, actuators get activated by local hormone concentrations and hormones diffuse between compartments.

In addition to the sensor-induced hormone excretions and diffusion, hormone levels can affect motors or other actuators. Finally, hormones can potentially affect the production or the degradation of other hormones, a process which is not used by the examples described in this article. All these processes, sensor-induced secretion, actuator-affection and hormone-to-hormone interactions, are only triggered in compartments where local hormone levels or sensor inputs exceed defined thresholds. These thresholds, together with the hormone-specific constants, are stored in a data structure called ‘genome’, which will be subject to an artificial evolutionary process.

3. Model 1: a simple obstacle-avoidance controller

Here we describe a very simple AHHS, which allows a robot to avoid obstacles. We assume a robotic hardware like it is depicted in Figure 2 and we also assume the compartmentalization that is depicted in this figure: The robot is equipped with two lateral distance sensors and with two motors. Our controller assumes three compartments in the robot: the left compartment holds the left sensor (S_l (t)) and the left motor (A_l (t)); the right compartment holds the right sensor (S_r (t)) and the right motor (A_r (t)); and the middle compartment contains no sensors and no actuators (see Figure 2). We use here just one single hormone (H ¹) that is excreted at a base rate of (α₁) and that has a fixed decay rate (γ₁). Our AHHS exploits the fact that the diffusion process together with the decay rate leads to a hormone gradient pointing to the compartment that received an above-threshold sensory input (S_x > 40), because a hormone value of s ¹ S_x is added there. S_x represents a reported sensor value on side x ∈ {l, r} and s ¹ defines how many hormone units are secreted per sensor unit. For the sake of simplicity, we assumed that both sensors report the distance to an obstacle in a linear manner (0 ≤ sensorvalue ≤ 255). The dynamics of the hormone values in all three compartments are modelled as in Equations (1)(1) (2) – (3)(3):

(1)

(2)

(3)

for a step function

(4)

The diffusion of the hormone Hⁱ at time t is modelled with the function , as described in Equation (5)(5), whereby dⁱ represents the constant diffusion coefficient of the hormone Hⁱ :

(5)

The robot's actuators are activated in correspondence to the local hormone level, as described in Equations (6)(6) and (7)(7). Please note that the robot drives straight, if both actuator inputs are of identical value and higher actuator inputs lead to a faster-moving robot. The factor a ¹ expresses how much one unit of hormone H ¹ increases the actuator inputs.

(6)

(7)

To test the conservation of mass (i.e. conservation of hormone within its lifetime between production and decay) and the influence of model parameters, we constructed a stock-and-flow representation [20] of our model using the software tool VENSIM™ [36] (see Figure 3). This allowed us to investigate the emerging hormone levels under defined regimes of sensor input data and to calculate how motor output will in turn be affected by varying hormone levels.

Figure 3. Stock-and-flow diagram of our simple AHHS controller. Boxes indicate ‘stocks’, which can hold (and accumulate) quantities. Double arrows indicate flows, through which quantities can shift from one stock to another. The cloud-like symbols indicate sources and sinks, through which quantities can enter the system or leave the system. Thin arrows indicate causal relationships in the manner of ‘A affects B’.

Using the stock-and-flow model, we investigated the hormone levels that emerge in a modelled robot, when no obstacles disturbed the robot. We investigated the equilibria that result from a sensor reporting a nearby obstacle. To do this, we simulated a sensor input of S_l (t) = 70 from time step t = 500 to time step t = 750, and a sensor input of S_r (t) = 70 from time step t = 1250 to time step t = 1500. In all other times, no sensor input was simulated (S_l (t) = S_r (t) = 0). These two sensor input pulses were aimed to mimic simple sensor data as it can result from sensing an obstacle on one side of the robot.

The altered equilibria in times of sensor inputs led to modulated motor inputs, which steer the robot away from the obstacle. After some time the path of the robot is clear again and the controller approaches its initial state (see Figure 4(a) and (b)). Thus, the motor-sensor loop represents an important external feedback loop, which is part of the homeostatic control loop that regulates the corresponding hormone. The only way for the robot to reach the equilibrium point again is to exert a behaviour that in turn leads to a new sensor input that allows the hormone level to converge to the homeostatic set point again. Table 1 shows the values we used for our stability analysis and sensitivity analysis of our stock-and-flow model. To test whether our AHHS controller will act stable if only noisy sensor data are available, we added a uniformly distributed noise of ±50% to all sensor data in our stock-and-flow model and depicted the resulting hormone levels and actuator responses. As can be seen in Figure 4(c) and (d), even high levels of sensor noise do not lead to significantly disturbed motor output. This is of course only true for the applied noise that operates on a different timescale than the hormone dynamics. Temporally correlated noise or noise of similar timescales as the hormone dynamics would certainly disturb the motor output. However, the concept of AHHS is to have such different timescales exactly to reach a desirable level of robustness.

Table 1. Parameters used in our stock-and-flow model

Parameter/variable	Value	Units
α1	20	[Volume unit]/[time step]
γ1	0.1	1/[time step]
	0	[Volume unit]
a ^1	0.05	[Motor unit]/[volume unit]
s ^1	1	[Volume units]/([sensor unit] × [time step])
d ^1	0.25	1/[time step]

Figure 4. (a) Dynamics of sensor values and resulting motor inputs (no sensor noise). (b) Corresponding dynamics of hormone concentrations in the three compartments (no sensor noise). (c) Dynamics of sensor values and resulting motor inputs (±50% sensor noise). (d) Corresponding dynamics of hormone concentrations in the three compartments (±50% sensor noise).

After we found that the applied sensor noise cannot disturb the system significantly, we investigated the controller's sensitivity to the major constant parameters, namely, decay rate of the hormone and its diffusion rate. The sensitivity analysis of the decay rate γ₁ is shown in Figure 5. The big shaded areas in this figure indicate big differences in the resulting motor control values for varied decay rates and, hence, a high sensitivity (see figure caption for details of colour-coded shades). The upper two sub-figures of Figure 5 show that the parameter decay rate (γ₁) affects the motor outputs significantly, as a low decay rate leads to a high homeostatic set point for the corresponding hormone H ¹.

Figure 5. Sensitivity analysis of the decay rate (γ₁). Two hundred independent simulation runs sampled γ₁ between 0.001 and 0.05: (a) resulting left motor outputs; (b) resulting right motor outputs; (c) resulting ratio between left and right motor output (0.5 means equal output, thus straight driving); (d) resulting ratio between right and left motor output. The shaded areas show the range of results that were found in varying the focal parameter. The wider the shaded area is, the higher is the sensitivity of the model to the tested parameter. Quartiles and percentiles are colour coded: Within the inner white band 50% of all results are located. Adding the two bright grey bands extends the span to 75% of all predicted results. The two black bands additionally indicate 95% of results and the two outermost darker grey areas indicate the span of all (100%) predicted results.

But what is more important in the case of a robot that moves with a differential drive is the ratio between the two motor outputs, as the more difference there is, the stronger the robot will turn in a curve. We calculated the ratio between the two motor outputs using Equations (8)(8) and (9)(9). Using these equations, we performed a sensitivity analysis again; using the same parameter range as in the analysis mentioned above:

(8)

(9)

The lower two sub-figures of Figure 5 show that this important ratio of motor outputs is not affected at the same order of magnitude than it looks like when one considers each single motor output alone. Thus, the model shows that lowered decay rates of motor-driving hormones will lead mostly to faster driving robots. The controllers ability to turn the robots is not very sensitive to the value of the parameter γ₁.

After investigating the controllers sensitivity to the parameter γ₁, we performed another sensitivity analysis on the diffusion coefficient of the H ¹ hormone, which is d ¹. Figure 6 shows a significantly lower sensitivity of the motor activation (movement speed) but an increased sensitivity of the motor-speed-difference (steering behaviour) to the diffusion coefficient, compared with the sensitivities observed by scaling the decay rates.

Figure 6. Sensitivity analysis of the diffusion coefficient d ¹. Two hundred independent simulation runs sampled d ¹ between 0.01 and 0.5: (a) resulting left motor outputs; (b) resulting right motor outputs; (c) resulting ratio between left and right motor output (0.5 means equal motor output, thus straight driving); (d) resulting ratio between right and left motor output.The shaded areas show the range of results that was found in varying the focal parameter. The wider the shaded area is, the higher is the sensitivity of the model to the tested parameter. Quartiles and percentiles are colour-coded: Within the inner white band 50% of all results are located. Adding the two bright grey bands extends the span to 75% of all predicted results. The two black bands additionally indicate 95% of results and the two outermost darker grey areas indicate the span of all (100%) predicted results.

The analysis of the stock-and-flow model showed interesting findings about the controller's stability and sensitivity. In a test run with a multi-agent simulation of an autonomous robot, we wanted to see the controller act in a structured habitat. The simulator used a model of a robot, as it is depicted in Figure 2 and a controller, as it is described in the section above. In the simulations shown here, a noise of ±20% was applied to all sensor inputs. Figure 7 shows that the robot was able to perform the obstacle avoidance task in a heterogeneous environment, in which various walls and columns blocked the robot's way. Although the controller performed quite well, several collisions with obstacles occurred. Nevertheless, most obstacles were avoided successfully and the robot was always able to free itself after a collision.

Figure 7. The described simple AHHS controller navigated the simulated robot safely through a highly structured environment. Collisions with obstacles occurred rarely. Black boxes indicate obstacles and walls. The grey rectangular box indicates the robot. The solid line shows the robot's trajectory. Crosses indicate collisions with obstacles. Two cones indicate the area covered by the two sensors. For settings of the controller, see text.

4. Model 2: using two hormones for a better obstacle avoidance

To better exploit the potential of the differential drive, we extended our AHHS controller in the following way: We introduced now a second hormone, whereby the hormone H ¹ is excreted only according to the left sensor input into the left compartment, thus this hormone now has a dedicated meaning: ‘There is an obstacle to the left.’ The hormone H ² is excreted only according to the right sensor input into the right compartment, meaning: ‘There is an obstacle to the right.’ H ¹ activates now only the left motor and H ² activates now only the right motor. In addition to that, the hormones decrease the activation of the contralateral side: Those hormone volumes of H ¹ that diffused to the right compartment now deactivate the right motor (linear) proportionally and the diffused volumes of H ² deactivate the left motor. Deactivation can even lead to backwards driving wheels if the deactivation is bigger then the activation. This way it becomes possible that both wheels are driven in opposite directions, what leads to a robot that spins on place without travelling. We assumed that this additional behaviour can decrease the likelihood of obstacle collisions significantly, compared with the first controller, which allowed no change in direction without forward movement. Figure 8 shows a stock-and-flow representation of the enhanced AHHS model, which was then analysed in the software tool VENSIM™ [36].

Figure 8. Stock-and-flow diagram of our enhanced AHHS controller. The model requires now six different stocks, because two hormones have to be modelled in the three compartments. There is a flow neither from the upper stock to the lower stocks, nor in the other direction. This is because hormones are never converted into other hormones in our AHHS. But, as described in the text, both hormones can modulate actuators in different directions simultaneously, as can be seen by the leftmost and by the rightmost thin arrow. Boxes indicate ‘stocks’, which can hold (and accumulate) quantities. Double arrows indicate flows, through which quantities can shift from one stock to another. The cloud-like symbols indicate sources and sinks, through which quantities can enter the system or leave the system. Thin arrows indicate causal relationships in the manner of ‘A affects B’.

The new model is now formulated as seen in Equations (10)(10) (11) (12) (13) (14)–(15)(15); which are mainly based on Equations (1)(1) (2)–(3)(3), except that one sensory input was removed for each hormone. The modelling of the diffusion process remained unchanged, according to Equation (5)(5).

(10)

(11)

(12)

with a step function θ as defined in Equation (4)(4).

(13)

(14)

(15)

again with a step function θ as defined in Equation (4)(4).

The activation of motors was changed significantly compared with the first controller model (see Equations (6)(6)–(7)(7)).

(16)

(17)

We investigated the sensitivity of the enhanced AHHS controller to the two most important parameters (Table 2): As can be seen in Figures 9 and 10, the improved controller showed to be less sensitive to these parameter values.

Table 2. Parameters used for analysing our enhanced AHHS controller

Parameter/variable	Value	Units
α1	20	[Volume unit]/[time step]
α1	20	[Volume unit]/[time step]
γ1	0.1	1/[time step]
γ2	0.1	1/[time step]
	0	[Volume unit]
	0	[Volume unit]
a ^1 = a ^2	0.05	[Motor unit]/[volume unit]
s ^1 = s ^2	1	[Volume unit]/([sensor unit]×[time step])
d ^1 = d ^2	0.075	1/[time step]
h ^1 = h ^2	0.075	[Motor unit]/[volume unit]

Figure 10. Sensitivity analysis of the diffusion coefficients d ¹ and d ². Two hundred independent simulation runs sampled d ¹ and d ² between 0.01 and 0.5 (both coefficients were always equal in each run): (a) resulting left motor outputs; (b) resulting right motor outputs; (c) resulting ratio between left and right motor output (0.5 means equal motor output, thus straight driving); (d) resulting ratio between right and left motor output. The shaded areas show the range of results that was found in varying the focal parameter. The wider the shaded area is, the higher is the sensitivity of the model to the tested parameter. Quartiles and percentiles are colour-coded: Within the inner white band 50% of all results are located. Adding the two bright grey bands extends the span to 75% of all predicted results. The two black bands additionally indicate 95% of results and the two outermost darker grey areas indicate the span of all (100%) predicted results.

Figure 9. Sensitivity analysis of the decay rates γ₁ and γ₂. Two hundred independent simulation runs sampled both decay rates between 0.001 and 0.05 (both rates were always equal in each run): (a) resulting left motor outputs; (b) resulting right motor outputs; (c) resulting ratio between left and right motor output (0.5 means equal output, thus straight driving); (d) resulting ratio between right and left motor output. The shaded areas show the range of results that were found in varying the focal parameter. The wider the shaded area is, the higher is the sensitivity of the model to the tested parameter. Quartiles and percentiles are colour-coded: Within the inner white band 50% of all results are located. Adding the two bright grey bands extends the span to 75% of all predicted results. The two black bands additionally indicate 95% of results and the two outermost darker grey areas indicate the span of all (100%) predicted results.

We implemented the enhanced AHHS controller also in a multi-agent simulation and observed a simulated robot driving for 10,000 time steps through a highly structured habitat: The arena was bound by walls and several columns blocked the robot's way. As expected, the robot managed to navigate in this habitat successfully (see Figure 11(a)). In addition, the controller was tested on robotic hardware and showed similar behaviour as the simulated robot [33].

Figure 11. The enhanced AHHS controller navigated the robot through a highly structured environment without colliding with an obstacle. (a) Black boxes indicate obstacles and walls. The grey rectangular box indicates the robot. The solid line shows the robot's trajectory. Two cones indicate the area covered by the two sensors. For settings of the controller, see text. (b) Stroboscopic image of a moving e-puck [22] robot controlled by an AHHS controller and obstacles [33].

One important design objective of our AHHS system was its applicability for multi-modular robotics. We aim for a controlling system that is able to control a complex robotic organism and that is able to adapt this control principle by means of evolutionary computation. To benchmark the AHHS system concerning this design objective, our current work is based on the simulations of modular robots (see Figure 12). The AHHS was implemented within the simulation environment Symbricator3D [21].

Figure 12. Three-module gait learning in the simulation.

Using this system, we started with randomized genomes and a preset body formation and selected organisms for reproduction based on the covered distance after 300 time steps. Obviously, this benchmark tests the evolvability of robotic organisms with AHHS in a gait-learning task, as those controllers, which evolved a sort of gait behaviour, are predicted to be preferentially selected for reproduction.

5. Multi-modular robotics: gait learning

Modular robotics is a field with many challenges. In contrast to classical robotics, the robot organisms (several connected robot modules) are able to autonomously change their bodies substantially. On the one hand, this allows for immense adaptations. On the other hand, it bears a challenge to the design of the robot controller. The robot modules should be controlled locally not to diminish the possibilities and robustness of body shapes. New controller design paradigms are necessary to manage these challenges.

The AHHS controller combined with evolutionary algorithms has a high degree of applicability to this domain because the natural division of a multi-modular robot organism into modules can be used for compartmentalizations. In addition it turned out that behaviours generated by AHHS can be well optimized with the genetic algorithm. In the following, we present the general approach in a study case and give an example of the typical evolved behaviour. Other in-depth studies of this scenario were reported in [34,35].

We apply the AHHS approach to a gait-learning scenario. Gait learning in legged robotics is a commonly studied task in evolutionary robotics [37]. However, here we investigate gait learning in modular robotics. Each module consists of one hinge and we connect three modules (see Figure 13), thus our focal system has three degrees of freedom. These three hinges are controlled in a decentralized way although the modules have a low-level communication channel by means of diffusing hormones.

Figure 13. Robot module configuration used, initial position and example of angled hinge. (a) Initial horizontal position and indicated possible rotation of the hinge. (b) Position after middle module angled its hinge to the maximal position.

In contrast to the standard tasks of gait learning and collision avoidance, the challenge of gait learning in modular robotics is more complex. The resulting gait is emergent due to decentralized and cooperative control of the actuators.

The same controller is executed in each module. Therefore, the gait-learning task includes several sub-tasks. The organism has to break the symmetry between head and tail, synchronize through collective cooperation and start moving into a common direction. We use AHHS with 60 rules and 15 hormones and a small population size of 20 AHHS controllers.

The progress of the best AHHS controllers from n = 6 runs of the artificial evolution lasting 180 generations each is shown in Figure 14. Starting with a randomly generated population with a median fitness (i.e. covered distance) of less than 0.02 a median fitness of above 0.5 is finally reached.

Figure 14. Progress of the best individuals per generation of the n = 6 evolutionary runs. The median fitness of the population is increasing, finally converging to approximate. Fifty percent of the maximum fitness, which is achieved by the best solutions after approximately 10 generations occasionally and after 30 generations on a regular basis.

In Figure 15 we show the hormone, actuator control value and sensor dynamics of a typical, evolved behaviour. Obviously, the robot has to perform a rhythmic behaviour to maximize the fitness function, that is, to cover a distance. Indeed, the hormone is oscillating (Figure 15a). It controls the hinge, which is in turn showing oscillations (Figure 15b). Each peak corresponds to a bending of the hinge, which results in a step of the robot organism, which, in turn, increases the covered distance. Figure 15(c) shows the corresponding oscillations in a proximity sensor that is sensing the differences in the distance to the ground induced by motions of the robot module.

Figure 15. Three-module gait-learning analysis of the evolved behaviour: (a) Dynamics of a relevant hormone, (b) the resulting control value of the actuator and (c) the corresponding oscillations in a proximity sensor of one module within the organism.

6. Comparison to the state of the art in bio-inspired robot control

Several studies also suggest hormone-inspired control for autonomous robots: The studies of [31] suggest a hormone-based system in which, in contrast to pre-programmed fixed-topology gait tables, a process of nearest-neighbour communication propagates messages through a multi-modular organism. This generates a decentralized system of motion control, as each module is informed by its neighbouring module about the current action of the neighbour, and can thus decide about its own behaviour. In this approach, messages drawn from a discrete set of messages are travelling in a wave-like manner through the organism. Although this spread of messages is a decentralized broadcast-based system of communication, hormones (and pheromones) show different characteristics. For example, hormones do not code for specific events (like messages), it is the set of concentrations (floating point values!) that brings a specific organ or cell to a specific state, which then affects its future behaviour.

In our AHHS, hormones are modelled like chemical substances that flow through a virtual (robotic) organism. For example, in our approach the conservation of mass is a feature that was important for us. As can be seen by the stock-and-flow diagram in Figure 3, the conservation of mass is guaranteed in our AHHS models. Thus, we conclude that the approach of [31] is closely related to the CPG-based (central pattern generators) approaches of gait development, as also their sub-modules (oscillators) affect their neighbours' behaviours based on linkage (communication) between modules. See, for example, [38] an evolutionary computation study of such bio-inspired systems, which originally were inspired by neural circuits in swimming or crawling vertebrates, which merely resemble segmentally coupled time-delayed neural oscillators. In contrast to such systems, our hormone-based approach (AHHS) is strongly based on a computer simulation of the flow of fluids with solved signal chemicals diluted in these fluids. This is found in higher organisms and in singular cells. Similar processes are also found in the flow of liquid food inside of insect colonies (trophallaxis) and also in slime mould aggregations, where travelling waves of a chemical signals coordinate the aggregation and motion of a ‘swarm’ of individual amoebas. In previous studies, we used both sources of inspiration to coordinate swarms of autonomous robots [10,39], thus the AHHS controller presented here stands in a line with previous research tracks. The most important improvements of the state of the art by the AHHS system is its applicability to multi-modular robotics (as demonstrated here) and its evolvability. Without evolutionary computation or other forms of self-programming and self-adaptation, a truly multi-modular system like SYMBRION and REPLICATOR will be infeasible, as it is almost impossible to hand-code the individual controllers for each robotic module in an organism consisting of ten to a hundred modules, allowing hundreds of thousands of different body morphologies.

Another variant of hormone-inspired control was suggested by [26,28]. In contrast to those models, our hormone controllers regulate the whole behaviour of the robot alone, whereas the other approaches use the hormones to modulate an underlying artificial neural network. Such a hormonal control scheme was also suggested in another study [29], where a hormone model was used to express ‘moods’ of the robot and triggering/affecting different pre-programmed and hand-coded behavioural controllers. In an approach that resembles the trophallaxis-inspired algorithm [10] in a swarm-robotic context, the study of [40] describes self-organized coordination of robot swarms in an agent-based simulation. It was used for geometric pattern formation (and ‘pattern healing’) in simulated robots and not for controlling multi-modular robotic organisms. A distance-gradient-based method for pattern formation in simulated robot swarms was investigated in [41,42], which has some overlap with the methods of the ‘hop-count’-based algorithm that was analysed in [43], as robots communicate with local neighbours and try to figure out their relative position in the aggregated pattern, which corresponds to multiple distance (hop-count) assessments in a chain of communicated messages. For regenerative patterning, a ‘robotic stem cell’ is described in [44], which is an encapsulated unit of functionality (an own controller) that is able to support self-repair in a robotic organism. It operates based on inter-robotic communication, which is based on the hormone-inspired communication principles described in [40], thus, as discussed above, it significantly differs from our AHHS controllers, which emulate the concentration changes of flowing chemical concentrations.

Thus, to our knowledge, our AHHS controller model, that we present here, is the first robot controller that simulates endocrine processes in the way it is modelled in the previous sections and that regulates the behaviour of the robots without any other controlling structure or pre-programmed functionality.

7. Discussion

In this article we introduced a novel and scalable robotic controller, which is inspired by chemical signalling in living organisms (hormones and cell signals). We analysed basic properties of two implementations of this novel controlling paradigm and analysed the sensitivity of such systems. The controller is described formally in this article, allowing the robotic community to generate its own implementations and allowing for independent analysis and verification. To achieve our numerical analysis, we used two different modelling approaches: For a proof-of concept, which is a demonstration that the controller is actually able to steer an autonomous robot, we used a self-made multi-agent simulator based on NetLogo. The simulation is a derivative of a swarm-robotic simulator called LaRoSim [10], which we used in multiple studies before and which was refined through several years in the swarm robotic project I-SWARM [6]. For analysing the basic properties of the control paradigm, we used a top-down modelling approach, which is based on VENSIM™ [36]. We constructed stock-and-flow models of the analysed robotic controllers, following the modelling school of system dynamics [19]. Using such models, we can study the ‘flow of hormones’ in the controller, which is an essential component in steering the robot, by explicitly ensuring important features like, for example, the conservation of mass in the system. Our results indicate that several parameter sets of the controlling system lead to qualitatively different obstacle avoidance behaviours of varying quality (efficiency).

In addition to freely driving robots, we analysed the controller also in an experiment using a robotic organism, consisting of three physically connected robotic modules, each of them operating with an autonomous instance of an AHHS controller. All three modules shared the same parameterization of their individual AHHS, thus, for achieving an effective gait performance, the controllers rule-sets have to result in a position-depending emergence of specific behaviours. For efficient gate walking, the modules have to contact their hinges in a specific time pattern, thus efficient gait-producing controllers have to synchronize to a specific spatio-temporal rhythm.

Our main focus is evolutionary robotics [45,46]: Efficient rule-sets and parameter-sets are not coded by hand, as we did it for the first two simple collision-avoidance controllers in this article. Instead, these parameterizations will emerge from a process of evolutionary computation, as we demonstrated with the gait-learning experiment.

An AHHS controller provides multiple advantages for such an evolutionary computation approach: It is easy to evolve AHHS, because only simple checks have to be performed after mutation and cross-overs to ensure that mutated/mixed AHHS controllers are valid. AHHS controllers provide smooth search landscapes for evolutionary computation and novel mutation operators can be tested: For example, hormones can be easily switched on or off in the artificial genome. Behaviour of the robot is actually generated by the dynamics of hormone values. These hormone values are significantly smoothed by the diffusion processes that we implemented. This ensures non-spontaneous robot actuation and the favouring of robust control loops in the evolutionary process. Because of these reasons, we assume that fitness landscapes are rather smooth in an evolving AHHS controller.

Valentino Braitenberg [47] demonstrated that simple, in his case hard-wired sensory-inputs can effectively navigate a moving machine (robot) to (or away from) a pre-designed target, mainly by exploiting environmental gradients that point towards that target (light, sound, temperature). The idea of homeostatic control of animal-inspired machines was already a major core component of cybernetics, as it was described by Norbert Wiener in the 1950s [48]. It assumes communication [49,50] of sensor values to central ‘control components’ that exert positive and negative feedback via actuators back to the sensor input. In contrast to Wiener's approach, who focused mainly on predicting future positions of a mobile target and on autonomously approaching such targets, our implementation is more focused on keeping the internal status of our focal robots by forcing them to show the desired behaviours. Our AHHS manipulates the behaviour of the robots in such a way that they achieve internal homeostasis of intrinsic hormone levels by moving away from obstacles, which disturb homeostatic set-points by triggering additional hormone secretion. Nevertheless, our approach follows the idea of cybernetics and can thus be called a cybernetic regulation achieved by virtual hormones: Also cybernetics involves a sort of filtering of sensory input, as it can be seen by our hormones, which integrate past sensory input over time. Even ±50% sensor noise did not affect hormone levels significantly. An important feature of our AHHS is the steady decay of such integrated information, as it is modelled by the hormones decay rate. This allows ‘forgetting’ of outdated information, as long as it is not reinforced by new sensory input.

Another important feature of our controller is the compartmentalization of our robot's virtual inner space. These compartments allow ‘computation in space’, because sensors and actuators are linked to those compartments that correspond to their position on the robot. This can be seen by the fact that we could use just one single hormone for avoiding obstacles at the left and obstacles at the right. It was the combination of compartmentalization, decay processes and diffusion processes that allowed one single hormone to regulate navigation into both directions.

We suggested a novel concept for engineering a controller for autonomous robots. So far, we investigated a system of difference equations. By using standard numerical method, we can also investigate our model as if it were a set of differential equations. Additional individual-based simulations allowed us to test the controller in environments with increased complexity. It showed that our controller acts stable, performs the given task and can deal with high levels of sensor noise. Our further work will test the AHHS to control joined multi-modular robotic organisms in simulation and on real robotic hardware.

Acknowledgements

This work is supported by the following grants: EU-IST FET project ‘I-SWARM’, no. 507006 and the FWF research grant no. P19478-B16, EU-IST FET project ‘SYMBRION’, no. 216342, EU-ICT project ‘REPLICATOR’, no. 216240.