On Visuo-Inertial Fusion for Robot Pose Estimation Using Hierarchical Fuzzy Systems

Abstract

This article presents a novel application of a three level fuzzy hierarchical system for the fusion of visual and inertial pose estimations. The goal is to provide accurate and robust pose measurements of an indoor volant robot, which operates in real working conditions, in order to facilitate its control. The first level corrects the error of the inertial measurement unit based on the acceleration measurements. The second level fuses the measurements from the visual sensor with the ones from the first level. Finally, the output of the fuzzy hierarchical system is available at the third level; the inputs of which are the previous system's state and the one of the second level. The achieved results are compared to ground truth pose estimations, which are used to analyze the behavior of the proposed fusion system against possible changes. The system provides accurate and precise measurements, while its straightforward design allows real-time implementation.

Keywords:

• data fusion
• fuzzy systems (FS)
• hierarchical fuzzy systems (HFS)
• inertial navigation
• visual pose estimation
• visuo-inertial fusion

NOMENCLATURE

EKF	=	extended Kalman filter
HFS	=	hierarchical fuzzy system
IMU	=	inertial measurement unit
MF	=	membership function
CU	=	climber unit
SU	=	swinging unit
T k	=	translation on one axis at time k
θ k	=	angle of one axis at time k
α	=	linear acceleration
γ	=	angular acceleration
V	=	linear velocity
ω	=	angular velocity
{C}	=	camera coordinates
{I}	=	IMU coordinates
{Im}	=	image coordinates
{W}	=	world frame coordinates
xx T yy	=	transformation matrix from xx to yy

1. INTRODUCTION

One of the most common problems in robotics is to estimate the robot's pose accurately. This can be utilized in many tasks, such as navigation, localization and manipulation. Pose estimation with visual sensors is a common robotics application and still remains an open research field. The mathematical background for extracting the pose vector leans mostly on the utilization of homographies and projection geometry (Hartley and Zisserman 2004). However, recently the fusion of visual and inertial data is increasingly becoming attractive and much effort is put into several realizations of novel methods and techniques. The main thrust towards this research is that the visual and inertial sensors have complementary attributes with each other, which provide flexibility androbustness when integrated. In particular, the high frequency measurements of the inertial measurement unit (IMU) sensors combined with low frequency ones derived from the cameras, finally produce high frequency pose estimates (Rehbinder and Ghosh 2003).

Many efficient visuo-inertial systems have been reported so far and they can be classified into three main categories: colligation, correction and fusion (Xu and Li 2007). In the colligation type, two heterogeneous measurements are combined in a way that one sensor provides the pose estimates, while the other verifies them (Zuerey and Floreano 2006). In the correction category, the measurements of one sensor are considered as the desired ones, while the other sensor's readings as the estimated ones. The aim is to minimize the error between the two measurements. The third type is fusion, which takes into account both sensors’ readings. The fusion of visual and inertial sensors can be used in several robotics-oriented applications, such as to estimate the egomotion and the environmental structure (Gemeiner et al. 2007), to stabilize a humanoid binocular structure (Panerai et al. 2002), and to track in 3-D an articulated arm (Tao et al. 2007), to name a few. The Kalman filter (KF) and all its formations such as the extended Kalman filter (EKF) and the unscented Kalman filter (UKF), are some of the most common fusion techniques in terms of this specific task. The EKF has also been used for estimating the pose of map-based mobile robots (Borges and Aldon 2002), to a Kalman-based visuo-inertial calibration (Mirzaei and Roumeliotis 2008), and to track rigid objects by applying a pose estimation algorithm (Youngrock et al. 2008). The visual pose estimation applications provide accurate measurements and they have been widely utilized for this task. However, most of these approaches present a high degree of error at the final transformation matrix. The reduction of these errors optimizes the accuracy of the final pose estimation (Zhang et al. 2006). Another way to increase the efficiency of such systems is to integrate an inertial sensor. Furthermore, additional information for the pose estimation problem is derived by combining the measurements from the camera with the inertial sensor's ones (Rehbinder and Ghosh 2003; Lobo and Dias 2003). In Bonnabel and Rouchon (2009), the pose is estimated by a geometrical observer-based approach.

During the last few decades, an increase in use of fuzzy logic has been reported for many applications, such as approximation (Zhang et al. 2009; Bing et al. 2009; Lendek et al. 2009), system control (Kyriakoulis et al. 2010; Cheong and Lai 2007 de Almeida Souza et al. 2007), fuzzy classification (Jahromi and Taheri 2008; Prokhorenkov and Kachala 2008), and fuzzy clustering (Schaefer and Zhou 2009; Lee and Pedrycz 2010). The extensive use of fuzzy systems stems from their applicability to nonlinear problems, and also due to the fact that humans can straightforwardly apply their expertise to the respective design through natural language representations. In fuzzy logic, unlike conventional crisp logic, the validity of any statement is an issue of a specific degree, quantified by a membership function (MF). Thus, the inputs are fuzzified by means of MFs and then processed by fuzzy logic operators respective to the conventional logic operators, such as AND, OR, etc. This is performed for each particular rule from a set of IF-THEN ones. The inferring mechanism of a system based on fuzzy logic also depends on the fuzzy implication function, i.e., the meaning of THEN in each rule. The results of the fuzzy logic operators are aggregated to form each distinguished output, according to a specific aggregation method. Last, in order to produce a conventional output, an appropriate defuzzification function is chosen to map the fuzzy outputs to crisp ones. For a deeper study of these concepts the reader may refer to Gupta (1995). However, the reported fuzzy systems are mostly restricted to two inputs with asfew membership functions (MFs) as possible. The main motivation for this is to keep the complexity low, since it increases exponentially according to the number of inputs and the number of the MFs: O(n ^k ), where k is the system's input variables and n is the number of the MFs. This is called the curse of dimensionality (Brown et al. 2004; Rattasiri et al. 2008), which is overcome by using hierarchical fuzzy systems (HFSs) (Lee et al. 2003; Brown et al. 2004; Raju et al. 1991). The complexity of an HFS being O((n − 1) · k ²) is significantly reduced compared to this of a counterpart standard FS. In Aja-Fernández and Alberola-López (2008), a solution to the HFS decomposition problem is described by means of transition matrices, where the final system's relation input-output is equivalent to the original one. By using transition matrices the computational load of the original HFS is restricted to simple mathematical calculations. Moreover, the computational load of an HFS can also be reduced by evolving it via a probabilistic incremental program evolution (PIPE), where the fuzzy sets are also tuned via evolutionary programming (Chen et al. 2007). The multidimensional FSs can also be converted into a two-level hierarchical FS that increases the run-time efficiency ofthe system as the number of rules are reduced (Joo and Sudkamp 2009). In fact, for each input the algorithm needs O(k n ^k ) operations while in a two-leveled HFS, O(2n ^k ) are required. Thus, HFSs are very attractive, especially in applications where the number of inputs and/or the number of the MFs cannot be kept low. In such cases, they are preferable to the common FSs.

Hierarchical fuzzy structures have a wide variety of applications such as classification (Goncalves et al. 2006; Tümer et al. 2003), clustering (Yager 2000), decision modules (Lee et al. 1999), forecasting (Chang and Fan 2008), and data fusion (Su and Chen 2004). Fuzzylogic has also been used for fusing inertial navigation systems with global positioning ones (Sasiadek et al. 2000). However, the use of fuzzy systems in visuo-inertial fusion and the proposed fusion with the hierarchical fuzzy system in particular, introduce a novel aspect. In this article, we propose a novel fuzzy fusion technique, which receives as inputs the pose measurements from a camera and an IMU, respectively. The noise of the IMU is compensated from the acceleration data by fuzzy means at the first level of the HFS. The pose estimation involves a robotic platform, on the body of which a set of markers is placed. A camera observes the robotic platform from an external moving position. In order to extract the pose accurately, the markers are placed on predefined locations. An IMU mounted on the platform measures its rotations and translations. The two heterogeneous acquired measurements are consequently fused. Three cascaded fuzzy logic units (FLU) process all these data, providing as output the pose vector. As the possible inputs are more than two, we adopted an HFS architecture which eliminates the dimensionality growth. This architecture is one with a minimized rule base (HFSwMRB) (Rattasiri et al. 2008) in order to have low processing times while preserving high accuracy. The HFS is assessed at two different fusion procedures: one for the translation and one for the rotation data, respectively. The system possesses four inputs: namely, the linear/angular acceleration acquired from the IMU readings, the translation/rotation computed from the acceleration data, the one from the visual data and, finally, the previous system's state. This recursiveness corrects the large deviations from the previous measured position. Finally, a system analysis is demonstrated assessing the efficiency of the proposed method. The rest of the article is organized as follows. In the second section, the pose estimation methods are described. In the third section, the overall hierarchical fuzzy fusion process is demonstrated, and each hierarchy level is detailed in respective subsections. In section 4, the criteria for the selected architecture and the analysis of the system for the selection of its optimal parameters are presented. In the experimental section, the proposed system is assessed for an indoor working environment and against traditional fusion techniques. Concluding remarks and a summary of the aspects of the proposed system are provided in the last section.

2. POSE ESTIMATION

The system's architecture is examined and tested for the needs of the ACROBOTER project (Stepan et al. 2009). The ACROBOTER system is aimed to operate indoors and its workspace is the volume of an entire room. There are two main subsystems: the climber unit (CU) and the swinging one (SU), as depicted in Figure 1 a. The first is responsible for moving across the ceiling by climbing along an anchoring grid. The second subsystem is a volant robotic platform hanged by means of a wired rope from the CU. The SU comprises ducted fans which forces the platform to move around the room. A detailed photograph of the SU prototype implementation is shown in Figure 1 a. The colored markers used for the visual pose estimation task are placed onthe top of the SU. The external moving position where the visual sensor is placed, is the CU. The camera is the Pointgrey Grasshopper (Point Grey Research Inc. 2010), while the IMU is the MTx model bought from the Xsens company (Xsens Technologies 2010). The interfaces used are IEEE 1394 for the camera and USB 2.0 to serial outputs for the inertial unit sensor, repsectively. The visual sensor is mounted on the climber unit, placed so that its optical axis is perpendicular to the ground. The physical arrangement of the involved parts of the mechatronic systems are depicted in Figure 1 b. Both sensors are connected to respective computers, the communication and synchronization of which are realized by thread programming. Both pose estimation systems, i.e., the visual and the IMU one, perform in parallel. The visual pose estimation process involves the camera's transformation matrix computation from the mounted encoders and the volant platform's pose estimation from the visual sensor. On the other hand, as the IMU shares the same frame of reference with the robotic platform, the measurements readings are transmitted with no further transformation. The communication architecture is illustrated in Figure 2.

Figure 1 The proposed system. (a) The overall mechatronic system; close-up photographs of the climber unit (up) and swinging unit (down); and (b) detailed CAD model design. The camera is mounted on the climber unit, and on the volant platform the four coloredmarkers are placed (color figure available online).

Figure 2 The proposed system communication architecture.

2.1. Visual Pose Estimation

The visual pose estimation used is the one presented in Kyriakoulis and Gasteratos (2010). The camera's frame of reference is attached to the camera and its origin is the camera's center. Unlike other systems in which the camera and the IMU are aligned and mounted on the same object, we placed them on different moving systems. The camera is mounted on the CU and is fixated towards the ground in order to observe the operating SU beneath it. The position of the camera is given by the CU's encoders, while the SU's one is derived from the visual data. The pose vector is estimated by tracking a set of four colored markers, as depicted in Figure 1 b. It worth mentioning that the choice of having only four markers was imposed by the application and the designof the suspending unit. The markers are tracked according to their respective color histograms and their motion vector. To facilitate the function of color tracking, the previous position of the markers is also taken into consideration. In each captured image the features are identified and the extrinsic parameters are computed. There are five different frames of reference: the image {Im}, the camera {C}, the climber unit {CU}, the inertial unit on the volant platform {I}, and the swingingunit {SU}, respectively. Frame {C} is rotated by −180^○ about the x axis. In order to align {C} and {Im}, {C} is further rotated about axis z _c by −90^○, Figure 1 b. The final camera's position is calculated by equations (1), and (2).

In equation (1), the transformation ^CU T _C  ·  ^C T _Im is calculated via the calibration matrix (intrinsic parameters, Figure 1 b). The rotation angles obtained from the matrix in equation (2) are directly sent to the volant's platform computer.

The camera used is set to 30 fps in the RGB color channel. The low processing times are achieved by utilizing the optimized codes of the Open Source Computer Vision (OpenCV) library (Open Source Computer Vision 2010) for the vision tasks concerning the pose estimation, i.e., SVD, homographies, and pixel normalization. These exhibit minimized resource demands, as well as low computational load. Every available pose estimate is transmitted to the volant subsystem to be fused. The fusion process is described in detail in section 3.

2.2. Inertial Measurement Unit (IMU)

The IMU is rigidly aligned in the volant platform and, as a result, they share the same frame of reference, {I}. The IMU's pose frame with respect to {W} is extracted by its readings. The orientation of {I} is calculated from the angular velocity ^W ω deduced from the inertial output. Besides the angular acceleration, the sensory drivers provide the option during the orientation profile set up to select the output format among either the rotation matrix ^W R _I , the Euler angles (roll, pitch, yaw), or the quaternion (α, bi, cj, dk). However, we selected the output of the angular acceleration data for our experiments, in order to assess our system against the noise not only during the measurements, but also during the calculation process. The system's translation (T) and angle (θ) dynamic model for each of the three axes is expressed by equations (3) and (4).

where V and ω are the linear and angular velocities of the moving robot, respectively.

and k, t are the sampling and time interval, respectively, while α and γ are the linear and angular accelerations of the moving object in the world frame {W}, respectively.

The acceleration output of the inertial sensor is represented in the moving frame (I), expressed as ^I α = ( ^I α _x , ^I α _y , ^I α _z ). The relationship between them is as follows.

where g is the linear acceleration of gravity, and q _k is the unit quaternion that represents the orientation of frame {I} referring to {W}. The gravity component is calculated from the robot's orientation. Depending on the pose, the tangential component of g is computed and subtracted from the affected axes.

3. HIERARCHICAL FUZZY FUSION SYSTEM

The utilized HFS has multiple inputs and a single output (MISO) follows an incremental structure according to the classification of Chung and Duan (2000). In the incremental structures, there is one FLU at each level, as shown in Figure 3. The same topology is utilized both for rotations and translations. We set up our system so as the most sensitive input, i.e., the IMU measurements, to be at the bottom level. Another important feature of this architecture is its inherent recursiveness. To this end, the last input, i.e., the previous computed measurement, receives a greater weight by being put at the top level, thus resulting into a smoother overall output.

Figure 3 Architecture of the adopted HFS.

There are five Gaussian MFs for all the variables, as they outperform those with less MFs. Since more MFs increase the complexity of our system, a fair trade-off is to restrict them to five. A further increase in MFs’ number would not provide significant accuracy improvement. There are 25 rules for each FLU and hence 75 in total. The chosen Gaussian MF which present good quantitative results is described in equation (8), where σ, c determine the shape and the center of the curve, respectively.

The arrangement of the MFs and the rule bases are shown in Figure 4 and in Tables 1 and 2, respectively. The detailed assessment and examination of the fuzzy sets attributes during their selection process follows in section 4. The specifications of the robot concerning its linear and absolute angular accelerations are ±1 g (9.81) and 0.17, respectively. Thus, the acceleration units are considered large when the acceleration of the robot obtains values close to the specifications and, thus, the centers of the respective MF curves have been put about these limits. The values around 0 are considered zero. All of the cases that do not fit as either large or zero are discriminated as positive or negative depending on whether the robot accelerates or decelerates, respectively: Except for the MFs, the behavior of any fuzzy system is also based on the chosen adjustment methods: namely, defuzzification, aggregation, and implication. The tuning of these principles is the same in all the three FLUs. The aggregation method is set to sum in order to have a smoother output. For the same reason, the implication is set to product and thedefuzzification method is set to centroid, so as to cover the output range in a higher degree.

Figure 4 Inputs of the HFS system. The distribution of the MFs is the same for all inputs. The ranges for the IMU, the camera and the previous output vary from [−20 20], while the acceleration input ranges from [−10 10].

Table 1. Rule base for the fuzzy logic units 2 and 3

	Very low	Low	Average	High	Very high
Very low	VL	L	L	A	A
Low	VL	L	A	A	H
Average	L	L	A	H	H
High	A	A	A	H	VH
Very high	A	A	H	H	VH

Table 2. Fuzzy rule base for the elimination of the error

If T is	and α is	then output is
Negative large	Negative large	Negative large
Negative large	Negative	Negative large
Negative large	Zero	Zero
Negative large	Positive	Negative
Negative large	Positive large	Positive
Negative	Negative large	Negative large
Negative	Negative	Negative
Negative	Zero	Zero
Negative	Positive	Positive
Negative	Positive large	Positive
Zero	Negative large	Negative large
Zero	Negative	Negative
Zero	Zero	Zero
Zero	Positive	Positive
Zero	Positive large	Positive large
Positive	Negative large	Negative
Positive	Negative	Negative
Positive	Zero	Zero
Positive	Positive	Positive large
Positive	Positive large	Positive large
Positive large	Negative large	Negative
Positive large	Negative	Negative
Positive large	Zero	Zero
Positive large	Positive	Positive
Positive large	Positive large	Positive large

3.1. FLU 1—Noise Reduction

FLU 1 is responsible for the reduction of the IMU's error and it has two inputs: translation (rotation) and linear (angular) acceleration. The acceleration data is derived from the sensor's readings while the translation and the rotation data are calculated via the dynamic model of equations (3) and (4), respectively. The rules set of this FLU is exhibited in Table 2. The goal of this unit is to take into account the translation and rotation data, while from the acceleration one to decide whether the robot is moving or not.

3.2. FLU 2—Data Fusion

The data fusion takes place at FLU 2. The two inputs to this unit are the output of FLU 1 and the pose data calculated via the visual data. The main goal of the proposed fusion system is to continuously provide the robot's pose. Taking into account the fact that the visual system suffers from occlusions, the fuzzy rules are formulated so as to rely more on the IMU data than the visual one. Thus, in the case of occlusions which results in a failure of the visual measuring system, the HFS will still provide accurate pose estimates. Hence, the rule base described in Table 1 fuses the two heterogeneous measurements efficiently.

3.3. FLU 3—Output Smoothing

FLU 3 lies at the final level of the applied HFS, and its two inputs are the output of FLU 2 and the system's state from the previous sampling interval. The outcome of FLU 3 is the final output of the HFS, i.e., the robot's pose. The purpose of this unit isto smooth the output values in order to further reduce the noise levels and to increase the robustness of the measurements. The output value at each sampling interval is close to the previous one, eliminating possible high jitters. In cases where the robot remains at a fixed position for a short period of time, our system recursively reduces the noise levels providing at every time interval pose estimates closer to the real ones.

4. SYSTEM PARAMETERS SELECTION AND ANALYSIS

We analyze and compare the minimization of the error functions produced by altering the σ parameter of the MFs. This analysis examines the final output values, which are provided by the different tested cases and are compared to the ground truth ones. The error function can be modeled and expressed using the Langrange multipliers. In this mathematical formulation, the error function has to be minimized while it is subject to certain constrains. When the problem is nonlinear, the multipliers are associated with the solution values and correspond to incremental or marginal prices (Luenberger and Ye 2008). Thus, the limitations in our system are the maximum error levels defined by the robot's requirements of locomotion, which are 5 cm and 2 deg for the translation and rotation error, respectively.

We examined the system's behavior when the deviation (σ) of the Gaussian MFs varies, as shown in Figure 5 a, and when the inputs to the HFS have an arbitrary arrangement. The total possible combinations are 4!, which are all given in Table 3, and in Figure 5 b the most representative ones are illustrated. Each column in this table corresponds to the respective input of the HFS as shown in Figure 3, while each row corresponds to the examined combination of inputs. The Tran s _IMU stands for the translation acquired from the IMU, α is the linear acceleration while Tran s _C is the translation estimated via the visual pose module. Finally, the term Output _k − 1 is the previous system's state. Figure 5 b summarizes all the different examined responses for the respective hierarchical structures. It is apparent that the selected one exhibits an optimal response as the acceleration data reduces the IMU's error directly.

Figure 5 Response of the HFS for different (a) σ and (b) hierarchical structures (color figure available online).

Table 3. All of the potential different input arrangements

(1)	(2)	(3)	(4)
Trans IMU	α	Trans C	Output k − 1
Trans IMU	α	Output k − 1	Trans C
Trans IMU	Trans C	α	Output k − 1
Trans IMU	Trans C	Output k − 1	α
Trans IMU	Output k − 1	Trans C	α
Trans IMU	Output k − 1	α	Trans C
Trans C	Trans IMU	Output k − 1	α
Trans C	Trans IMU	α	Output k − 1
Trans C	α	Trans IMU	Output k − 1
Trans C	α	Output k − 1	Trans IMU
Trans C	Output k − 1	α	Trans IMU
Trans C	Output k − 1	Trans IMU	α
α	Trans C	Output k − 1	Trans IMU
α	Trans C	Trans IMU	Output k − 1
α	Trans IMU	Trans C	Output k − 1
α	Trans IMU	Output k − 1	Trans C
α	Output k − 1	Trans IMU	Trans C
α	Output k − 1	Trans C	Trans IMU
Output k − 1	α	Trans C	Trans IMU
Output k − 1	α	Trans IMU	Trans C
Output k − 1	Trans IMU	Trans C	α
Output k − 1	Trans IMU	α	Trans C
Output k − 1	Trans C	α	Trans IMU
Output k − 1	Trans C	Trans IMU	α

The selected order takes into account that the acceleration data smooth the acquired data. As a result, the acceleration and the IMU inputs have to be processed at the same level. In order to maintain robustness and the capacity of reducing the error, we process these inputs at level 1. Concerning the previous system's state, it is imported at the last stage, i.e., on the top of the hierarchy, in order to smoothen the overall estimation. In fact, all the other three inputs are obtained at the same time step and, therefore, the system performs better when they interact with each other. The visual data is the third input, which allows direct visuo-inertial fusion.

Because the acquired results are compared with ground truth measurements, the MFs selection criterion is based on the output which provided the minimum of the acquired errors. The MFs’ width and length and the position of their center leads to different output results. Furthermore, by selecting the optimal MF type, the physical alignment of the shape provides the minimum error. As the EKF fusion technique is the most common approach for the visuo-inertial fusion task, it is used as a benchmark for the evaluation of the proposed fusion system.

5. EXPERIMENTAL RESULTS

The purpose of the proposed application is to fuse the two heterogeneous measurements. In order to test the efficiency of the HFS application, we compared the output of the proposed fusion method with the traditional EKF as implemented in Kyriakoulis and Gasteratos (2010). The ACROBOTER robot, possessing redundant degrees of freedom, can perform arbitrary movements into the workspace; however, it is always restricted by specific velocity, acceleration, and rotation limits. The environment is a dynamic one; thus, prior to the system movement start, a path planning algorithm must be executed which computes a desired path. The pose estimation system is used during the movement phase in order to control the trajectory of the robot. Thus, the experiments were designed so that the robot has to follow a pre-computed path, thus with known rotations. The desired robot's trajectory is used to benchmark the results acquired both from the HFS and the EKF. The two sensors were tuned so that one of them intentionally acquires erroneous data, i.e., we applied external disturbances to the visual sensor as the IMU already suffers from drift error and we intended to further evaluate our application's efficiency. The plots in Figure 6 display the results during this experiment. Due to the disturbances, the camera provides faulty pose estimates. Concerning the widely used EKF, we tuned it so as to follow as close as possible the more reliable sensor, i.e., the IMU. The EKF, albeit it followed the IMU data, it exhibits high errors due to the disturbances. On the other hand, the HFS application proves its robustness and reliability as it not only rejects the disturbances, but also reduces the drift error of the IMU data being closer to the desired ones.

Figure 6 Output values from the IMU (blue), the visual pose (red), the EKF ([?]), and the proposed fusion method with the HFS application. The desired trajectory is shown with the dotted lines (magenta) (color figure available online).

The overall acquired measurements are shown in Tables 4 and 5. In these tables, it is apparent that the EKF is malfunctioning when one sensor fails to provide accurate measurements, while the HFS fusion does not. The proposed fusion method maintains its precision and its accuracy, even in the cases where one of the sensors fails. As the robot's input is a desired path, these values are used as benchmark for the acquired measurements. Thus, the accuracy is computed as follows.

Table 4. Error evaluation of the tested methods concerning the translation with intentional faulty visual measurements

	IMU	HFS	EKF*	Visual
Tx (cm)	1.71	1.21	5.10	48.59
Ty (cm)	2.55	1.35	6.91	17.75
Tz (cm)	3.79	2.11	1.74	23.57
Accuracy	91.25%	92.10%	89.39%	60.02%
Precision	92.88%	96.76%	91.38%	71.52%

Table 5. Error evaluation of the tested methods concerning the rotations with intentional faulty visual measurements

	IMU	HFS	EKF*	Visual
Rx (deg)	0.56	0.81	1.38	11.31
Ry (deg)	1.94	1.53	3.68	13.92
Rz (deg)	1.36	1.04	5.12	43.28
Accuracy	94.25%	95.89%	90.41%	69.73%
Precision	94.10%	95.35%	93.17%	75.68%

On the other hand, the precision is calculated in order to measure the repeatability of the system. The precision for each measurement is computed by the following

The proposed fusion system was further evaluated against the EKF. The acquired results from the EKF are compared with the ones acquired from the HFS, as shown in Figure 7. Furthermore, we examined our system performance when the last input is not the previous state but the output of the EKF. By observing Figure 7 and Tables 6 and 7, we conclude that the proposed visuo-inertial fusion system outperforms both the EKF and the hybrid one. More specifically, in Figure 7 the output of the system follows better the desired curve than the EKF or the hybrid system. From Tables 6 and 7, it is deduced that the proposed system is precise, accurate, and exhibits low error levels in all the experiments made.

Figure 7 The output values when using (a) the HFS, (b) an EKF ([?]), and (c) a hybrid (EKF + HFS) one (color figure available online).

Table 6. Error evaluation of the tested methods concerning the translations

	HFS	HFS + EKF	EKF*
Trans x (cm)	1.937	3.55	5.37
Trans y (cm)	2.68	6.12	6.71
Trans z (cm)	1.06	1.27	0.87
Accuracy	90.18%	87.29%	86.83%
Precision	94.57%	92.65%	93.02%

Table 7. Error evaluation of the tested methods concerning the rotations

	HFS	HFS + EKF	EKF
Roll (deg)	0.94	1.87	1.41
Pitch (deg)	1.25	2.38	2.18
Yaw (deg)	0.62	0.97	0.74
Accuracy	93.24%	88.71%	89.67%
Precision	95.17%	93.96%	94.16%

Apart from the efficiency of our proposal, it is crucial to evaluate its processing times. It is worth mentioning that the fine tuning of the fuzzy rules is a time consuming process and is application dependent. Therefore, the tuning is implemented off-lineand only the final configuration is being demonstrated here. The real-time processing criterion for the proposed system is based on whether the robot achieves to stop within a safe distance, which lays within the space between the volant platform and an arbitrary obstacle on its trajectory. In order to avoid collision, the platform must decelerate. At least two frames must be recorded during that time in order to be able to control the platform safely. The robot's maximum speed is and its maximum deceleration is . From equations (3) and (5), we set V _k + 1 = 0, and , whence the time required to stop is t _stp  = 0.102 (s) and the distance covered is d = 0.05 m. The time needed for a single pose estimate is 21.23 msec, which is considered real-time as the time required to stop is four times larger.

The overall evaluation of the HFS application for visuo-inertial fusion against the EKF as applied in Kytiakoulis and Gasteratos (2010) is apposed in Table 8. It is obvious that the HFS fusion outperforms the EKF one. However, the main drawback is that the processing time has been significantly increased. Nevertheless, the presented HFS fusion meet the system's timing requirements as the time needed in order to control the robot's immobilization within a safe distance is 102 msec and the computational time for the HFS is 21.23 msec. The error's standard deviation from all the tested data of the HFS measurements is low and proves its efficiency. An overview of the improvements with the proposed method, relative to previous ones, is as follows: 1) the system is able to fuse any different kind of information and not just the same type as traditional fusion methods; 2) the drift error of the IMU is reduced by the process without the need of extra filtering, even when the vision system operates under poor conditions, i.e., occlusions; 3) the use of the acceleration data, which is the output of the sensors, is directly used for the compensation of the undesired movements reducing the process noise; and 4) the overall precision and accuracy of the system are higher than those acquired from the widely used fusion techniques without increasing the resource demands, just the processing times.

Table 8. Overall evaluation of the proposed fusion with HFS compared to EKF ([?])

	HFS	EKF
Proc. time (msec)	21.23	7.27
Accuracy	92.85%	89.07%
Precision	95.46%	92.93%
St. dev error	1.94	3.60

6. CONCLUSION

In this article, we presented a novel HFS-based fusion technique that provides accurate pose estimations to a suspended mechatronic platform, using visual and inertial information. The system initially reduces the IMU's noise. Then, the camera measurements are fused with the inertial ones in a respective FLU. Finally, at the last stage the previous state of the robot's pose is imported in order to smooth the output values. The robot's error tolerances for the translation and the rotation measurements are 5 (cm) and 2 (deg), respectively. The proposed application provides pose estimates that meet these requirements even in the presence of severe external disturbances to one of the sensors. Furthermore, the system's behavior when the Gaussian MF parameters are altered was examined. We also prove that the selected input arrangement provides better results than the tested ones. Finally, the proposed system was compared to the one of Kyriakoulis and Gasteratos (2010), where an EKF is utilized, which is one of the most commonly used visuo-inertial fusion techniques. From the error values, it is clear that our system performs precisely and accurately. The main drawbacks of the HFS arethe increased processing times compared with the traditional fusion techniques and the fact that its off-line adjustment is time consuming. However, when the final configuration is applied to the pose estimation module, the HFS operates in real-time.

The proposed fusion system is capable of fusing heterogeneous data and introduces a novel approach towards the visuo-inertial fusion. Furthermore, the presented fuzzy approach broaches the implementation of HFSs to mechatronics vision systems. Therefore,apart the target application, which is the 3-D pose control of a volant platform at indoors environments, this system can be utilized in a series of applications where 3-D pose estimation is necessary including constructions and ship building, manufacturing, medical applications and rehabilitation etc.

ACKNOWLEDGEMENTS

This work is supported by the E.C. under the FP6 research project for Autonomous Collaborative Robots to Swing and Work in Everyday EnviRonment ACROBOTER, FP6-IST-2006-045530.

Notes

*VL = Very Low; L = Low; A = Average; H = High; and VH = Very High.

The first input is the translation (t) and the second is the acceleration (α).

*Trans _IMU = T _I ; Trans _C = T _C ; and = .

The bold one (illustrated at the first line) is the combination used for the implementation of the final system.

*[?].

*[?].

*[?].

*[?].