Prof. Ackermann, vision-based navigation and the PG master’s degree programme at HfT Stuttgart

ABSTRACT

The two topics of the article seem to have absolutely nothing to do with each other and, as can be expected in a contribution in honor and memory of Prof. Fritz Ackermann, they are linked in his person. Vision-based Navigation was the focus of the doctoral thesis written by the author, the 29th and last PhD thesis supervised by Prof. Ackermann. The International Master’s Program Photogrammetry and Geoinformatics, which the author established with colleagues at Stuttgart University of Applied Sciences (HfT Stuttgart) in 1999, was a consequence of Prof. Ackermann’s benevolent promotion of international knowledge transfer in teaching. Both topics are reflected in this article; they provide further splashes of color in Prof. Ackermann’s oeuvre.

KEYWORDS:

• Vision-based navigation
• visual odometry
• photogrammetry and geoinformatics
• international master’s degree program

1. Introduction

In 1987 and certainly for many years before, Prof. Ackermann’s home was decorated with a memorable world map that immediately caught the eye of every guest. The author had been working at the Institute of Photogrammetry (IfP) for a year and was invited to Prof. Ackermann’s home along with all the staff members of the institute. And there was this map of the world, studded with pins, each indicating a place that Ackermann visited on his flights. London, Lisbon, Lausanne, Ottawa, Helsinki, Hamburg, Rio de Janeiro, the locations of the ISPRS Congresses from 1960 to 1984 were marked and many more cities. Locations where the ISPRS Midterm Symposia took place were among them, but also remote places to which Prof. Ackermann was invited due to his worldwide reputation. The map has remained in my memory as a synonym for Ackermann’s being at home in the world of ISPRS.

That, in return, the photogrammetric world came to Prof. Ackermann in Stuttgart I was able to witness during my first PhoWo participation in 1987. Since 1973, Ackermann has organized the Photogrammetric Week (PhoWo) in Stuttgart together with Carl Zeiss, Oberkochen. PhoWo is an education and technology transfer event of high scientific level that combines science and practice. Lectures in the morning and device demonstrations in the afternoon were the defining element of PhoWo at that time.

Linking teaching and practice was also the central concern of the International Training Centre for Photogrammetry (IPO). In the 1980s and 1990s, Prof. Mohl and Prof. Mohr organized annual courses lasting several months at Stuttgart University of Applied Sciences with practical training on photogrammetric instruments for participants from developing countries. Prof. Ackermann was closely associated with the IPO and his advice strengthened the initiative, which was jointly supported by a foundation for development cooperation together with industry.

IPO funding came to an end and the time has come to make a real move toward bachelor’s and master’s degree programs in the German education system. The German Academic Exchange Service DAAD encouraged to develop postgraduate courses with high significance for developing countries and with the IPO experience at its back, the geomatics group at HfT Stuttgart was well prepared for an international master’s degree program in photogrammetry and geoinformatics.

The background to the other topic of the article, vision-based navigation, has its origin in the Collaborative Research Center SFB 228 “High Accuracy Navigation – Integration of Navigational and Geodetic Methods”, which was funded by the German Research Foundation from 1984 to 1995. IfP was involved in this SFB and worked intensively on the subject of GPS in photogrammetry, in particular on the kinematic differential positioning of the aerial camera with high accuracy for the purposes of aerotriangulation or for the direct absolute positioning of airborne sensors. Against this backdrop, airborne laser scanning (ALS) was developed. In addition to digital image matching, ALS was the second major project of the IfP in the 1980s, as Prof. Ackermann himself put it in his very personally formulated review of the development of the IfP during his service from 1966 to 1992 (Ackermann 2016).

The innovative research into image matching at IfP already had some practical successes to its credit. The correlation of two image regions with the tools of least squares estimation led to a significant increase in accuracy compared to the previously used methods and offered possibilities for the evaluation of the matching quality. At IfP, Prof. Förstner had made a name for himself with an interest operator named after him and had advanced a method for point transfer with feature-based matching. Matching the robustly extracted points was used as an alternative or supplement to computationally intensive least squares image matching. Hence, it was obvious to build on all these foundations to contribute with vision-based navigation to the Collaborative Research Center SFB “High Accuracy Navigation”.

2. Vision-based navigation

The computer vision community started in the 1980s to reconstruct camera pose (position and attitude) and three-dimensional structure from camera images and called this problem structure from motion (SFM). The background and visions of and for SfM were manifold: On the one hand, it was the awareness that humans perceive a 3D structure from the projected 2D motion field (retina) of a moving scene, on the other hand, the robotics and automotive industries were looking for solutions to estimate the correct motion of a robot or vehicle with the input of a single camera or a stereo camera. As a photogrammetric technique, the SfM method is well known to photogrammetrists. The fundamental processing steps that lead to the simultaneous determination of the camera orientation and the 3D structure of a scene from an aerial image block are known as automated digital aerotriangulation.

Ambitious goals, such as the Mars Exploration Program of NASA, gave an enormous boost to research in the 1980s. SfM was not an ideal method for a Mars robot because SfM computes the pose (orientation) of all images and the 3D structure of a scene (mostly represented by the coordinates of tie points) simultaneously in a final step using bundle adjustment. What is needed, however, is an estimate of the camera’s 3D movement as soon as a new image is captured – ideally in real time. In 2004, Nister introduced the term visual odometry for this process of incrementally estimating pose and structure, using only the input from cameras. An excellent review of the fundamentals of visual odometry is given by Scaramuzza and Fraundorfer (2011). In this review, Scaramuzza and Fraundorfer write “Although the first two decades (the 1980s and 1990s) witnessed many offline implementations, only in the third decade did real-time working systems flourish, which has led VO to be used on another planet by two Mars-exploration rovers for the first time.”

The work on visual navigation at IfP fits into this context. The author worked in the early 1990s on his PhD thesis with the title “Image sequence analysis for passive navigation”; from today’s point of view a contribution to the work in the field of visual odometry. The state of the art in image capture have been video cameras that recorded single frames and frame grabbers that converted the analog video stream into a digital one and stored it on a computer. Such equipment was installed in a Mercedes-Benz test vehicle. Experiments were conducted on a Mercedes-Benz test site in Rastatt, Germany, which was set up for testing driver assistance systems. Mono image sequences were recorded, on which the developments and analysis subsequently focused.

In computer vision, most research work during this time was done with stereo cameras. Typical steps were the extraction and matching of corners in stereo image pairs. Tracking was done using triangulated corner locations, which were then incorporated into the motion estimation step. The latter is related, although under quite different prerequisites, to the photogrammetric research of the 1970s on block adjustment with independent models, to which Prof. Ackermann made well-known contributions.

Not surprisingly, the second line of development of visual odometry (VO) is limited to the use of a single camera. Multiple camera solutions were added in later developments, but will not be considered here. The single camera solution was flawed by the fact that motion and structure are determined using only an arbitrary scale, and absolute scale determination requires integration with other sensors (e.g. GPS, IMU, range sensors) or measurements of elements in the scene (e.g. control points, size of objects).

Typical steps of monocular VO are similar to those of stereo VO, but differ most in motion estimation. Extraction and matching or tracking of points or lines in the images is followed by estimation of motion between successive time instants (t_i, t_i–1). Early in the development of VO systems, it was discovered that bundle adjustment is particularly well suited for local optimization. This is used iteratively to update the trajectory and, if necessary, the reconstructed structure over the last m images.

In his work on monocular VO (Hahn 1995), the author used the Förstner interest operator as a feature detector and solved feature correspondence both feature-based and area-based (with least squares matching). He estimated motion via of the orientation of an image pair (t_i, t_i–1) and recovered structure represented by the triangulated coordinates of matched and tracked features, for which he developed a 3D Kalman filter approach. Control points were included to coordinate in a higher-level coordinate system and thereby also to obtain absolute scale for the trajectory and structure.

In order to obtain a bundle adjustment optimized solution for the image sequence as a reference for the VO estimation, the bundle adjustment program PAT-B developed at IfP under Prof. Ackermann (for aerial triangulation) was used. The motion and structure parameters obtained from the VO solution were introduced into PAT-B as approximate values to compute an optimized simultaneous solution for the acquired image sequence. The experiments carried out on the test site in Rastatt over a distance of 120 m showed that the estimation of the vehicle’s own motion with respect to a stationary coordinate system succeeded with an accuracy level of 0.1–1 gon for the orientation and with 0.1 − 0.3 m for the position of the camera.

3. International master’s degree program photogrammetry and geoinformatics at HfT Stuttgart

The Bologna Declaration was signed in 1999 by the Ministers of Education from 29 European countries. This declaration initiated a process to create a European higher education area that offers students and graduates great freedom to move between universities across national borders. Many European countries, including Germany, were not prepared for this with their education systems, and so a system was adopted that is essentially based on two main cycles, undergraduate and graduate.

In ISPRS, intensive thought has always been given to how technology transfer should be structured. A very good example of a sustainable technology transfer initiative is PhoWo, which can look back on a success story of more than one hundred years. It started with the intention of bringing together scientists, developers and practitioners and was organized in this tradition by Prof. Ackermann together with industrial partners over a period of two decades (1973–1993).

Technology transfer for sustainable development in developing countries was particularly on the minds of Prof. Shunji Murai. In 2000, Prof. Murai, ISPRS President from 1992 to 1996, wrote down his thoughts on this subject, which he introduced with the question: “Does education in developing countries need technician level or higher level such as master and doctoral course?” Boiled down to a minimum, his answer is that education needs leaders with higher knowledge and research skills (Murai 2000). Leaders (Murai’s attention first turned to leaders from abroad) should be excellent teachers as well as researchers, good friends and great supporters. The researcher is expected to be able to “solve unique and strange problems in developing countries, the answer to which cannot be found in existing textbooks”. Good friends “understand people, culture and infrastructure in developing countries” and a great supporter has the organization and its employees in mind when it comes to technology transfer, especially the capable employees who are to be supported, for example, with a scholarship for higher education, training or research (Murai 2000).

At the end of the 1980s, it was time to make a real transition to bachelor’s and master’s degree programs in the German education system. Thoughts like those formulated by Murai also existed in the geomatics group at HfT Stuttgart when the concept for an international master’s degree program in photogrammetry and geoinformatics was developed. At the DAAD, there was a funding line for so-called development-related postgraduate courses (EPOS), which was an additional incentive to give the master’s program an international orientation and to specifically involve developing countries.

The curriculum developed by the author and his colleagues was accredited in 1999 by ASIIN, the accreditation agency for degree programs in engineering, computer science, natural sciences and mathematics. Reaccreditation took place in 2008 and 2016. Accordingly, the responsible ministry for science approved the establishment of the International Master’s program in Photogrammetry and Geoinformatics in 1999. The author had the pleasure of leading the course in its first years from 1999 to 2001, before becoming vice president of the university. Since then, his colleague Prof. Schröder had directed the course with great care and continuously at full capacity with 25 students per course.

The EPOS application to the DAAD was also successful. EPOS funding aims at “specialists and executives from developing and emerging countries who are pursuing further qualifications in postgraduate courses in order to become future decision-makers and partners for Germany” (EPOS, 2020). From the very beginning, the Master’s program in Photogrammetry and Geomatics has had the opportunity to award up to 8 individual DAAD full-time scholarships to highly qualified and carefully selected participants to pursue their academic education at HfT Stuttgart. Since EPOS funding is also time-limited, postgraduate programs must periodically reapply for renewal of funding to the multinational selection committee, which has always been successful to date.

Let me conclude with a few more figures (PG, 2022). Today, the master course can proudly look back on 23 years of experience in international postgraduate education. During this time, more than 500 graduates have successfully completed their studies. Most of them have returned to their home countries, if not immediately, then after a few years of working in Germany, and have made careers in government agencies, industry and universities. The students came from 89 different countries until today, which presumably no other educational institution in photogrammetry, remote sensing, geoinformatics in Germany can refer to. The intensive alumni work of the study program is also pleasing, which is mainly due to the commitment of the colleague Prof. Behr. In 2022, many alumni – good friends and great supporters – will meet again for the already 12th alumni conference/summer school AGSE, which will take place this year in Kerala, India.

4. Concluding remarks

This article describes research activities on visual navigation initiated by Prof. Ackermann toward the end of his service at the University of Stuttgart. Although Prof. Ackermann devoted most of his research activities to aerial photogrammetry, he also supported topics off the streamline, as a look at the PhD theses he supervised reveals (Ackermann 2016). Recognizing the opportunities of new technological developments at a very early stage was one of his particular strengths.

The introduction of a two-cycle system of degrees initiated by the Bologna Declaration has changed university education in Germany enormously. Recognizing opportunities that arise from this change is the content of the second topic of this article. With its development-related postgraduate orientation, the international master’s program in photogrammetry and geoinformatics at HfT Stuttgart is a successful example of a sustainable contribution to technology transfer in many countries around the world.

Disclosure statement

No potential conflict of interest was reported by the author.

Additional information

Notes on contributors

Michael Hahn

Michael Hahn is Professor for Photogrammetry and Remote Sensing at Stuttgart University of Applied Sciences, Stuttgart, Germany. He studied Geodesy at the University of Karlsruhe from 1978 to 1983. From 1986 to 1990 he was Assistant of Prof. Ackermann and until 1998 Senior Lecturer at the IfP. He received his PhD degree from the University of Stuttgart under the supervision of Prof. Fritz Ackermann, Prof. Wolfgang Förstner and Prof. Dieter Fritsch. From 1996 to 1997 he was ARC Research Fellow and Adjunct Professor at the Space Centre for Satellite Navigation, Queensland Univ. of Technology in Brisbane, Australia. In 1998, he joined Stuttgart University of Applied Sciences, became Course Director of the International Master’s Program in Photogrammetry and Geoinformatics, served as Vice President for Science, Research and Internationalization from 2001 to 2007, and subsequently directed the competence center Geodesy and Geoinformatics. In 2015, he was appointed a member of the Baden-Württemberg Center of Applied Research BW-CAR. The University of Queensland, Brisbane, Australia awarded him an honorary professorship in 2002, the University of Tehran, Iran, an adjunct professorship in 2015.