Projects

D1: Invasive Software-Hardware Architectures for Robotics

Principal Investigators:

Prof. T. Asfour, Prof. W. Stechele

Scientific Researchers:

D. Gabriel, F. Paus

Abstract

The main research topic of subproject D1 is the exploration of benefits and limitations of Invasive Computing for humanoid robotics. Our goal is to use complex robotic scenarios with concurrent processes and timely varying resource demands to show high-level invasion and negotiation for resources. Invasive computing mechanisms should allow for efficient resource usage and fast execution times while keeping resource utilisation and predictability of non-functional properties, e.g. power, dependability, security, at an optimal level. Therefore, research on techniques of self-organisation are key to efficient allocation of available resources in situations where multiple applications bargain for the same resources. To take full advantage of invasive computing, our algorithms will be able to run on different hardware configurations and support dynamic switching between them.

In the first funding phase, we defined a robotics scenario comprising several algorithms which are required for the implementation of visually guided grasping on a humanoid robot. A number of selected applications have been analysed to provide design specification requirements for projects of the A, B, and C areas. Fixed resources but changing load situations were used for initial evaluations of these algorithms on an invasive hardware platform. Based upon knowledge gained from these experiments, we established performance metrics which were used later on to adapt parameters of running applications in order to provide optimal throughput while available computing resources change. The mentioned adaptable parameters range from simple threshold values to more complex algorithm partitioning schemes. Exploiting information provided by a given invasive computing platform and combining it with the previously defined metrics allowed us to demonstrate improved application performance and quality as compared to state-of-the-art techniques.

In the second funding phase, we investigated resource-aware prediction models and algorithms and evaluated robotic applications on the demonstrator platform. Learning of resource-aware prediction models allows a context-aware and speculative resource management by providing hints about future resource requirements based on previous robot and system experience. We developed a novel resource-aware algorithm for collision-free motion planning that employs self-monitoring concepts in order to identify the difficulty of a given planning problem and dynamically adapt resource usage based on problem difficulty and current planning progress. Furthermore, we evaluated robotic applications on the demonstrator platform. We developed several strategies to map computer vision algorithms (Harris corner detection and SIFT feature extraction and matching) to homogeneous/heterogeneous MPSoCs and evaluated these algorithms on resource-aware programming models.

In the third funding phase, the research activities in Project D1 will focus, on building an invasive memory system—as integral part of a robot control architecture, which encodes prior world and task knowledge, sensorimotor experience and application resource requirements associated with robot actions and scene context. Such memory system will allow 1) encoding, storing and retrieving information about resource requirements, 2) learning prediction models of robot actions and their consequences (world state) as well as their computational resources and 3) implementing speculative resource management system based on resource-aware prediction models learned from experience. In contrast to Phase II, we will consider parallelism in robot actions as it is the case in bimanual tasks or in manipulation and locomotion tasks of a humanoid robot. The approach will be validated in the context of a comprehensive perception and affordance pipeline which combines a wide variety of robot vision algorithms for scene understanding and action selection. Furthermore, we will address the challenges posed by the legacy software of existing robotic systems as currently the code of used algorithms would require complete re-engineering in order to benefit from invasive computing. We will investigate methods for automatic porting of the non-invasive code of the perception and affordance to invasive multicore platforms.

Synopsis

The goals of the project in Phase III are the design of a resource-aware robot memory system as well as the developement of an invasive wrapper for legacy code. Furthermore, we will investigate resource-aware image processing on prosthetic hands.

Resource-aware robot memory system: To perform whole-body grasping, manipulation and locomotion tasks in unknown environments, a humanoid robot needs to understand the environment, the objects compromising it as well as the interaction possibilities, i.e. the affordances of the current scene. Therefore, we will address the research question of using of active vision and active perception methods, by which a humanoid robot autonomously explores its environment and builds affordance-based scene representations which allow the selection of actions and control strategies to achieve the task goal. We will investigate how such representations can be enriched by information about application resource requirements associated with the involved algorithms, robot actions and scene contexts. To facilitate resource-aware action selection and speculative resource management, we will implement a resource-aware robot memory system consisting of 1) prior knowledge memory, 2) long-term memory with knowledge learned from experience and 3) working memory with knowledge about the current world state. We will investigate how resource awareness can improve system performance in the context of scene exploration, planning and execution of parallel robot actions such as locomotion and manipulation tasks. We will mainly focus on active scene exploration for the generation of whole-body actions, as schematically shown in Figure 0.7. During exploration, the robot needs to distribute resources between essential low-level control tasks, e.g. balancing, locomotion and collision-avoidance, high-level planning tasks and the actual scene exploration. On the one hand, a resource-aware active perception will aggressively use resources to improve scene understanding if, for example the robot is in a secure position and no expensive planning algorithm runs. On the other hand, if low-level control or high-level planning occupy most of the resources, the scene exploration will use the remaining resources adaptively to still produce a scene understanding with lower quality, i.e. with more uncertainty as well as lower spatial and temporal resolution.

Invasive wrapper for legacy code: There exists a large body of non-invasive code for robotic applications, e.g. within the aforementioned perception-action framework or the perception pipeline. Multicore platforms offer beneficial performance/power ratio, even if legacy code remains single-threaded. However, access conflicts on shared resources (memory, on-chip communication, I/O) might create large fluctuations in execution time, violating even soft real-time requirements. Traditionally, manual legacy code re- engineering is used to resolve such conflicts. Invasive mechanisms offer methods for application isolation and requirement enforcement. During Phase II, we have investigated manual integration of invasive mechanisms into legacy code. In Phase III we plan investigation of automatic methods. The goal is to investigate benefits and limitations of an invasive wrapper around robotic legacy code, in order to support porting to invasive multicore platforms.

Image processing on the KIT prosthetic hand: We investigate how current progress in the image processing community, especially deep convolutional neural networks (CNNs), can be trained and deployed on resource-constrained hardware on a prosthetic hand. The applications we investigate are image classification and segmentation of known objects. The results from these stages are required for higher-level semi-autonomous control strategies which ease the mental burden for the user of the prosthesis. Together with other subprojects, D1 investigates how accelerators (FPGA, TCPA) can be implemented on the prosthesis' hardware and how invasive computing benefits in this resource-constrained context.

The KIT Prosthetic Hand has an in-hand camera which can perceive objects which the user wants to grasp. Using the built-in embedded compute system, the image can be classified and segmented.

Approach

Resource-aware prediction models for humanoid motion generation: In Phase II, we developed methods to predict world state changes and the corresponding resource requirements for a simple robotic tasks like a pick-and-place scenario. Such models allow to implement special pre-invade requests as a novel mechanism of the invasive computing concept. In order to demonstrate the benefits of resource prediction and invasive mechanisms on robotics applications, we will extend our work towards the generation of whole-body humanoid motion in unstructured environments. To this end, we rely on active perception approaches for scene exploration

The perception pipeline and the different parts it consists of are shown in blue. The goal is to derive a resource-aware realisation of the pipeline with its different components to allow for dynamic adaptation to external resource demands and varying problem complexity.

Resource-aware algorithms: In Phase II, we developed and evaluated novel resource-aware motion planning algorithms [Krö+16] with self-monitoring capabilities, which allow the identification of the underlying planning problem difficulty and to dynamically adapt resource usage based on the planning difficulty and progress. In Phase III, we will extend these adaptation mechanisms to consider the resource demands and priorities of different tasks of the robot. As an example, low-level control algorithms usually have high priority as they have to ensure smooth motion execution and safe operation for both the robot and humans interacting with it. However, a humanoid robot that is currently in a safe position with multiple stable support contacts does not need as many resources for whole-body motion generation and balancing compared to a robot moving through a cluttered environment. In contrast, high-level planning algorithms usually have low priority as their results are needed to complete a given task, and where a delay in providing such results does not compromise functionality.

The optimisation cycle between scene exploration, goal-oriented manipulation and resource usage illustrates the contributing factors and how they influence each other.

Invasive wrapper: During Phase II we investigated manual integration of invasive mechanisms into robotic vision applications. In Phase III, we plan to investigate automatic methods, applied to non-invasive code from the perception pipeline. The proposed approach for porting robotic legacy code to invasive multicore platforms consists of five steps, (a) profiling of non-invasive code, in order to identify application-specific access patterns of periodic tasks, (b) design space exploration on the target platform, in order to identify potential operating points, (c) shared resources partitioning, based on the application-specific access patterns, (d) establishing and configuring an invasive wrapper around the individual legacy applications, in order to isolate access to shared resources, and (e) run-time switching between operating points, in order to enforce end-to-end latency guarantees of legacy code on the invasive multicore platform.

Application profiling, design space exploration, and shared resources partitioning. Upper left: Expected profiling results. Lower left: Design space exploration indicating Pareto-optimal operating points and requirement boundaries. Upper right: Coarse-grained partitioning approach. Lower right: Fine-grained partitioning approach.

A comprehensive summary of the major achievements of the first and second funding phase can be found by accessing Project D1 first phase and Project D1 second phase websites.

Publications