Welcome to this special issue of Autonomous Robots (AURO) that features articles extended from conference papers of the 2020 Robotics: Science and Systems (RSS) conference, which was held virtually in July 2020. The articles were carefully selected for invitation to this special issue out of the conference papers that had already gone through a highly competitive selection process. We hope that the readers will appreciate the additional material included and the quality improvements resulting from the journal review and article revision process.

There are sixteen articles in this special issue, which can be broadly categorized into the following groups:

  • Perception, Localization, and Scene Understanding,

  • Robot Learning,

  • Coordination of Multi-Robot and Human-Robot Teams,

  • Planning and Control, and

  • Robotic Systems.

In the first group—Perception, Localization, and Scene Understanding—we have four articles. “Correspondence identification for collaborative multi-robot perception under uncertainty” by Gao et al. (2021) explores multi-robot collaborative perception, in particular, the problem of correspondence identification, the ability of multiple robots to refer to the same object in their own field of view. The authors introduce a novel principled approach that transforms the correspondence identification problem into a graph matching problem under the framework of regularized constrained optimization. The article “Embodied scene description” by Tan et al. (2021) puts forward the notion of embodied scene description, which exploits the embodiment ability of agents to find optimal viewpoints in their environment for scene description tasks. A learning framework with the paradigms of imitation learning and reinforcement learning is then established to teach the agents to generate corresponding sensorimotor activities. The article “LatticeNet: fast spatio-temporal point cloud segmentation using permutohedral lattices” by Rosu et al. (2021) proposes LatticeNet, a novel approach for 3D semantic segmentation, which takes raw point clouds as input. The proposed lattice allows for fast convolutions while keeping a low memory footprint. The LatticeNet approach is also used for instance and dynamic object segmentation.“OverlapNet: a siamese network for computing LiDAR scan similarity with applications to loop closing and localization” by Chen et al. (2021) proposes a modified version of Siamese Deep Neural Network to estimate similarity between pairs of LiDAR scans collected by autonomous cars. The paper then shows that this similarity can be used effectively to help with loop closing in the context of SLAM and global localization.

In the second group that focuses on Robot Learning we have five articles. “Affordance-based robot object retrieval” by Nguyen et al. (2021) develops a model to predict the appearance of an object based on the verbal description of its usage. In addition, the authors present a new dataset of verb-object pairs denoting a potential object usage for a variety of objects and usages. The article “Expert Intervention Learning: An online framework for robot learning from explicit and implicit human feedback” by Spencer et al. (2021) makes an observation that expert feedback, whether made via intervention or not, provides information about the quality of the current state of the robot, the quality of the action it executes, or both. The authors then utilize this observation to constrain the learner’s value function and provide an efficient online learning approach for it. The article “Learning latent actions to control assistive robots” by Losey et al. (2021) aims to achieve intuitive, user-friendly control of assistive robots by embedding robot’s high-dimensional actions into low-dimensional and human-controllable latent actions learnt from offline task demonstrations. The authors evaluate their approach via user studies on a group of subjects including some with disabilities. The article “Learning temporal logic formulas from suboptimal demonstrations: Theory and experiments” by Chou et al. (2021) presents a method for learning multi-stage tasks from demonstrations by learning the logical structure and atomic propositions of a consistent linear temporal logic formula. In addition to simulated experiments, the authors show that their approach can learn a real-world multi-stage tabletop manipulation task using a physical 7-DOF arm. The article “VisuoSpatial Foresight for physical sequential fabric manipulation” by Hoque et al. (2021) develops a framework for learning fabric dynamics that can be efficiently reused to accomplish different sequential fabric manipulation tasks with a single goal-conditioned policy. The authors show that it significantly improves the efficiency of fabric folding and enables physical sequential fabric folding tasks that were not previously possible.

In the group focusing on Coordination of Multi-Robot and Human-Robot Teams we have four articles. “ALGAMES: a fast augmented Lagrangian solver for constrained dynamic games” by Le Cleac’h et al. (2021) studies dynamic games and introduces an augmented Lagrangian-based solver that handles trajectory-optimization problems with multiple actors and general nonlinear state and input constraints. The authors evaluate the solver in the context of autonomous driving on scenarios that involve high complexity of interactions. The article “Bayes-Nash: Bayesian inference for Nash equilibrium selection in human-robot parallel play” by Bansal et al. (2021) models shared workspaces with humans and robots trying to achieve independent goals as general-sum games, constructs a framework that utilizes the Nash equilibrium solution concept to consider the interactive effect of both agents and finds corresponding Pareto-optimal solutions. The studies performed by the authors on human participants interacting either with other humans or with different robot agents show performance similar to that of human-to-human parallel play interactions. The article “Dynamic multi-robot task allocation under uncertainty and temporal constraints” by Shushman et al. (2021) develops a multi-robot allocation algorithm that decouples the key computational challenges of sequential decision-making under uncertainty and multi-agent coordination, and addresses them in a hierarchical manner. On the theoretical side, the authors show that the approach is optimal in expectation and complete under certain assumptions, and on the experimental side, they show its computational benefits against other approaches and validate the results on simulated multi-arm conveyor belt pick-and-place and multi-drone delivery dispatch in city domains. The article “Heterogeneous graph attention networks for scalable multi-robot scheduling with temporospatial constraints” by Wang et al. (2021) proposes a novel heterogeneous graph attention network model to learn multi-robot scheduling policies. The authors show that the proposed model is end-to-end trainable via imitation learning on small-scale problems, generalizes to large, unseen problems, and outperforms state-of-the-art multi-robot scheduling methods on a variety of test scenarios involving both homogeneous and heterogeneous robots.

In the fourth group—Planning and Control—we have two articles. The article “Fast nonlinear risk assessment for autonomous vehicles using learned conditional probabilistic models of agent futures” by Jasour et al. (2021) presents fast non-sampling based methods to assess the risk for trajectories of autonomous vehicles in systems where probabilistic predictions of other agents’ futures are generated using Deep Neural Networks. The authors show that their proposed methods are effective for quickly assessing the probability of highly unlikely events. The article “Stochastic spatio-temporal optimization for control and co-design of systems in robotics and applied physics” by Ethan et al. (2021) provides a novel sampling-based stochastic optimization framework suitable for the general class of semi-linear spatio-temporal partial differential equations which describes many systems in robotics and applied physics. The proposed framework is utilized for simultaneous policy and actuator co-design optimization, and the authors demonstrate its efficacy on a variety of systems in robotics and applied physics including an infinite degree-of-freedom soft robotic manipulator.

Finally, the group Robotic Systems contains one article: “AlphaPilot: autonomous drone racing” by Philipp et al. (2021). It presents an impressive system for autonomous, vision-based drone racing combining learned data abstraction, nonlinear filtering, and time-optimal trajectory planning. The system has been deployed at the first autonomous drone racing world championship, the 2019 AlphaPilot Challenge, where it has successfully guided the drone through tight race courses reaching speeds up to 8 m/s and ranked second.

We sincerely hope that the readers will enjoy these articles extended from selected RSS 2020 papers. We would also like to thank the reviewers who worked hard on providing excellent reviews for the manuscripts. Finally, we would like to thank Gaurav Sukhatme for letting us assemble this special issue and Sarvagnan Subramanian for helping us with the process.

Guest Editors:

Maxim Likhachev, Carnegie Mellon University, Pittsburgh, USA

Sven Behnke, University of Bonn, Bonn, Germany