M3ED: Multi-Robot, Multi-Sensor, Multi-Environment Event Dataset

Kenneth Chaney, Fernando Cladera, Ziyun Wang, Anthony Bisulco, M. Ani Hsieh, Christopher Korpela, Vijay Kumar, Camillo J. Taylor, and Kostas Daniilidis

Abstract: We present M3ED, the first multi-sensor event camera dataset focused on high-speed dynamic motions in robotics applications. M3ED provides high-quality synchronized and labeled data from multiple platforms, including ground vehicles, legged robots, and aerial robots, operating in challenging conditions such as driving along off-road trails, navigating through dense forests, and performing aggressive flight maneuvers. Our dataset also covers demanding operational scenarios for event cameras, such as scenes with high egomotion and multiple independently moving objects. The sensor suite used to collect M3ED includes highresolution stereo event cameras (1280×720), grayscale imagers, an RGB imager, a high-quality IMU, a 64-beam LiDAR, and RTK localization. This dataset aims to accelerate the development of event-based algorithms and methods for edge cases encountered by autonomous systems in dynamic environments.

The dataset can be found at https://m3ed.io and the code used to pre-process the data is available at https://github.com/daniilidis-group/m3ed.

Status: Accepted to CVPRW 2023.

Paper Access: CVPRW Proceedings.

NODEO: A Neural Ordinary Differential Equation Based Optimization Framework for Deformable Image Registration

Y. Wu, T. Z. Jiahao, J. Wang, P. A. Yushkevich, M. A. Hsieh, J. C. Gee

Abstract: Deformable image registration (DIR), aiming to find spatial correspondence between images, is one of the most critical problems in the domain of medical image analysis. In this paper, we present a novel, generic, and accurate diffeomorphic image registration framework that utilizes neural ordinary differential equations (NODEs). We model each voxel as a moving particle and consider the set of all voxels in a 3D image as a high-dimensional dynamical system whose trajectory determines the targeted deformation field. Our method leverages deep neural networks for their expressive power in modeling dynamical systems, and simultaneously optimizes for a dynamical system between the image pairs and the corresponding transformation. Our formulation allows various constraints to be imposed along the transformation to maintain desired regularities. Our experiment results show that our method outperforms the benchmarks under various metrics. Additionally, we demonstrate the feasibility to expand our framework to register multiple image sets using a unified form of transformation,which could possibly serve a wider range of applications.

Status: Accepted to CVPR 2022.

Preprint available on arXiv: https://arxiv.org/abs/2108.03443

Topology Control of a Periodic Time-varying Communication Network with Stochastic Temporal Links

L. Shen, X. Yu, and M. A. Hsieh

Abstract: Mobile agents can form communication networks with links that emerge and disappear over time. Information transmission on such networks must pass through a sequence of links that are activated in a certain chronological order. Uncertainties in a link’s activation or deactivation risk violating the chronological order of the formation of links in paths for information transmission. The reasons for these uncertainties can be varied, but their presence prevents the application of existing topology control tools for mitigation. We propose an approach to leverage the pre-stored knowledge of the scheduled links’ events and create time-respecting subpaths accordingly. By estimating the impact of the stochastic timing on each subpath and by measuring how influential the subpaths are, we are able to measure the impact of uncertainties on any link event’s timing on the whole network’s information transmission performance. Reducing the uncertainty in the timing of those links with the highest impact can largely improve the network’s performance. Our method is tested and validated with simulation results.

Status: Accepted to ACC 2022.

Preprint available on arXiv: To come.

Flow-Based Control of Marine Robots in Gyre-Like Environments

G. Knizhnik, P. Li, X. Yu, and M. A. Hsieh

Abstract: We present a flow-based control strategy that enables resource-constrained marine robots to patrol gyre-like flow environments on an orbital trajectory with a periodicity in a given range. The controller does not require a detailed model of the flow field and relies only on the robot’s location relative to the center of the gyre. Instead of precisely tracking a pre-defined trajectory, the robots are tasked to stay in between two bounding trajectories with known periodicity. Furthermore, the proposed strategy leverages the surrounding flow field to minimize control effort. We prove that the proposed strategy enables robots to cycle in the flow satisfying the desired periodicity requirements. Our method is tested and validated both in simulation and in experiments using a low-cost, underactuated, surface swimming robot, i.e. the Modboat.

Status: Accepted to ICRA 2022.

Preprint available on arXiv: https://arxiv.org/abs/2203.00796

Learning to Swarm with Knowledge-Based Neural Ordinary Differential Equations

T. Z. Jiahao, L. Pan, and M. A. Hsieh

Abstract: Understanding decentralized dynamics from collective behaviors in swarms is crucial for informing robot controller designs in artificial swarms and multiagent robotic systems. However, the complexity in agent-to-agent interactions and the decentralized nature of most swarms pose a significant challenge to the extraction of single-robot control laws from global behavior. In this work, we consider the important task of learning decentralized single-robot controllers based solely on the state observations of a swarm’s trajectory. We present a general framework by adopting knowledge-based neural ordinary differential equations (KNODE) — a hybrid machine learning method capable of combining artificial neural networks with known agent dynamics. Our approach distinguishes itself from most prior works in that we do not require action data for learning. We apply our framework to two different flocking swarms in 2D and 3D respectively, and demonstrate efficient training by leveraging the graphical structure of the swarms’ information network. We further show that the learnt single-robot controllers can not only reproduce flocking behavior in the original swarm but also scale to swarms with more robots.

Status: Accepted to ICRA 2022.

Preprint available on arXiv: https://arxiv.org/abs/2109.04927

KNODE-MPC: A Knowledge-Based Data-Driven Predictive Control Framework for Aerial Robots

K. Y. Chee, T. Z. Jiahao and M. A. Hsieh

Abstract: In this letter, we consider the problem of deriving and incorporating accurate dynamic models for model predictive control (MPC) with an application to quadrotor control. MPC relies on precise dynamic models to achieve the desired closed-loop performance. However, the presence of uncertainties in complex systems and the environments they operate in poses a challenge in obtaining sufficiently accurate representations of the system dynamics. In this letter, we make use of a deep learning tool, knowledge-based neural ordinary differential equations (KNODE), to augment a model obtained from first principles. The resulting hybrid model encompasses both a nominal first-principle model and a neural network learnt from simulated or real-world experimental data. Using a quadrotor, we benchmark our hybrid model against a state-of-the-art Gaussian Process (GP) model and show that the hybrid model provides more accurate predictions of the quadrotor dynamics and is able to generalize beyond the training data. To improve closed-loop performance, the hybrid model is integrated into a novel MPC framework, known as KNODE-MPC. Results show that the integrated framework achieves 60.2% improvement in simulations and more than 21% in physical experiments, in terms of trajectory tracking performance.

Status: Published in IEEE RA-L. https://doi.org/10.1109/LRA.2022.3144787

Preprint available on arXiv: https://arxiv.org/abs/2109.04821

Learning and Leveraging Features in Flow-Like Environments to Improve Situational Awareness

T. Salam, V. Edwards, and M. A. Hsieh

Abstract: This letter studies how global dynamics and knowledge of high-level features can inform decision-making for robots in flow-like environments. Specifically, we investigate how coherent sets, an environmental feature found in these environments, inform robot awareness within these scenarios. The proposed approach is an online environmental feature generator which can be used for robot reasoning. We compute coherent sets online with techniques from machine learning and design frameworks for robot behavior that leverage coherent set features. We demonstrate the effectiveness of online methods over offline methods. Notably, we apply these online methods for robot monitoring of pedestrian behaviors and robot navigation through water. Environmental features such as coherent sets provide rich context to robots for smarter, more efficient behavior.

Status: Published in IEEE RA-L. https://doi.org/10.1109/LRA.2022.3141762

Preprint available on arXiv: https://arxiv.org/abs/2109.06107

Controlling stochastic switching

Using control to shape stochastic escape and switching dynamics

Dhanushka Kularatne, Eric Forgoston, and M. Ani Hsieh

Abstract: We present a strategy to control the mean stochastic switching times of general dynamical systems with multiple equilibrium states subject to Gaussian white noise. The control can either enhance or abate the probability of escape from the deterministic region of attraction of a stable equilibrium in the presence of external noise. We synthesize a feedback control strategy that actively changes the system’s mean stochastic switching behavior based on the system’s distance to the boundary of the attracting region. With the proposed controller, we are able to achieve a desired mean switching time, even when the strength of noise in the system is not known. The control method is analytically validated using a one-dimensional system, and its effectiveness is numerically demonstrated for a set of dynamical systems of practical importance.

Status: Published online. doi: 10.1063/1.5090113

Coordinating AGVs for Automated Warehouses

Coordination of multiple AGVs: a quadratic optimization method

Valerio Digani, M. Ani Hsieh, Lorenzo Sabattini, and Cristian Secchi

Abstract: This paper presents an optimization strategy to coordinate a fleet of Automated Guided Vehicles (AGVs) traveling on ad-hoc pre-defined roadmaps. Specifically, the objective is to maximize traffic throughput of AGVs navigating in an automated warehouse by minimizing the time AGVs spend negotiating complex traffic patterns to avoid collisions with other AGVs. In this work, the coordination problem is posed as a Quadratic Program where the optimization is performed in a centralized manner. The proposed method is validated by means of simulations and experiments for different industrial warehouse scenarios. The performance of the proposed strategy is then compared with a recently proposed decentralized coordination strategy that relies on local negotiations for shared resources. The results show that the proposed coordination strategy successfully maximizes vehicle throughput and significantly minimizes the time vehicles spend negotiating traffic under different scenarios.

Status: Published in Autonomous Robots.  Preprint to come.

Planning Optimal Paths in General Flows

Going With The Flow: A Graph Based Approach to Optimal Path Planning in General Flows

Dhanushka Kularatne, Subhrajit Bhattacharya, and M. Ani Hsieh

Abstract: Autonomous surface and underwater vehicles (ASVs and AUVs) used for ocean monitoring are typically deployed for long periods of time and must operate with limited energy budgets. Coupled with the increased accessibility to ocean flow data, there has been a significant interest in developing energy efficient motion plans for these vehicles that leverage the dynamics of the surrounding flow. In this paper, we present a graph search based method to plan time and energy optimal paths in static and time-varying flow fields. We also use tools from topological path planning to generate optimal paths in different homotopy classes to facilitate simultaneous exploration of the environment by multi-robot teams. The proposed strategy is validated using analytical flow models, actual ocean data, and in experiments using an indoor laboratory testbed capable of creating flows with ocean-like features. We also present an alternative approach using a Riemannian metric based approximation for the cost functions in the static flow case for computing time and energy optimal paths. The Riemannian approximation results in smoother trajectories in contrast to the graph based strategy while requiring less computational time.

Status: Accepted and to appear in Autonomous Robots. Preprint to come.