2025; Master's Thesis: Part 3 of 4: Drone Control

Introduction

This is part three of the four part series covering my Master's Thesis at KAIST titled "Accelerating Policy Learning for Robust Control of Robotic Manipulators and Aerial Vehicles via Physics-Informed Guidance" under Professor Dong Eui Chang. The full thesis can be found at KAIST library: THESIS. It has in-depth explanation of all the concepts, simulation and control design, and the experimental results. Please note that these posts do not cover the entire thesis in detail and is instead built upon my thesis defence slides.

This blog post covers the drone control algorithm based on approximate Hamilton-Jacobi-Bellman (HJB) solutions with residual reinforcement learning.

Problem Statement

We have a drone flying over the sea. The high-level task is to hold the payload and fly through a given trajectory to the goal position. We utilize the same tracking problem design, however this time we define it as an output regulation problem:

with an exosystem f_bar that generates trajectories, and the task is to design control u such that

Under this, the controller can be designed using Nonlinear Systems Toolbox (NST). NST provides an easy access to approximate HJB solutions for optimal control problems using polynomial approximations.

Simulation Design

The simulation has 4 major components:

• Crazyflie 2.1 brushless that is controlled using mass-normalized thrust f_m and spatial angular velocity ω,

• Reference trajectory exosystem: quintic polynomial trajectories using TOGT-Planner that is modeled as an autonomous exosystem,

• Disturbances: wind drag, thrust noise, mass and inertial variations,

• JAX JIT compilation for fast GPU simulations utilizing Crazyflow as the base simulator.

The drone system is represented by position r, velocity v and attitude quaternion q.

Note that the reference trajectories are for position r_d and yaw angle ψ_d.

Control Design

Firstly, to utilize NST, we require two conditions for the system dynamics to satisfy:

• Control dimension ≥ output dimension. We have 4 control inputs and for tracking we require both position tracking and attitude tracking, but using q will make the output dimension higher. Hence we would like to utilize yaw angle, ψ but that transformation within output function would make the polynomial approximation difficult.

• States, control, exostate and output must be in Euclidean space. But we have quaternion representing drone attitude with q∈S^3,

To solve the first problem, we need to dive into geometric methods. We can utilize Hopf fibration to obtain the dynamics with thrust direction ρ∈S^2 and half yaw angle ψ_N∈ R instead as

Please note that here, we remove the upside down configuration since the mapping between q and (ρ, ψ_N) is not one-to-one for all drone attitudes. Note that this mapping is similar to Watterson and Kumar's drone control on SO(3) paper but utilizes Professor Chang's approach which is a more generalized version.

For the second case we utilize stable embedding technique that extends ρ∈R^3 as

We can now utilize NST to compute the model-based controller.

We then utilize RL with an adaptive blending approach and residual RL controller design (section redacted).

Results

Due to the simulation and control design, our approach took only 12 minutes to train on a standard personal computer!

Our approach has faster learning and the highest performance among the ablations. Here Vblend and Ablend are the adaptive blending mechanisms. Without any further fine-tuning, we also performed experiments against the Lee controller under wind and mass disturbances, and our controller had the lowest tracking RMSE in all cases:

A video showing the comparison is given below under wind and mass disturbance (2x speed). Note that we utilize the racing rings to show that the tracking is performed correctly.

We also tested the algorithm under extreme scenarios:

• High speed trajectories beyond the nano drone's dynamic limits

• Extreme disturbances (multiple wind sources, swinging payload and clipped propellers)

The video below shows the performance (1x speed).