2025; Master's Thesis: Part 2 of 4: Manipulator Control

Introduction

This is part two of the four part series covering my Master's Thesis at KAIST titled "Accelerating Policy Learning for Robust Control of Robotic Manipulators and Aerial Vehicles via Physics-Informed Guidance" under Professor Dong Eui Chang. The full thesis can be found at KAIST library: THESIS. It has in-depth explanation of all the concepts, simulation and control design, and the experimental results. Please note that these posts do not cover the entire thesis in detail and is instead built upon my thesis defence slides.

This blog post covers manipulator control using PID and disturbance observer (DOB) along with auto-tuning using reinforcement learning.

Problem Statement

Please note that while it is similar to my other research work with Hanwha, this work was performed independently and is not connected directly. Only the hardware resources like the Stewart platform from Acrome and the concept of using manipulators on a ship were utilized. We also utilize the OpenManipulator-X robot for our hardware validation instead.

We have a 4-DOF(degrees of freedom) manipulator on a ship in the sea and the high-level task is to lift a payload and hold it stably beneath a drone. To represent the task mathematically, we consider this as a tracking problem. That is, for a system,

with dynamics f, states x, disturbance d and an output y, we design control input u to track a reference y_d, such that

Under this, the overall task is defined as robustly tracking the given trajectory.

Simulation Design

Our simulation has three major components with controllable parameters:

• Sea-state simulation

• Ship motion simulation

• Manipulator simulation

We also apply disturbances to the manipulator through varying link mass, payload mass, and friction. This domain randomization allows the agent to learn policies robust to various scenarios.

Control Design

We first designed a feedback linearizing PID controller with feedforward acceleration for tracking control, and a nonlinear disturbance observer (NDOB) to compensate disturbances as

where θ_e = θ_d - θ is the output tracking error, K_p, K_i, K_d are the diagonal PID gain matrices, z_d is the NDOB state, L is the diagonal NDOB gain matrix, and d_hat is the estimated disturbance.

We would like to tune these PID gains automatically using RL and also perform gain adaptation based on the operating conditions. However, the range of gains to search is extremely high, in R_(>0)^3n.

We note that using control input tau = tau_c - d_hat, and for slow varying disturbances, the error dynamics for each joint j’s error dynamics can be decoupled and written as

Using Routh-Hurwitz criterion provides us sufficient conditions for asymptotic stability:

We utilize this to reduce the action space of the PID tuning RL agent into a safe subspace as

 where each joint’s action a_{p_j}, a_{d_j}, a_{i_j} ∈ [10^(-3), 1]^3, and K_pmax, K_dmax are maximum gain values.

However, there are problems in this design:

• RL agent can continuously alter gains causing bad gain adaptation during fast changes,

• Noise can alter the gains frequently, causing unstable behavior,

• The action space reduction holds only for slow varying disturbances.

We force the agent to update gains only when necessary, through a DOB based trigger mechanism. We compute a disturbance estimate feature and activate based on confidence intervals (section redacted).

To embed prior knowledge into the RL agent, we consider behavior cloning (BC) expert gains/gain schedulers. Most approaches utilize:

• Ziegler-Nichols (ZN) gains,

• Expert manually tuned gains,

• Heuristic gain adaptations.

However, we needed a simple, fast and effective expert design, preferably through verbal instructions. We realized that large language models (LLMs) have been trained upon control theory data and could be utilized for this task!

To generate such expert gain schedulers, we utilized a multi-agent scheme (section redacted):

• Agent 1 converts verbal instructions into code and runs the simulation

• Agent 2 reasons to suggest new ideas based on simulation results

• Agent 3 reasons to generate instructions for new schedules given the constraints and suggestions.

By utilizing an evolutionary algorithm we iteratively improve the expert design which is then behavior cloned to implicitly embed prior knowledge into the agent.

Results

We utilize DDPG and TD3 as the learning algorithm and performed experiments on the OpenManipulator Robot. The experiments were performed on 7 different approaches comprising of ablations and prior RL-based PID tuning approaches.

Our approach had the lowest outliers, and deviation, signifying consistent performance, and had 75% improvement over ZN gains, 35% improvement over manually tuned gains, and 10% improvement over RL-Box  approach.

A brief video comparing ZN-DOB and our approach is shown below: