2025; Master's Thesis: Part 4 of 4: Differentiable Simulations

Introduction

This is part four of the four part series covering my Master's Thesis at KAIST titled "Accelerating Policy Learning for Robust Control of Robotic Manipulators and Aerial Vehicles via Physics-Informed Guidance" under Professor Dong Eui Chang. The full thesis can be found at KAIST library: THESIS. It has in-depth explanation of all the concepts, simulation and control design, and the experimental results. Please note that these posts do not cover the entire thesis in detail and is instead built upon my thesis defence slides.

This blog post partially covers the differentiable simulation based manipulator auto-tuning design from the thesis.

Going beyond gradient estimation

In RL (here specifically actor-critic algorithms):

• Actor performs action given the state of the system,

• Environment shares reward with the critic; RL wants to maximize cumulative reward,

• Critic estimates the value of actions, which is used to approximate the actor’s gradient.

Differentiable simulations for policy learning can be viewed as an extension:

• Actor performs actions given the state of the system,

• Environment shares reward; we want to maximize cumulative reward/minimize total loss,

• We just differentiate the reward/loss with respect to actor to obtain the analytic gradient.

Note that differentiable simulation for policy optimization extends the concepts from model-based techniques by backpropagating through the high-fidelity simulation directly rather than probabilistic estimation approaches.

Simulation Modelling

To enable gradient flow from the optimization signal back to the controller parameters, the entire control loop must be differentiable. We implement the simulation environment for the manipulator and ship motion using NVIDIA Newton, a high-performance differentiable simulation engine. The manipulator is simulated using a modified Featherstone algorithm with Semi-implicit Euler integration, while the sea-state simulation is written as NVIDIA Warp kernels to allow backpropagation through disturbance. FBPID controller and NDOB (from part 2) are reutilized and reimplemented as Warp kernels, but with zero integral gain. The modified Newton simulation is still being used for other research work and hence is not public currently.

We utilize differentiable sim to optimize both the gain tuning agent (π_ν) and the NDOB gain matrix (L). To ensure positive NDOB gains, we reparametrize the matrix with learnable parameter Γ as

The loss function to minimize is constructed as

Truncated BPTT

Differentiable simulations utilize backpropagation through time to compute the analytic gradients from all time steps, all the way back to initial step. However, due to the long horizon of length N, the gradients can explode or vanish over time. Since the PID gain tuning agent acts every H steps for N/H blocks, we truncate the loss into short horizons by stopping the computation graph every H steps. The final parameter gradients are computed as

and the parameters are updated using gradient descent  ν←ν-η∇_ν L,   Γ←Γ-η∇_Γ L.

Training and Simulation

With differentiable simulation based approach, the loss decreases rapidly compared to standard RL training within only 100 steps with stable training and high sample efficiency. This took about 30 minutes to train on a standard personal computer. We found the average PID gains for each joint to converge to physically meaningful values over time. We utilize IsaacSim for the training and evaluation rendering.