Sim2Real Unitree G1 Locomotion with RL

During my Summer 2025 research at Washington University in St. Louis, my main goal was to improve the documentation for Unitree's updated unitree_rl_lab pipeline. The update had recently moved the repository from the deprecated Isaac Gym to Isaac Lab, and the instructions did not cover several steps required to train, validate, and deploy a G1 walking policy. I tested each stage, recorded the errors and corrections, and prepared a reproducible workflow for future lab members.

Research goal and reproducibility

The lab was already working within an established research question: can the lab train a walking policy, challenge it in a second simulator, and deploy it to the Unitree G1?

My contribution focused on reproducibility. I checked whether the newly released Isaac Lab workflow contained enough information for another researcher to repeat the three stages. At each stage, I compared the repository instructions with the behavior I observed, documented missing steps, issues, how I solved them, and verified the corrections before continuing.

Project videos

These recordings show the walking policy moving through training, validation in two simulators, and hardware deployment.

What the literature contributed

The project followed the reasoning behind Humanoid-Gym, which presents a train-in-simulation, validation in MuJoCo, then transfer to hardware workflow. The paper explains why the middle step matters: agreement across two physics engines is a stronger robustness check than replaying a policy only where it was trained. It also describes several ideas that shaped my understanding of the pipeline:

• PPO and privileged learning: actor learns from deployable observations while the critic can use additional simulator-only information during training.
• Partial observability: the (physical) robot never exposes the complete simulator state, so the policy must act from a limited observation history.
• Domain randomization: variations in friction, mass, motor strength, delay, and sensor signals make a policy less dependent on one perfectly tuned virtual robot.
• Layered control: the paper’s learned policy runs at 100 Hz and produces joint targets for a 1,000 Hz PD controller. The policy generates targets, and the PD controller stabilizes the physical response at a higher frequency.
• Reward design: velocity tracking, gait phase, smooth actions, stable contacts, posture, and energy use must be balanced. These terms define the walking behavior learned during training.

The broader survey Humanoid Locomotion and Manipulation: Current Progress and Challenges in Control, Planning, and Learning placed this work in a greater context. It compares model-based control, reinforcement learning, imitation learning, and emerging foundation-model approaches. For this project, its most relevant conclusion was that simulation-trained RL can produce robust locomotion, but humanoids make the sim-to-real gap especially difficult because of their high dimensionality and intrinsically unstable dynamics. It also reinforced that learned and model-based control are complementary: the policy can generate joint targets while a lower-level controller enforces fast physical response.

There were many discussions I had with my PhD supervisors in terms of how we can structure a RL policy: a master policy that controls small policies, the existence of a curriculum in which slowly introduces new policies (or increased difficulty), hierarchical, and other ideas.

Testing the pipeline

1. Establishing the training baseline

I first trained the native G1 velocity tasks in Isaac Lab. The flat-terrain policy converged to a useful walking behavior in roughly two hours, while a rough-terrain run took about one hour and forty minutes. This was a bit surprising given that a rough terrain has a more complex walking motion in order to successfully learn how to walk. The mysteries of RL training I guess. Then, I replayed both policies in simulation and located the exported PyTorch and ONNX policies needed for the next stage.

These runs confirmed that the GPU, robot asset, RL task, and exporter worked together. Training took a few hours. The integration work between Isaac Lab, MuJoCo, the controller, and the robot took most of the project time.

2. Making Sim2Sim validation work

My first deployment attempt used an older Unitree repository built around the deprecated Isaac Gym stack. The MuJoCo controller expected a 47-element observation, while the Isaac Lab policy expected 123 inputs. I stopped this test because the observation schemas were incompatible and continued with Unitree’s updated Isaac Lab workflow.

The updated workflow exposed several documentation gaps. Missing Git LFS assets left the G1 model without rigid bodies or contact sensors. The repositories gave conflicting library installation locations. The tested simulator version was MuJoCo 3.2.7. The configuration used enable_elastic_band, and the 29-DoF scene had to be selected explicitly. I traced each error, corrected the environment, and rebuilt the controller.

Unitree’s Bluetooth remotes paired without creating a usable Linux joystick device, and the connections dropped unpredictably. I tested the Bluetooth configuration, input services, and alternative mappings. A wired Xbox-compatible controller appeared immediately as a joystick device and allowed the MuJoCo test to proceed. I added two checks to the procedure: use a wired controller and confirm its device index before starting the simulator.

3. Moving from simulation to the G1

The first real-robot command failed because the repository’s example network name did not exist on the workstation. The wireless interface did not receive the expected low-level state stream. Despite both workstation and Unitree G1 were connected to the same network and ROS topics were accessible. I switched to a direct Ethernet connection, configured the workstation on the robot’s subnet, selected the actual wired interface, and established communication with the G1.

From the Go1 connection guide, I followed the same setup flow as for the G1:

1. Connect the Ethernet cable from the PC to the robot.
2. Run the following commands on the PC:

sudo ifconfig enp2s0 down
sudo ifconfig enp2s0 192.168.123.XX/24
sudo ifconfig enp2s0 up

I consolidated the hardware sequence into a lab-specific deployment guide: verify the robot model, begin in zero-torque or damping mode, raise the robot while it is supported, establish ground contact, activate the policy, and keep both the physical and software emergency stops ready. The procedure also monitors the low-level state because the model identifier in the outgoing commands must match the robot’s reported mode.

Setbacks and how I addressed them

A second policy experiment

After the baseline pipeline worked, I began testing a slightly more robust walking policy with external disturbances. This experiment was initial attempt to see how to model these forces on the training stage in a curriculum approach.

With Zaid Ahmed, I traced the real-time IMU fields available to the C++ deployment layer and added a base-linear-motion term. The first test produced erratic behavior. That failure was informative: the training configuration placed the term in one location, while the deployment code inserted it elsewhere, and the available accelerometer signal did not have the same physical meaning as base velocity. I corrected the ordering and launched a roughly 15-hour retraining run.

This test gave me the clearest technical lesson of the project. The network will return an action for any numeric vector with the expected shape. It does not check whether an entry has the correct physical meaning. This is another topic discussed with my peers. Although the policy heavily punishes sudden and fast changes in articulations, it does not eliminate the possibility of that happening when testing the policy in simulation or in real life. Additional safeguards must be built on top of the output of the policy.

Deployment speed and reliability

Policy inference was somewhat fast for just the walking scenarios. However, for more complex goals, and multi-purpose policies, this showed to be very slow. The preparation before inference caused most of the delay: resolving software versions, locating exported checkpoints, rebuilding controllers, pairing input devices, selecting the correct interface, and moving the robot through safe startup states.

The most effective improvements were therefore operational:

• Freeze versions: keep known-good Isaac Lab, MuJoCo, compiler, and SDK versions.
• An explicit policy path: store exported policies in one stable location.
• Use wired controllers: verify the joystick index before launch.
• Use direct Ethernet: connect the workstation to the robot for policy control.
• Check the robot state: verify the robot model and low-level state before sending commands.
• Keep the stop procedure visible: review the emergency-stop sequence before every hardware test.

Once those decisions were documented, deployment changed from exploratory debugging into a short, repeatable checklist.

My contributions to the lab

I reduced the number of undocumented decisions between “train” and “walk.” The next lab member can begin with a tested sequence of versions, commands, checks, and fixes.

What I learned about reinforcement learning

• Policies learn interfaces as well as behaviors: Observation order, scale, history, and physical meaning must remain identical across training and deployment.
• PPO fits parallel simulation well: It may be sample-inefficient, but thousands of simulated environments make data comparatively cheap and policy updates scalable. It works well if your workstation is able to process that. However, I believe that better cost-efficient alternatives should be more prevalent.
• The reward is the specification: Stable posture, contact timing, smooth actions, energy penalties, and velocity tracking shape the gait that emerges.
• Privileged information belongs to training: An asymmetric actor-critic can give the critic complete simulated state while keeping the actor limited to observations available on the robot.
• Domain randomization targets robustness: Varying dynamics and adding disturbances discourages dependence on one exact simulator, although the ranges still require careful tuning. It’s definitely recommended to study this topic more (and I’m planning on doing that).
• Sim2Sim is a risk-reduction tool: Successful execution in a second physics engine shows that the policy and deployment code are compatible outside the training simulator. If setup quickly, it’s a step that might stop someone from deploying a bad policy in little time.
• State estimation is part of RL deployment: Sensors and estimators must reproduce the state variables used during training; a raw sensor channel may represent a different physical quantity.
• Safety surrounds the learning algorithm: Damping modes, supported startup, model checks, low-level monitoring, and emergency stops are part of a successful real-robot experiment. Transitions between policy and hard-coded control should always have guardrails.

Selected references

X. Gu, Y.-J. Wang, and J. Chen, Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfer, 2024.
Z. Gu et al., Humanoid Locomotion and Manipulation: Current Progress and Challenges in Control, Planning, and Learning, 2025.