Learning to plug in a network cable with imitation learning

A low-cost SO-101 arm learns a contact-rich precision task end to end from camera images: pick up an SFP+ direct-attach cable and push it into the second port of a network card until it latches. The port leaves only a few tenths of a millimetre of play, while the arm itself cannot position its gripper better than a few millimetres. The policy is ACT (Action Chunking with Transformers), trained on my own teleoperated demonstrations and improved with human corrections (DAgger). The arm and cameras run on a small server without a usable GPU. The policy runs on a desktop GPU and talks to the robot over the network.

Video 1: Several autonomous insertions in a row as the policy sees them (side, close-up of the port, wrist camera), 2× speed. Between attempts I put the cable back by hand. In the second attempt I shift the connector inside the gripper on purpose, and the policy has to recover several times before it gets it in.

Why this is hard

An SFP+ module is 13.55 ± 0.25 mm wide and 8.55 ± 0.15 mm high. The cage it slides into is 14.00 ± 0.10 mm wide and 8.95 ± 0.15 mm high. That leaves about 0.4 mm of play in total, roughly 0.2 mm per side, and in the worst case of the tolerances only 0.1 mm. Once aligned, the module has to be pushed in with up to 18 N until it latches.

The SO-101 is a hobby arm made of 3D-printed parts and Feetech STS3215 servos. The servo encoders resolve 0.088°, but the gearboxes do not hold that precision:

• A dial-indicator test measured about 0.9° of backlash per servo, against ≤ 0.5° in the datasheet.¹
• The servo firmware ignores position errors below 10 encoder counts, another 0.9°.
• Between shoulder and gripper are 20 to 30 cm, so a single degree in one joint already moves the gripper 3.5 to 5 mm.
• My follower runs its 7.4 V servos on a 5.4 V supply and sags 15 to 25° below its target under load.

The arm’s positioning error is therefore an order of magnitude larger than the clearance of the port. Moving to a taught pose cannot solve the task. The loop has to be closed visually: the policy watches the module and the port in a close-up camera, corrects in small steps, and uses contact with the edges of the cage to guide the module in.

Setup

• Robot: SO-101 follower and leader arm (LeRobot), STS3215 servos, parallel gripper. The leader arm is used for teleoperation and for corrections.
• Cameras: a side view, a wrist camera, and a close-up of the port (side_nic). The close-up turned out to be by far the most important input (see below).
• Compute: the robot server streams camera images and joint state to a PC with an RTX 4070 Ti SUPER. ACT needs about 13 ms per action chunk there, compared to about 630 ms on the server’s CPU.

Data

Over about a week I recorded about 2.5 hours of data: 166 demonstrations (83 min) and 142 DAgger corrections (65 min). Every episode is rated and trimmed in a small browser-based episode editor I wrote for this project (rating 1 to 5, start and end cut, notes). The cleaned datasets are then exported in LeRobot format, and only well-rated episodes make it into training: 147 of the 166 demonstrations (64 min after trimming) and 113 of the 142 corrections (30 min). The corrections shrink much more than the demonstrations, because cutting them down to the actual recovery and correction removes a lot of idle time and moving back home.

How ACT works

ACT (Action Chunking with Transformers) was introduced by Zhao et al.² for exactly this kind of problem: precise manipulation with cheap, imprecise arms, learned from a few dozen demonstrations. Two ideas make it work:

• Action chunking. Instead of one action per step, the policy predicts the next k actions at once (here k = 100, 3.3 s). A task of 1000 steps then needs only about 20 decisions instead of 1000, so small errors compound much less. Chunks also capture motion that is hard to explain from a single frame, such as pausing before a push.
• A conditional VAE. Human demonstrations are not consistent: the same situation is solved slightly differently each time. During training, a second encoder looks at the recorded action chunk and compresses its “style” into a latent variable z. The policy learns to produce the chunk given the observation and z. At test time z is set to zero (the mean style), so the policy commits to one consistent motion instead of averaging between demonstrations.

The policy itself is a transformer. Each camera image goes through a ResNet18, and every cell of the resulting feature map becomes a token. Together with the joint state and z, the encoder sees 902 tokens per step. The decoder has one learned query per future time step and outputs the 100 joint targets in a single pass. The model has 52 M parameters.

Fig. 1: ACT as configured here (LeRobot implementation). Dashed: only used during training.

Training

The state input is the six joint positions plus the six joint loads reported by the servos, the output the six target joint angles of the leader arm.

• First training from scratch: 100k steps, batch 8, about 5 hours on the GPU.
• After each DAgger round: fine-tuning on demonstrations plus corrections for 40k steps.

Fig. 2: Port camera during an autonomous run: approaching the port, cable latched

DAgger: teaching recovery

Behaviour cloning only sees successful demonstrations. Once the policy drifts into a state it never saw (too high, slightly off to the side, jammed), errors compound. With DAgger, the policy runs and I take over only when it is about to fail. The correction is recorded and added to the training data.

I extended LeRobot’s DAgger loop so it can be run single-handed while holding the leader arm:

• One key: space pauses the policy and drives the leader arm to the follower’s pose. The correction starts automatically 3 s later. Space again ends it and hands control back to the policy.
• Rewind: while paused, the arrow keys move both arms back along the policy’s last 10 s, so a correction can start from where things began to go wrong (similar to the rewind-and-refine idea of Genie Centurion).
• Following RaC, each correction is a recovery back to a familiar state plus a correction of the current subtask, saved as its own episode.

Video 2: DAgger from the outside, 2× speed: the gripper misses the cable while grasping, so I take over with the leader arm

Things I learned

The network matters more than expected. Over Wi-Fi, one raw observation (three images, 2.8 MB) took about 330 ms. Every action chunk was therefore planned for a pose the arm had already left, and the arm sometimes froze and jumped. A direct gigabit cable brought this down to 24 ms, and the motion became much smoother without any code change.

Re-planning every step is not always better. The ACT paper re-plans at every step and averages overlapping chunks (temporal ensembling). On this arm that was clearly worse: slower, stop-and-go motion. The ALOHA arms in the paper follow their targets almost exactly, so the measured pose matches the plan and starting a new plan from it costs nothing. My follower runs 7.4 V servos on 5.4 V and lags its target. That gap between target and actual position is exactly what produces the insertion force, and frequent re-planning from the measured pose eats it away. Executing each chunk for about 50 steps worked best.

Fig. 3: One recorded run per variant from the same start position. Top: change of the commanded joint targets per control tick (hover for values). Default blending with long segments (1) is smoothest and fastest. Per-step temporal ensembling (4) lags and shows large spikes. Between 1 to 3 the differences are small for single runs.

The policy uses force, and mostly one camera. I measured how much the predicted actions change when single inputs are altered (400 frames, mean over the 100-step chunk):

Input altered	Shoulder / elbow, approach and insertion
Joint loads replaced	5 to 6°
Side camera blanked	2 to 3°
Wrist camera blanked	7 to 10°
Port close-up blanked	16 to 21°

The loads influence the shoulder and elbow, the joints that push the cable in, about as much as the policy’s own error. The close-up of the port dominates everything else.

Limitations

• Lighting. The policy is sensitive to changes in light. Over the day, sunlight from the window changes brightness, shadows and reflections on the table, and runs in different light were noticeably worse. Brightness and contrast augmentation during training does not cover changing shadows and reflections. More varied recording conditions or controlled lighting would be needed.
• Teleoperation and corrections with the SO-101 are hard. The leader arm uses the same hobby servos, so its readings jump and drift, and the follower does not follow it reliably: backlash, dead zones and the under-voltage lag add up to millimetres. Precise, repeatable teleoperation is not possible with this hardware, which limits both the quality of the demonstrations and of the DAgger corrections. Better servos or a stiffer arm would help more than more data.
• Occasional stalls during the final push, where the policy makes smaller and smaller corrections. ACT sees only the current frame, so it cannot tell “pushing and sliding in” from “pushing and jammed”.

Status

The policy grasps and inserts the cable reliably in most runs, and the DAgger rounds made it noticeably better at recovering from a bad grasp or a misaligned approach. Next steps are more targeted corrections in jammed states and a cleaner comparison of the input features.

Footnotes

1. Robonine, Testing of Feetech STS3215 servomotor: backlash, repeatability and torque: 1.3 mm play at the tip of an 86 mm lever, a firmware dead zone of 10 counts. ↩

2. Tony Z. Zhao, Vikash Kumar, Sergey Levine, Chelsea Finn. Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware. Robotics: Science and Systems (RSS), 2023. arXiv:2304.13705 ↩