Digital Twin Framework for Virtual Visuo‑Haptic Teleoperation

Abstract

Optical tweezers empower piconewton‑scale micromanipulation for delicate biomedical tasks, yet most existing visuo‑haptic frameworks are tailored to simple shapes and offer limited force awareness. This work introduces a comprehensive digital twin that unifies a simulated microrobot environment, learning‑based pose and depth estimation, motion simulation and model‑based haptic rendering into a single platform. By leveraging a ROS‑connected bimanual interface with dual haptic devices, the framework reproduces representative motion trends and renders consistent force feedback. In simulated cell‑delivery experiments the visuo‑haptic interface reduced fluctuations in contact force and robot‑trap distance by over 50 percent and boosted task success from 30 percent to 80 percent.

Key Contributions

• A digital twin‑driven visuo‑haptic teleoperation framework for complex‑shaped optical microrobots that unifies bimanual interaction, 3D visual reconstruction and model‑based haptic rendering within a common simulation environment.
• A learning‑based pose and depth estimation module enabling deployment‑oriented alignment between physical optical‑tweezers observations and digital‑twin reconstruction of the microrobot and trap configuration.
• Quantitative evaluations demonstrating that the digital twin reproduces out‑of‑plane rotation trends observed in reference experiments with root‑mean‑square errors of 3.31° and 1.54° under two control strategies.
• Numerical verification that the rendered haptic force matches the fitted optical‑force model with sub‑centi‑Newton errors (RMSE ≈ 0.0074 along the axial direction).
• User studies showing that model‑based haptic feedback reduces contact forces and positional deviations by approximately 53 % and 55 % and improves task success from 30 % to 80 %.

Framework Overview

Fig. 1 overview of the proposed digital twin framework. The pipeline integrates a user interface with dual haptic devices, a reference optical‑tweezers setup supplying images and parameters, a learned pose/depth estimation module, an Isaac Sim digital‑twin environment and model‑based haptic rendering.

The bimanual interface sends commands through ROS to update trap configurations and microrobot state within the digital twin. Microscopic images from the physical setup can optionally be processed by a perception module to estimate microrobot pose and depth, enabling state alignment between real and virtual scenes. The reconstructed 3D environment delivers visual feedback on a monitor while the model‑based haptic module synthesizes force cues to the operator.

Experimental Results

Motion Control Fidelity

To evaluate whether the digital twin replicates reference microrobot behaviors, two control strategies from Zhang et al. were reproduced. Under Control Strategy A the characteristic trap spacing d* was varied and the resulting out‑of‑plane rotation angle was measured. The digital twin followed the experimentally observed trend, decreasing rotation with increasing spacing and achieving a root‑mean‑square error (RMSE) of 3.31° and a mean‑absolute error (MAE) of 2.71°. Under Control Strategy B, with a fixed power‑distribution parameter (m = 1.5), the simulated rotation profile matched the reference data with RMSE = 1.54° and MAE = 1.29°.

Consistency of Haptic Rendering

Numerical comparisons between the pre‑scaled haptic force and the fitted optical‑force model were carried out along representative axial and radial directions. The axial direction yielded an MSE of 5.46×10⁻⁵ and RMSE of 0.0074, while the radial direction produced an MSE of 2.10×10⁻⁴ and RMSE of 0.0145. These results indicate that the rendered force channel faithfully conveys model‑estimated interactions prior to device scaling.

User Evaluation

Five volunteers (aged 22–27) performed simulated cell‑delivery trials under two interface conditions: with and without haptic feedback. Each participant completed two trials per condition. The haptic interface reduced the mean contact force from 5.39 to 2.57 and its standard deviation from 6.78 to 3.18, and lowered the mean microrobot‑to‑trap distance from 0.04 to 0.02 and its standard deviation from 0.058 to 0.026. The proportion of successful trials increased from 30 % to 80 %, indicating that force cues improved interaction regulation and alignment.

Demonstration Video

Demonstration of the digital‑twin driven visuo‑haptic teleoperation interface (provided video).

Paper and Citation

The complete manuscript is available in the conference proceedings. Below is a suggested BibTeX entry for citing this work.

@inproceedings{tan2026digitaltwin,
  title={A Digital Twin Framework for Virtual Visuo-Haptic Teleoperation of Complex-Shaped Optical Microrobots},
  author={Tan, Zongcai and Wei, Lan and Zhang, Dandan},
  booktitle={MARSS},
  year={2026}
}