Ant Group Open-Sources LingBot-Touch to Give Robots a Shared Sense of Touch

Robbyant, the embodied AI company within Ant Group, has open-sourced LingBot-Touch, a tactile foundation model designed to give robots a touch sense that works across different sensors and different robotic hands. The company announced the release on Nov. 10, and the model, code and technical report are available on Hugging Face and GitHub. 

The same physical contact can produce significantly different raw signals depending on the sensor used. To enable learning of such physical contacts in a sensor-agnostic manner, three common sensing principles for tactile perception are supported: magnetic tactile sensing, visual tactile sensing and piezoresistive sensing. All of these sensing principles are mapped to a shared fingertip space, and contact locations and force distributions are modeled over time. In order to enable action models to process such tactile tokens in a sensor-agnostic manner, different encoding pathways are defined for each of the aforementioned sensing principles. 

The pre-training dataset is of almost 5,000 hours of real-world tactile interaction data from numerous sources can include interaction from human wearing tactile-gloves, and human-operated teleoperated dexterous-hand(s). LingBot-Touch specifically targets the class of fine manipulation tasks, e.g. turning pages of a book, unscrewing a bottle cap, and recognition of a Mahjong tile. 

Robbyant used 26 real world manipulation tests, using 2 dexterous hands (Sharpa Wave, XHand) and 4 action models (LingBot-VLA, π0.5, Diffusion Policy, T-Rex). cross the four action models, the average success rate rose by about 14 to 26 percentage points when the touch layer was used, compared with using no tactile input. In one representative test, using T-Rex with Sharpa Wave to unscrew a bottle cap, the success rate was 56.25 percent without touch, 62.5 percent with raw sensor signals and 90 percent with the unified tactile representation. 

Robbyant further showed transfer to new hardware, such as the magnetic sensor on XHand not being included in the tactile data used for pre-training. All 26 test tasks were performed using a lite version of the touch layer, with approximately 5.3M parameters, adding between 0.13 percent and 0.16 percent to the size of the downstream action models it supports. 

Note that the reported results are all from internal tests by Robbyant. The announcement does not say how many trials were performed for each of the individual tasks and hence will have to await results from independent reproduction and test on other hands or sensors.