Robotic VR System to Teleoperate Robots for Performing COVID-19 Swab Tests

A research team co-led by City University of Hong Kong (CityU Hong Kong) has developed an innovative HMI system, which consists of flexible, multi-layered electronic skin and provides both visual and haptic feedback to users. The system can teleoperate the robot to imitate the user's actions to perform complicated tasks. It demonstrates the potential for conducting COVID-19 swab tests and nursing patients with infectious diseases.

The HMI system links users to robotics or computers and plays a significant role in teleoperating robotics. However, conventional HMIs are based on bulky, rigid and expensive machines, and the lack of adequate feedback to users limits their application for conducting complicated tasks. The closed-loop HMI system is based on soft, skin-integrated electronics, which enable wireless motion capture and haptic feedback via Bluetooth, Wi-Fi and the internet.

The advanced HMI system, named Robotic VR, is an integration of visual, auditory and haptic VR via skin-integrated electronics. It uses a layout of multilayer stacking, where the bottom layer of skin-tone elastomeric silicon serves as a soft adhesive interface, which can be mounted on the skin and joints of the user. The layers are interconnected with a collection of chip-scale integrated circuits and sensing components, including resistors, capacitors, a Bluetooth module, a microcontroller unit (MCU), and soft sensors and actuators developed by the team.

The sensors of the robotic VR system can accurately detect and convert subtle human motion into electrical signals, which are processed by the MCU and wirelessly transmitted to the target robot. In this way, the user can teleoperate the robot to imitate his motion to accomplish tasks remotely. The pressure sensors on the robot can send feedback signals to control the vibration intensity of the haptic actuators through the Bluetooth module, thus providing haptic feedback to the user. The user can then precisely control and adjust the motion of the robot’s arm according to the intensity of the feedback.

The HMI system supports three wireless transmission methods – Bluetooth (up to tens of meters), Wi-Fi (up to about 100 meters), and the Internet (worldwide) – which can be adjusted according to the practical applications. The new system is stretchable and can be tightly mounted on human skin and even the whole human body for a long time. In addition, the interface provides both haptic and visual feedback systems, providing an immersive experience for users. The HMIs could teleoperate various machines. With the advanced circuit design and outstanding mechanical characteristics, the newly invented HMI system has great potential for applications in commercial and biomedical engineering fields. For example, it can be used to remotely drive unmanned cars. A user with a hand disability can remotely manipulate a robot to carry heavy goods using the HMI system. The sensors attached to the user can monitor and transmit muscle signals to control the robot.

On the biomedical engineering side, doctors can remotely control robots to conduct surgery by wearing the HMI system with VR glasses. And medical workers can remotely manipulate the robot to look after infectious patients or collecting bio-samples, thus greatly decreasing the infection risk. The team conducted experiments to demonstrate the system's potential applications, such as remotely controlling the robotic hand to collect throat swab samples for COVID-19 tests, and to teleoperate humanoid robots to clean a room and even provide patient care. The team is developing a next-generation system to provide the robotic collection of swab tests. The researchers expect the new-generation HMI system to help provide a new approach in wirelessly connecting people to a robot or virtual character in the metaverse.

“This new-generation flexible human-machine interface system enables teleoperating robotics for conducting complicated tasks,” said Dr. Yu Xinge, Associate Professor in the Department of Biomedical Engineering (BME) at CityU.

Related Links:
City University of Hong Kong