Surgical dexterity investigation and classification using deep learning on a virtual reality simulator

Abstract

Surgical dexterity is an essential criterion to evaluate candidates for surgical competency. Many factors may affect surgical dexterity but they were not studied in depth in previous works. There was a lack of evidence presented using objective measurements to identify factors that could potentially influence surgical dexterity. Hence, this thesis aims to investigate the correlation between various human factors and the manual dexterity of surgeons, with the aid of a 3D virtual reality simulator and objective measurements. A custom data acquisition module was developed, namely “Green Target Module”, to acquire positional data of hand movements from the subjects when controlling a cursor in a 3D virtual reality (VR) scene. The positional data were recorded and extracted into seven objective parameters, which were endpoint accuracy, endpoint precision, motion path length, economy of movement, motion smoothness, motion path accuracy and motion path precision. Body posture, visual magnification and handedness were investigated to identify the setups that resulted in better performance. In addition, a questionnaire was filled by all subjects to collect their background information and habits, such as specialty, years of experience, sleeping duration, coffee intake and video games ability, in order to investigate how these human factors affect the surgical dexterity. A total of 34 subj ects from different surgical backgrounds were recruited for the experiments. All subjects performed better with sitting posture, 10x visual magnification and when using the dominant hand. No significant differences were found across groups with different daily sleeping hours. In terms of specialty, oral and maxillofacial surgeons recorded significantly longer path length and lower economy of movement, motion path accuracy and precision compared to ophthalmology surgeons, obstetrics and gynaecology surgeons, and neurosurgeons. However, they performed smoother motions compared to ophthalmology surgeons, obstetrics and gynaecology surgeons, and general surgeons. In terms of experience, surgeons with 6 to 10 years of experience performed shorter motion path length and better economy of movement than those with less than 6 years and more than 10 years of experience. Interestingly, surgeons who had less than 11 years of experience performed better in motion path accuracy, motion path precision, motion smoothness and endpoint accuracy compared to surgeons who had more or equal to 11 years of experience. For coffee consumption, surgeons with daily coffee intake of less than 1 cup performed significantly smoother path, higher motion path accuracy and precision compared to those who consumed more. Surgeons with exposure to video games recorded shorter path length and better economy of movements, endpoint accuracy and precision compared to those without. Finally, deep learning based on convolutional neural network was used to classify the category of factors related to human dexterity. The highest average accuracy and weighted F1-score for classifying specialty, year of experience, daily sleeping hours, daily coffee consumption, and video game exposure were (97.29%, 94.25%), (90.04%, 85.18%), (90.37%, 90.3%), (90.97%, 84.6%) and (92.9%, 92.65%). In conclusion, surgical dexterity has been investigated and classified using deep learning on a 3D virtual reality simulator.