Computer vision system for industrial screwing automation

Abstract

This project proposes a software that incorporates computer vision algorithms to detect screw types, screw locations, and to locate screw holes on an object to ensure a smooth flow of automated assembly processes. The existing systems are found to be less adaptable for performing automated assembly and do not satisfy real-time constraints. These systems are affected by several factors that exist in the industrial environment such as lighting conditions and calibration issues which affect the effectiveness of the automation. This encouraged to develop an adaptable system, which is adaptable to variation in object locations, lighting conditions and works in real-time constraints. This achieved by developing two subsystems, where firstly, a camera is mounted above a screw tray to detect screws by using You Only Look Once version 3 (YOLO v3) detection algorithm with Darknet. YOLO v3 is trained on a collected dataset and validated using two approaches: train/test split and 3-fold cross validation. Secondly, another camera is mounted above an object to localize screw holes on the object by using a blob detector technique. A graphical user interface is designed to show the results and to make the system more user-friendly and easy to monitor. Experimental results show that the screw detection subsystem is able to detect the screws under different lighting conditions with mAP of 93.8% and localization accuracy with a maximum error of 1.26% in the x-axis and 2.84% in the y-axis. Also, the blob detector subsystem is able to localize the screw holes with a maximum error of 0.26% in the x-axis and 0.58% in the y-axis. Besides that, both subsystems are able to work in real-time constraints with a speed of 7-10 FPS. It is envisaged that the computer vision software will make the assembly process more effective and increase productivity, also enhance the flow of the process.