Video-Based Fatigue Estimation for Human-Robot Task Allocation Optimisation

Human-centric manufacturing paradigm requires the human-robot collaboration (HRC) system to place the well-being of workers at the centre of manufacturing processes. Hence, optimising human workers' fatigue during human-robot task allocation in an HRC system is crucial for human-centric manufacturing. A prerequisite for this is an objective assessment of human fatigue. However, existing fatigue assessment relies on wearing clunky contact sensors, which are not user-friendly on an unstructured manufacturing shop floor. To address this limitation, we propose a video-based fatigue estimation method in which the boundary-aware dual-stream MS-TCN algorithm is proposed to detect operation type and operation repetitions from the video, and then the detected results are input into a fatigue model to estimate the worker's fatigue. In addition, the estimated human fatigue from the video is used as a basis for a human-robot task allocation optimisation model. The optimisation objective is to minimise cycle time while constraining human fatigue within an acceptable range. The experiment results show the validity of the fatigue estimation method, the superiority of the operation segmentation algorithm, and the effectiveness of the human-robot task allocation optimisation model.