Sequential Pattern Learning of Visual Features and Operation Cycles for Vision-Based Action Recognition of Earthmoving Excavators

Abstract

Excavator action recognition is a major task for cycle-time and productivity analysis. Many researchers have developed vision-based methods and showed promising results for automated action recognition. However, previous methods did not fully incorporate sequential working patterns of excavators although it is crucial to explain their natural operation procedure. To overcome the limitations, this paper proposes a vision-based excavator action recognition framework that considers the sequential patterns of visual features and operation cycles. The framework was validated with 1,340 images collected from actual earthmoving sites, and the average precisions and recall rates were 90.9% and 89.2% respectively. This research performed additional experiments with the other CNN model, and the precision and recall rate were decreased to 77.5% and 71.6% respectively. The experimental results showed the applicability of the developed framework and the positive impacts of sequential pattern learning. This study can contribute to more reliable automated cycle-time analysis and productivity measurement.