Material recognition and location system with cloud programmable logic controller based on deep learning in 5G environment

Intellectualization and unmanned manufacturing have been an inevitable trend in industrial development. The landing of intelligent applications is one of the current challenges in the industry. Due to the hierarchical architecture of the industrial automation pyramid, traditional programmable logic controllers (PLCs) that are usually employed in the field cannot cooperate with artificial intelligence (AI) algorithms that require massive data and computing resources. Therefore, it is necessary to research the virtualization of traditional PLCs as dockers, which can be deployed in the cloud, edge, or field. Cloud PLCs can be easily integrated with AI, big data, and cloud computing to achieve intelligent decision-making and control and break down data islands. The visual sorting system has attracted increasing attention for its ability to accurately detect the position of objects. Many deep learning–based methods have achieved remarkable performance in computer vision. Additionally, the requirement of a network is fundamental for guaranteeing data transmission with low latency and high reliability. The combination of 5G and time-sensitive networking (TSN) can achieve the deterministic transmission of several industrial applications. According to the above challenges, joint control between cloud PLCs of low-level devices and visual sorting systems in a reliable network is critical and has industry potential. In this study, we propose a deep learning–based material recognition and location system with a cloud PLC, which is demonstrated in a 5G-TSN network. First, traditional PLC is virtualized to realize flexible PLC function deployment in the field and cloud. Second, we establish a cloud-based AI platform and design a You only look once v5 (YOLOv5)-based object detection algorithm to locate the position and recognize the types of materials to obtain pixel coordinates. Third, the camera calibration method is used to transform pixel and world coordinates, and the material information consists of the world coordinates, types, and timestamps that are sent to cloud PLC. Finally, the commands are transmitted by the 5G-TSN environment from cloud PLC to the low-level devices for real-time control of the multi-crane cooperative. We establish an experimental system to demonstrate the significance and effectiveness of the proposed scheme, which synergistically controls multi-crane operation. The mean average precision (mAP) of material location is up to 99.65%, sorting accuracy reaches 96.67%, and the average consuming time is 25.99 s, which meets the requirements of low latency and high precision in industrial applications.