Abstract: Aiming at the problems of high cost, low accuracy, and poor real-time performance in the monitoring of abnormal data in industrial production processes, a multimodal instrument data anomaly detection method is proposed for the digital twin factory. Firstly, a dynamic non-monotonic focus mechanism is introduced into the YOLOv5s object detection model to detect the position of pointer-type instruments by improving the model. Simultaneously, affine transformation and perspective transformation are utilized to calibrate images with skewness and rotation. Secondly, variable convolution is introduced into the ESPNet segmentation network to adaptively capture non-linear deformations of pointer and dial elements in the image, thereby extracting key information from the complete pointer and dial. Then, a Hough transform operation is applied to fit the line where the pointer is located, and contour tracking is used to extract the valid range of dial data. The pointer-type instrument reading is calculated based on linear proportional relationships. Finally, instrument data obtained from images and corresponding sensor data are comprehensively modeled, and anomalies are detected from the perspective of multimodal data analysis. Experimental results demonstrate an accuracy of 96.21% in pointer-type instrument recognition, an average detection speed of 0.216 s, and a data anomaly detection rate of 99.43%. This method can accurately and rapidly identify data anomalies to meet the requirements of critical indicator monitoring in industrial production.