Objective In order to provide decision-making basis for subsequent litchi flower thinning, fruit retention and precise fertilization application, this work evaluated the flowering rate of each litchi by analyzing UAV remote sensing images of litchi canopy.Method The remote sensing image of each litchi canopy was segmented through instance segmentation algorithm. The flowering rates were classified into four categories combining with comprehensive judgment of horticultural experts, which were 0, 10%?20%, 50%?60%, 80% and above. ResNet50, ResNeXt50 and ShuffleNetv2 were adopted to compare flowering rate classification. Due to the great advantages in recognition accuracy, number of parameters, training and verification time, ShuffleNetv2 was adopted as the instance segmentation algorithm, and optimized by introducing the spatial attention module (SAM) to increase the model’s learning of location information, and improve the accuracy of litchi canopy flowering classification without significantly increasing the number of parameters. Result Through comparison of the mainstream deep learning algorithms, the classification accuracy of ResNet50, ResNeXt50 and ShuffleNetv2 reached 85.96%, 87.01% and 86.84% respectively, and the improved ShuffleNetv2 reached 88.60%, higher than the above three algorithms. The verification time of single canopy image on test set using ResNet50, ResNeXt50, ShuffleNetv2 and the improved ShuffleNetv2 were 8.802, 9.116, 7.529 and 7.507 ms respectively, showing that the improved ShuffleNetv2 single canopy image got the shortest verification time. Conclusion The improved ShuffleNetv2 can excavate and learn more detailed flowering information of litchi canopy, with high recognition accuracy and great advantages in the evaluation of litchi flowering, providing an intelligent decision support for flower protection and sparseness, and precise control of production.