Abstract: Factors like a poor generalization ability and overfitting in the training process of the traditional optical flow method cause a low recognition rate of micro-expression. In the feature extraction stage, the convolution network is trained independently and serially at each pyramid level according to the idea of residual structure, and an optical flow vector expression of micro-expression images is refined step by step by combining attention mechanism so as to construct a pyramid optical flow model based on key frames. The final feature representation is obtained by cascading fusion with local binary patterns from three orthogonal planes (LBP-TOP) features, which effectively extracts the spatio-temporal texture features and optical strain information of video sequences. In the aspect of classification model, when the deep learning recognition model is applied to micro-expression classification, convolutional neural network (CNN) cannot achieve a close association between the key facial regions and the corresponding emotional tag vectors, which leads to poor classification performance. Therefore, this paper proposes a transfer learning network framework for micro-expression cross datasets based on CNN and graph convolutional network (GCN) to analyze the concealed relationship between image fusion features and tag vectors, which uses the quantitative advantages of macro-expression to assist micro-expression recognition. Experiments are carried out on two micro-expression databases, CASME II and SAMM, to realize the classification of four kinds of emotion data, and the recognition rate rises from 57.56% to 75.93%.