Electrocardiogram (ECG) signal is an important basis for the diagnosis of arrhythmia and myocardial infarction. In order to further improve the classification effect of arrhythmia and myocardial infarction, an ECG classification algorithm based on Convolutional vision Transformer (CvT) and multimodal image fusion was proposed. Through Gramian summation angular field (GASF), Gramian difference angular field (GADF) and recurrence plot (RP), the one-dimensional ECG signal was converted into three different modes of two-dimensional images, and fused into a multimodal fusion image containing more features. The CvT-13 model could take into account local and global information when processing the fused image, thus effectively improving the classification performance. On the MIT-BIH arrhythmia dataset and the PTB myocardial infarction dataset, the algorithm achieved a combined accuracy of 99.9% for the classification of five arrhythmias and 99.8% for the classification of myocardial infarction. The experiments show that the high-precision computer-assisted intelligent classification method is superior and can effectively improve the diagnostic efficiency of arrhythmia as well as myocardial infarction and other cardiac diseases.