Medical image registration is very challenging due to the various imaging modality, image quality, wide inter-patients variability, and intra-patient variability with disease progressing of medical images, with strict requirement for robustness. Inspired by semantic model, especially the recent tremendous progress in computer vision tasks under bag-of-visual-word framework, we set up a novel semantic model to match medical images. Since most of medical images have poor contrast, small dynamic range, and involving only intensities and so on, the traditional visual word models do not perform very well. To benefit from the advantages from the relative works, we proposed a novel visual word model named directional visual words, which performs better on medical images. Then we applied this model to do medical registration. In our experiment, the critical anatomical structures were first manually specified by experts. Then we adopted the directional visual word, the strategy of spatial pyramid searching from coarse to fine, and the k-means algorithm to help us locating the positions of the key structures accurately. Sequentially, we shall register corresponding images by the areas around these positions. The results of the experiments which were performed on real cardiac images showed that our method could achieve high registration accuracy in some specific areas.