Quantitatively analyzing hematoxylin & eosin (H&E) histopathology images is an emerging field attracting increasing attentions in recent years. This paper reviews the application of computer-aided image analysis in breast cancer prognosis. The traditional prognosis based on H&E histopathology image for breast cancer is firstly sketched, followed by a detailed description of the workflow of computer-aided prognosis including image acquisition, image preprocessing, regions of interest detection and object segmentation, feature extraction, and computer-aided prognosis. In the end, major technical challenges and future directions in this field are summarized.