Abstract: This paper presents a new method of medical assistant diagnosis for medical field. This method combines the image information of patients (such as CT image) and numerical data (such as clinical diagnosis information), and uses the combined information to automatically predict the patient's condition. Based on this method, a medical assistant diagnosis model based on deep learning is proposed. The model takes image and numerical data as input, and outputs the patient's condition. Therefore, this method can use more comprehensive information, help to improve the accuracy of automatic diagnosis and reduce the diagnosis error; in addition, using the proposed model can process multiple types of data at one time, which can save the diagnosis time to a certain extent. The effectiveness of the proposed method is verified on two datasets. The experimental results show that the method is effective and can improve the accuracy of auxiliary diagnosis.