Objective: A mathematical model of tumor growth and invasion under the radiotherapy was established to simulate the effect of radiotherapy on tumor growth and sensitivity analysis. Methods: A mathematical model of tumor growth and invasion with time evolution before and after radiotherapy was established. The model included four key variables in the process of tumor invasion: tumor cells, ECM, MDEs and oxygen. The linear quadratic (LQ) model was used to simulate the survival probability of tumor cells after radiotherapy, and the effects of different radiotherapy schemes and radiotherapy coefficients on the treatment effect were discussed. Traditional radiotherapy and intraoperative targeted radiotherapy were compared. Results: under the premise of constant total dose, the results of radiotherapy were directly proportional to the radiotherapy coefficient, but not related to the radiotherapy frequency; the therapeutic effect of intraoperative targeted radiotherapy was better than that of standard treatment. Conclusion: Simulation results are basically consistent with the clinical experimental results. As a more efficient treatment method, intraoperative targeted radiotherapy can provide a new idea for clinical tumor treatment.