In view of impulsion duration shortens when classical antennas transmits high power microwave, quasi-optical mode converter is optimized and simulated. On the basis of the quasi-optical theory, VLASOV mode converter which has input mode TE03 and frequency 94GHz is subject investigated. Model parameters of mode converter are obtained using optimization design. The result shows that radiation wave beam is directional and sharpens obviously. Main lobe gain is up to 29.46dBi, output power centered in main lobe, side lobe gain become lower, radiation power efficiency is 87.36%.