This paper introduces an enhancement to the transient stability of a wave energy conversion system (WECS) by using the coot optimization algorithm (COA) combined with an anti-windup method. This combination helps in elimination of the windup issue in the integral term of the proportional-integral (PI) controller during power system faults leading to a significant enhancement for the transient stability of the WECS connected to the grid. The COA is utilized to select the PI controller parameters and the anti-windup method back-calculation coefficients. The WECS converts the linear vertical motion into electrical energy by using an Archimedes wave swing device connected to a linear synchronous generator. Minimization of the generator's stator losses and maximization of the generator's real power is accomplished by the utilization of a generator-side converter (GSC). Also, both the point of common coupling voltage and the capacitor link voltage are set at their reference values by utilizing a grid-side inverter (GSI). An optimal design for the PI controllers in both converters is achieved by direct application of the COA to the MATLAB/Simulink model. A comparison is made among the results obtained by the COA and those obtained by other recent optimization algorithms under different grid fault conditions.