Ammonia (R717) is widely deemed to be the potential working fluid of the ocean thermal energy conversion (OTEC) system. To further improve the performance of OTEC Rankine cycle, R717-based binary non-azeotropic blends are employed as the working fluids, where R134a and R125 are selected as the additives. The thermodynamic model as well as the cost estimation model of OTEC system is developed to demonstrate the potential of power generation. The energy, net efficiency, and Levelized cost of electricity are evaluated and examined for OTEC system employing different blends, encompassing R717/R134a and R717/R125. The results indicate that the thermodynamic and economic indexes of R717-based blends are superior to pure fluids (i.e., R134a, R125, and R717). To maximize the net power, the mass fractions of R717 are 0.1 and 0.5 for the R717/R134a and R717/R125 blends, respectively, in a 30 kW-scale OTEC system, where the net power are higher than that of the pure R717 by 134% and 87%. In addition, R717-based blends perform better in energy cost than pure fluids, with 0–67% lower Levelized cost of electricity. Especially, a large-scale OTEC power plant is more economically profitable than a small one.