Offshore wind and wave energy have the potential to be significant sources of future global electricity production, reduce carbon emissions, decrease dependence on energy importation, and stimulate economic growth in coastal and remote areas. Fixed-foundation and floating offshore wind and wave energy technologies are at different stages of development, but they all have the potential to success- fully function in the renewable energy sector if developers can provide reliable, efficient technologies that can survive their harsh environment to be economically profitable. To achieve this, developers need to consider reliability simultaneously with power production and cost early in the design process. This thesis uses risk- and reliability-based design optimization to consider reliability, cost, and performance during subcomponent, device, and system design to enable the exploration of optimal solutions in offshore wind and wave technologies. The included work advances the state-of-the-art of reliability-based design optimization (RBDO) in offshore renewable energy systems via three research foci: 1) establishing relationships between component reliability, failure costs, power production, and layout optimization of offshore wind arrays, 2) evaluating how geometry optimization of WECs affects component reliability and power production, and 3) quantifying how co-location of offshore wind turbines and wave energy converters (WECs) in the same ocean space affects power production, reliability, and cost. Through these research foci, this thesis aims to achieve the objective of improving the design and market competitiveness of offshore renewable energy systems by establishing relationships between component reliability and systems optimization and creating methods for including reliability into design at component and system levels.