Assessment of wave energy converter (WEC) technology, commercial readiness and promise of economic competitiveness and monitoring of development progress are of crucial importance during and at all stages of the development. The assessment process requires employment of problem-relevant, fit-for-purpose, practical assessment criteria. The identification and the use of appropriate assessment criteria and methods has been an important issue since the beginning of WEC technology development. However, the investment into both the determination of assessment criteria and their application has not been adequate, given their crucial importance in guiding the technology development process to success in the most effective way. In recent years, the effort in determining assessment criteria and metrics have increased globally. This paper highlights a breadth of vital considerations supporting the specification of problem-relevant, realistic, effective assessment criteria. Following reflection on motivation, impact and benefits of technology assessment, a list of idealized expectations of assessment criteria and methods is provided and, subsequently, the significance of "realistic and effective is defined. The core contribution of this paper is the collection of aspects that need to be considered to specify the requirements for problem-relevant, realistic and effective assessment criteria, methodologies and tools. These assessment criteria considerations are formulated in the form of questions to show possible choices that can be made in the requirement specification for assessment criteria. Finally, some overall conclusions of the above considerations are drawn.