The sustainable production and supply of raw materials (“nonenergy raw materials”) and primary energy carriers (“energy raw materials”) is a core element of many policies. The natural resource base for their production and supply, and the access thereto, are limited. Moreover, raw material supply is high on environmental and social impact agendas as well. A broad, quantitative framework that supports decision makers is recommended so as to make use of raw materials and primary energy carriers more sustainably. First, this article proposes a holistic classification of raw materials and primary energy carriers. This is an essential prerequisite for developing an integrated sustainability assessment framework (ISAF). Indeed, frequently, only a subset of raw materials and primary energy carriers are considered in terms of their source, sector, or final application. Here, 85 raw materials and 30 primary energy carriers overall are identified and grouped into seven and five subgroups, respectively. Next, this article proposes a quantitative ISAF for the production and supply of raw materials and primary energy carriers, covering all the sustainability pillars. With the goal of comprehensiveness, the proposed ISAF integrates sustainability issues that have been covered and modeled in quite different quantitative frameworks: ecosystem services; classical life cycle assessment (LCA); social LCA; resource criticality assessment; and particular international concerns (e.g., conflict minerals assessment). The resulting four areas of concerns (i.e., environmental, technical, economic, and social/societal) are grouped into ten specific sustainability concerns. Finally, these concerns are quantified through 15 indicators, enabling the quantitative sustainability assessment of the production and supply of raw materials and primary energy carriers.