Evaluating the extent to which an individual building consumes energy in excess of its peers is the first step in initiating energy efficiency improvements. Energy benchmarking offers such an assessment, albeit simplified. A widely used energy benchmarking tool is the Environmental Protection Agency’s (EPA) Energy Star® Portfolio Manager (PM), which is based on the Commercial Buildings Energy Consumption Survey (CBECS) database. This tool relies on standard linear regression models between building energy use intensities (EUIs) and certain building shell characteristics and equipment types. The statistical significance of these models is rather poor, and they only include continuous building variables. In an effort to improve these models, we have investigated the use of a state-of-the-art, tree-based ensemble learning methodology, the random forest (RF) algorithm, to identify the most influential CBECS parameters that impact building EUI for medium office and school building types. Surprisingly, none of the building features turn out to be really influential. Many of the CBECS building features found to be ranked relatively high are not the ones included in the PM models, and the resulting RF models were only marginally better than the linear regression PM models. These findings cast doubts on the veracity (i.e., the accuracy and completeness) of the data contained in theCBECSdatabase.More careful appraisal is warranted given the widespread use of this database for extracting meaningful generalized correlations between energy use and various building and related characteristics. The poor statistical significance of these correlations has important implications for ongoing building energy conservation policy instruments, such as energy reporting and disclosure legislation being enforced by numerous U.S. cities.