August 27, 2015
By Mikhail Haramati
What role will evaluation play in program development of the future? How can implementer’s best utilize new datasets and analytical tools to reduce the cost and increase the timeliness of EM&V? What role will evaluation play in the Clean Power Plan? How can certification be used to increase the credibility of the industry?
These are among the questions addressed by panels at the recently-held International Energy Program Evaluation Conference (IEPEC) in Long Beach. The agenda and full papers are available for the first time on the IEPEC website at http://www.iepec.org.
Opinion Dynamics had a strong presence at the conference, with seven of its evaluation professionals presenting on panels and at the poster session. These presentations included attribution methods for finance programs, survey methods at a time of low phone response rates, effects of seasonality on energy savings, how to increase participation in demand response programs, and constructing matched pairs for assessing program impacts. Staff also presented on methods for understanding and measuring spillover, web survey best practices for increased response rates, and a poster on enhancing the self-report method for free-ridership.
During a panel on effectiveness of finance incentives, Megan Campbell described recommended methods for estimating impacts of finance programs based on an evaluation currently underway of the CPUC finance pilots. In her presentation, Campbell made the distinction between the impacts of rebates on savings, the impacts of financing on savings, and how to parse the two in programs that include both. Campbell detailed the pros and cons of available methods for assessing savings. She also provided recommendations of best-fit methods based on the measure characteristics and type of program deployed.
Opinion Dynamics Vice President Tami Buhr had papers in two sessions at the conference. During a session on residential survey and data collection challenges, Buhr suggested improvements for designing survey research in an era of declining phone response rates. Based on an experiment with the Mass Save® brand, Buhr tested response rates from mail, web and phone with varying incentive levels and reminders. She found that a mixed mode design was most effective as it reached different demographics with lower fielding costs and higher response rates than any other mode alone. Contrary to other studies, she also found that up-front incentives boost response rates more than a chance to be entered into a prize drawing.
One of the highlights of the conference was the opening keynote speaker, Cass Sunstein, former head of the White House Office of Information and Regulatory Affairs, and author of the ground-breaking book, “Nudge.” Sunstein emphasized the importance of getting the numbers right for the costs and benefits of the Clean Power Plan. He explained that cost-benefit analysis is central to the Clean Power Plan and energy efficiency is central to cost benefit analysis. According to the EPA’s careful calculations, the health and GHG benefits that will occur as a result of the emissions reductions in the Clean Power Plan, are expected to save U.S. consumers more than $8 billion in total. This includes electric bill savings, as well as healthier communities and reduced air pollution-related deaths.
Another highlight was the session on Certification of Evaluation Professionals. This panel session invited comments from attendees on whether or not the time is now for certification of evaluators. The session drew so many participants and so many comments, it had to be stopped in order to close the session on time.
Certification sets standards and methodologies for current and future generations of regulators, utilities and evaluation practitioners. Opinion Dynamics’ philosophy towards evaluation includes both “art” and “science.” For the science part, we have staff leaders on methodology, statistics, sampling, economics, psychology, research, etc. The trick comes when considering the “art” part—numerous decisions about everything from how to clean data to how to ask a survey question in an unbiased way make our work defensible—and issues such as these can’t be legislated through a common method or even explained in a consistent manner, since the nuances of each situation matter. We believe the art and science of evaluation must stand the rigorous test of time. For that reason, Opinion Dynamics supports a certification process for the evaluation community, and continues to follow this topic with great interest.
Of special mention was the Lifetime Achievement Award presented to Ralph Prahl for his contributions to the energy efficiency industry over the past 30 years. As quoted in the IEPEC award submission form: “Few, if any, in the industry can match his understanding of both the big issues, as well as the detailed intricacies of impact, process, and market evaluations across the commercial/industrial and residential sectors and across the full range of technologies and markets.” We congratulate Ralph Prahl on this honor and thank him for his dedication to improving the quality of evaluation in the industry.
Download our conference newsletter and presentations below.