Contact Information

Liz Law: elaw@opiniondynamics.com | Zach Ross: zross@opiniondynamics.com  |  Have a topic you’re interested in? Let us know! info@opiniondynamics.com

Transcript

Liz Law, Editor and Content Specialist: Welcome to another podcast from Opinion Dynamics. Today we are going to talk about embedded evaluation, what it is how it differs from other EM&V approaches, and share some of our experiences utilizing embedded evaluations with our clients in the energy industry. I am Liz Law, Editor and Content Specialist at Opinion Dynamics. And I’m joined today by one of my favorite people to chat with Zack Ross, a Director specializing in directing large portfolio evaluations and technical research to support clients in TRM updates and market characterization. So Zack, can you start us out with some clarifications? Just basically, what is embedded evaluation?

Zach Ross, Director: Thanks, Liz. Happy to be here and talk with you today. And yeah, that’s a great question. So let me back up for a second. When people think about evaluation from a 10,000-foot view, what they often think of as what we might call retrospective evaluation, where a program administrator runs some sort of program or intervention and finishes up and then they hire someone to come in and take a look at, hey, how did this do? Right? I think that’s what people often think of when they think of evaluation activities. You know, however, in the EM&V space, these days, a lot of different terms are tossed around with different intents about evaluation, and embedded evaluation being one of them. When we talk about embedded evaluation at Opinion Dynamics, what we’re specifically thinking about, are evaluations where we’re engaged by a program administrator kind of, before a program even starts, or alongside the implementation of a program or intervention, and then we do our work kind of concurrently with that program happening. In a hypothetical Venn Diagram of types of evaluation, you might have heard terms, real-time evaluation, developmental evaluation, that evaluation shares some characteristics with both of those things. But it’s not quite the same thing as either one.

LL: I really liked the use of retrospective as opposed to standard or traditional, or what we are just evaluation, which is honestly how we often hear it, but I can understand that in a general term, what that means to be, you know, kind of working with the program implementer, or the administrator in the beginning, but what does it look like in practice, like in these programs, when we jump in as an evaluator? What are we doing?

ZR: Yeah, no, that’s a great question. So, you know, I think before I even sort of describe what an embedded evaluation might look like, I think the most important thing to be aware of is kind of what I just started talking about. But I want to expand that a little further. So that, you know, we are engaged with a program from the get-go. So, you know, a program administrator sets out to implement some sort of program, and they typically are going to have some specific set of objectives about what the program is trying to accomplish, how they’re going to do it. And then a theory of change like around that.. that intervention, you know, why they think that that program may work in the way that they are planning it to try to do the thing that they want. As the program’s implemented, it will inevitably, you know, have challenges that it has to overcome things may change in a market with customers, and a program may adjust its approach in order to address these. And again, thinking back to what we talked about a moment ago, a retrospective evaluation might come in and not see any of that… that has happened, right? It might come back and years later and look at a program and say, okay, cool, you did this contrast and embedded evaluation often is going to include an initial review and understanding of what is the intent of what a program administrator’s trying to do? Has the program been designed in a way that it is valuable that we can come in and actually answer questions that are going to help a program administrator understand whether a program is doing what they want? We likely will review data collection plans, you know, help understand what data is being gathered. Is that data sufficient to answer the questions that… that we, or the administrator, have? Do we need to gather additional data? Does that need to happen kind of in real-time or not? You know, once we do all of those things, that’s often, but not always, followed by rapid research designed to provide, you know, early feedback and recommendations to an implementation team. So, you know, trying to use evaluation activities to inform a program as it’s running instead of… okay, if programs concluded, now let’s provide some recommendations for the next time you run it or a future year. You know, I think being able to conduct activities like this is often helpful if a program has the flexibility to pivot while it is operating. And I guess I should say, you know, embedded evaluations can take any kind of form of things that we are typically used to seeing in our industry. This can be process evaluation-related, impact evaluation-related… both. But really, at the end of the day, what this really requires is that we have a collaborative relationship between a program administrator and implementation team, and as evaluators that we have clear roles about who is collecting what data, who is answering what questions and who is kind of doing what in terms of program activities.

LL: That makes sense to me. And I’m so I’m relatively new to the… to the energy industry. And even with that, I’ve been here a couple years. It doesn’t seem to me that this is an entirely new approach because I do feel like in a lot of situations, we are working with our clients from the get-go. So, what are some common uses of embedded evaluation that are already established? How is this known or implemented currently?

ZR: Yeah, no, and that’s a great point, Liz, this is absolutely not a new approach. I think currently in the energy space, you know, this, this sort of type of evaluation is getting some renewed interest and discussion. And certainly, people are kind of naming it more often. But… but this is not something that has been come up with in recent years, and we’ve been doing this kind of work for a long time at Opinion Dynamics when we need to. So, some common uses of embedded evaluation, I think, really, the main thing, where I have cut my teeth on doing some of this kind of work is, you know, really related to where a program is in its lifecycle. You know, I think the most kind of classic example of this is pilot programs. So, a program administrator, or an implementer, comes up with a new idea for a program, we want to, you know, try an intervention that we haven’t done before, and see if it helps us achieve goals that we are setting out to achieve. When you’re really early in a program lifecycle for a pilot, or even a program that maybe has matured out of a pilot phase, but not quite into a, we know exactly what this is, it’s stable, it’s going to give us what we want year over year, embedded evaluation is really helpful. Because you’re testing a concept, right? You’re trying to figure out, is this pilot, is this program designed, going to give me what I want? And if it’s not, how do I then adjust that to kind of recalibrate and see if we can be successful? So, when we’ve been programmed in this kind of stage of its lifecycle, you know, I think for administrators and implementers really benefit from working with evaluators upfront, often even before a pilot launches, to conduct market research to make sure that assumptions being made about what a pilot is trying to do are grounded in the realities of the market. And then, as we talked about a little bit before, you know, to work with an evaluator to make sure the pilot is constructed in a way that it can be evaluated. once a pilot launches, or again, a kind of early-stage program, that maybe has matured out of pilot phase, is running, you know, these programs can benefit from rapid feedback that we can provide using an embedded evaluation approach to give insights as to, you know, what do we need to tweak about the kind of strategy we laid out here to make sure that we are achieving what we think we can with the intervention we have designed. And again, I think, you know, I would contrast this with a more retrospective approach where you’d wait till you have all these results in for a program before realizing, oh, no, we haven’t quite hit exactly what we’re looking for.

LL: I want to ask; it sounded for a minute there like you were going to move on and talk a little bit about how the embedded evaluation can offer impacts of a pilot after the pilot has completed. And so, I’m wondering about scalability of a pilot. And the potential, it seems to me that it would be natural to continue asking those questions. But at that point, is it still an embedded evaluation? Or are you looking into something new, I guess, help me kind of wrap my brain around it, I suppose?

ZR: No, that’s a great, great question, Liz. So, you know, I think as we started to find a better evaluation, the beginning of this podcast, really, the only defining characteristic is that we’re there alongside every step of the way. And so sure, it continues to be an embedded evaluation at that point, even if you’re starting to scale something. But you know, trying, to make sure we are tracking where things are at in an intervention, and providing a program administrator with or an implementer, with what they need to know, kind of on a continuous basis, is the piece that’s really important here. So let me try to provide an example, right? Let’s say a program administrator is wanting some new kind of program with some sort of new measure or technology or delivery mechanism that is trying to help a certain type of customer achieve energy savings in a way that we haven’t seen before. You know, we might not have really any idea what you might get out of, you know, operating a program in this kind of way. And so, you know, to prudently invest ratepayer funds, program administrator might say, “Okay, we’re gonna pilot this program design with four businesses or five households, right? And then Opinion Dynamics will kind of be with us to understand the customer experience and then do some measurement and verification of what energy savings we get out of something.” That might be your kind of very early stage of a pilot. Then we might provide some estimates of okay, “hey, I’ve among a small group of customers, here’s what we’re seeing in energy savings.” This seems like a good piece of information, it seems like we have a sense of what we’re doing here, now, let’s scale this up a little bit more – try to deliver this intervention to more customers – see if these same sorts of assumptions hold, right. And you can keep going on that, you know, by orders of magnitude, right? And the role of the embedded evaluator does not go away just because you are scaling a program. At some point, we are always going to do summative retrospective-looking research and try to summarize what has happened here over time. And so, your research questions may change a little bit as you evolve. But I don’t think that takes anything away from evaluation still remaining embedded.

LL: Thank you for that clarification. Other than pilots, are there other uses for embedded evaluation that we see more commonly? Or, you know, like you said, it’s not a new thing… so it’s been around for a while; it’s fairly common to see.

ZR: Absolutely. So, you know, market transformation evaluations – in many cases, we need to establish baselines for those performance metrics. Or, you know, we often call them market indicators that help us understand, are we achieving the theory of change that we think we are. And this is, you know, kind of fundamentally tends to require embedded evaluation in the sense that if we don’t take the time to collect data, and measure baselines for those indicators that we have come up with before the intervention has really progressed, we might not be able to tease out what has happened naturally, versus what is happening as a result of the market transformation intervention. And again, this is because these programs are intervening at a kind of much higher level than your typical sort of energy efficiency program. They’re trying to influence market structures, trying to, you know, really do broad-based sort of changes rather than kind of individual customers decisions. And so, we have to have a really comprehensive understanding of the market that a program is trying to influence, how a program administrator thinks they’re influencing that market, who the market actors are, so on and so forth. And without having a really kind of embedded approach, you know, understanding what the idea behind the market transformation, you know, intervention even was to begin with, it’s really challenging to do that well.

LL: So it sounds like to me, especially in the market transformation, but also in some of the, of what you described with the pilots that having an embedded evaluation really helps for your evaluation to dig deeply or have like a depth of knowledge about the questions. One thing that is standing out to me, both in the discussion of pilots and market transformation, is, you know, the discussion of things that are new, whether that is testing new and innovative interventions in pilots, or also, you know, coming up with unexpected results, and therefore having those results early enough to be able to respond to them or see what they might indicate. And it seems to me that it would naturally lend itself to some of the emerging technologies. The energy industry, especially over the last few years technologies are just advancing one after another really quickly. You know, are we seeing embedded evaluation utilized in conjunction with some of these emerging technologies? Can you speak to that a little bit more? Because I’m, I am really enthused, I believe a lot of folks in the industry are enthused by the advancements, just innovations that we’ve seen lately.

ZR: Absolutely. I think you’ve called out a very specific place where that evaluation work is really important, and where we are seeing great desire from our clients to engage with them somewhat different way than we may have in the past in order to get the insights that they need. So yeah, I mean, we’ve talked, you know, throughout this conversation so far about, hey, things are new, we don’t totally understand if something’s gonna work, embedded evaluations, really important to make sure that you can track you know, throughout something scaling up, you know, a better understanding of what’s going to happen in emerging technology is a perfect example of that. So I’ll give you one specific example that I think you know, our firm has, has publicized some work we’ve done on recently, you know, we completed a study recently for PG&E, looking at the potential of electric vehicles in the automated demand response space. And this study, you know, kind of grew over time in terms of what we were looking at. You know, we started by, you know, helping look at a set of different capabilities of EV charging providers and third parties and being able to control and manage residential EV charging to support a technology field test. So, what when we started working with PG&E on this project, I’m not sure that we even knew precisely what all the research questions we were we were looking to answer were. We kind of started with this first phase of getting information, trying to understand what things might be valuable to begin with. And we were able, you know, through those insights to help figure out, okay, there’s really something here, we can now go and actually do a field test, measure a lot of, you know, stuff here to help them understand better, you know, what capabilities we might actually have, and then help think about how those efforts could be scaled using the data that we had gathered to inform kind of a better understanding of what you know, interventions might be available long term. And I think this is just a really great example of how embedded evaluation can give you a very different sort of support from an evaluator as compared to a more retrospective approach where maybe we wouldn’t have even come in to look at a program like this until after all of those phases that I described had already happened, right. So you know, I don’t have the space to go into this in-depth on this show. And many of my colleagues are much more informed about the particular study than I am. But I know we’ll be putting a link in the show notes to the study (available here) and more information about it on our website, for anybody who’s interested.

LL: Yeah, absolutely. That’s really exciting to think about, you know, being able to start from we’re not exactly sure of all the research questions to say, Hey, look at this potential, especially given that I think it was in over the last couple years, something like two dozen new electric vehicles have been announced or launched within… within the US. And that’s, that’s phenomenal. I think, knowing, you know, being able to questions around the role that the EVs can play in DR. We’re hearing a lot. And, like you said, it is far beyond the scope of this podcast today to talk about that. But we do have the full report up on our website, it’s, um, it’s public. And we will have a link to that in the transcripts. So, folks who are interested in digging into the nitty gritty of what we were able to find out. And some of the technologies we worked with are welcome to do that. So. So this really leaves me with only one question I want to mention, again, that I am two years into the energy industry so there may be historical nuance here that I am missing. But why would we not use an.. or why would clients not use an embedded evaluation? Like why wait until after the program has ended to see if it was successful? You know you can have real-time or near real-time results to inform the next stage or.. or pivot if things aren’t working. So, can you think of situations where embedded evaluations would be the wrong choice? Where maybe doesn’t make sense?

ZR: Yeah, no, it’s an… it’s an interesting thing to wrestle with here. You know, I think we pride ourselves as a firm on working very closely with our clients to understand what their specific needs are, and to build evaluation approaches for them that help meet those needs, in the way that’s the most effective for them. And there’s a lot of considerations there, you know, the amount of time that it’s going to take to do a pretty good piece of work, evaluation costs, exactly what a regulatory requirement for reporting on a given program might be. So you know, certainly, if you have a newly developing program, you have a market transformation program, if you have a program administrator working on intervention, where you’re kind of building the plane as you’re flying it, you know, we think that it is the right choice to work with us early on in the lifecycle of a program to help make sure that we’re there to help provide some early feedback. But you know, for clients, who might have more limited budgets available for the type of research that they’re able to do to support their work, and who might, you know, really have a mature program that they think they understand pretty well, and that they want to just do some basic tracking of, you know, looking at, you know, okay, did we continue to achieve what we have been achieving over time in the way that we think we’ve been achieving it. It might not always be the best use of resources, the best use of a program, you know, Administrator implementers time, best use of our time to use an embedded evaluation approach, in a kind of full-fledged way. You know, again, I always tell my clients that we would rather know ahead of time, what we’re looking at with them, and at least to some very early discussions of, okay, here’s what you’re setting out to do. And maybe we don’t do anything for the next nine months after we’ve initially started a project with you. But yeah, there are certainly times where, you know, you’d rather spend your resources after the fact looking back to just verify that things are going in the way that you’ve expected rather than kind of have us in there every single step of the way. But again, these things are all going to be dependent on what’s happening in a market. You might have what you think is a really mature program and then you may have a sudden change to a federal standard or to a technology available that might disrupt what’s going on. So that evaluation is always great to do when you can, and we will happily work with you to think about, you know, is it the right choice in a given situation?

LL: 100%, when you mentioned, you know, you might have a federal regulation change or something else. I mean, I know everybody says the only constant is change, but in the energy industry and in the landscape, energy landscape that really feels to be to be very true. Was there anything else about embedded evaluation that I’ve missed? I feel like I’ve got a pretty good grasp around it. So, unless you want to interrupt me? Well, I think, you missed your chance.

ZR: I think I think we’re good. Liz.

LL: Thank you for taking time out of your busy schedule to chat with me today. You are a wealth of knowledge with over a decade of experience in the energy industry and Opinion Dynamics. And so I will never tire of picking your brain to better understand the nuances of the work we do and the impact of the work that we do here. I really appreciate it.

ZR: It’s been great to … sorry…

LL: No, you go ahead.

ZR: Great to be here. And I’ve really appreciated the conversation and look forward to talking to you again soon.

LL: All right. Well, that wraps us up for today. Thank you, everyone, for listening. And we will be back before too long with more conversations on current topics affecting the energy landscape with some more of our industry experts here at opinion dynamics.

End music.