Transcript

Sarah Naiman, Ph. D (00:10):
Hello everyone and welcome to today’s installment of our Threads of Equity podcast. My name is Sarah Naiman and I’m joined by Jen Loomis. Today we’re really excited to talk about some advice that we had given the Department of Energy on how to improve the availability of their workforce development program. So Jen, maybe to kick us off, could you tell us a little bit about what clean energy programs or workforce programs the DOE has been doing or the Department of Energy?

Jen Loomis, Ph. D (00:44):
Yes, definitely. So yeah, Opinion Dynamics was really happy to do some work for the US Department of Energy’s Energy Efficiency and Renewable Energy Program office. For short, I’ll be referring to them as EERE. And EERE has developed a number of programs over the past decade or so that are targeted at US college students and giving them the skills and knowledge and experience to hopefully pursue a career in the clean energy sector. And so a lot of these collegiate programs are structured as competitions. And so these are competitions where students get together in teams and complete the challenges that are with the different competitions and they gain experience working on interdisciplinary teams, honing their verbal and written skills to present their findings, and just learning general technical knowledge about clean energy sectors. And these really span all of the technology offices that are housed in EERE. So some of the competition programs are with the water power technology office, solar energy technology office, wind energy technology, office vehicles, geothermal and building technologies.

SN (02:11):
Wow. What a wide variety of topics that people can get engaged in. I think in terms of the programs and as we’re thinking about threads of equity being the theme of this podcast, could you just share a little bit of how these programs are thinking about equity? Who are they trying to engage and what do these programs try to accomplish outside of promoting a larger energy workforce?

JL (02:38):
Yeah. Well, so diversity, equity, inclusion are of growing interest and importance to the Federal Department of Energy. And so it’s important to them that they’re able to measure and investigate questions about diversity and equity in their collegiate competition programs. One of the things EERE asked us to do was help them determine how best to evaluate these program’s outcomes, but also evaluate diversity, equity, inclusion within the programs. And so these programs actually touch a lot of different people. Students clearly are one of the biggest focuses, and so diversity within students is really paramount here. So, we want to know the students’ racial and ethnic diversity. We want to know their age. Are there non-traditional students? We want to know their majors. Are we just focused on STEM majors or are we also reaching other majors like social scientists? And we also want to know things about maybe were they a first-generation college student? Did they come from a rural community? Because another important thing is that the outcomes and benefits do affect a variety of communities. We don’t want them to just benefit communities that are already doing great. And so if we can recruit students from historically marginalized populations or communities, they may be able to bring back their knowledge to benefit their home community. So there’s other populations of interest as well with these programs. Each student team comes with a faculty mentor that guides them through the competition challenges and helps teach them sometimes through coursework and sometimes through extracurricular activities. And so, the faculty are also important. We want students to see themselves and their faculty mentors and see the Federal Department of Energy as a place that they may want to work in the future. And so again, how do we recruit these mentors, and how do they encourage and engage the students?

SN (04:49):
That’s such an important piece. I know, at least as a social scientist, having been in an environmental program, trying to look to see are there people like me in the examples or in some of the case studies that we’re working with. So I can imagine how powerful that might be for the students themselves.

JL (05:10):
Agreed. Yeah, and the interdisciplinary nature of the teams is important too, so that the social scientists are on equal footing with the architects or with the engineers or with the chemists, and so that they can all work together and gain these interdisciplinary coordination skills.

SN (05:31):
It is definitely an important piece to be able to talk technically. I know even at our company as a social scientist talking with an engineer and making sure we’re using the same language or able to come to an agreement about how to proceed with a project or how to move forward, it is definitely a skill that can be useful for careers to come, not only in the energy space, but elsewhere as well. Everything is usually interconnected.

JL (05:59):
Definitely that’s a skill that the future workforce would definitely benefit from having.

SN (06:05):
Alright, so in thinking about these programs and the different actors, the different elements that you might be looking at, could you talk a little bit about what the programs we’re tracking or how we evaluate a program? How do we determine if it’s a valuable and what does that mean?

JL (06:24):
Yeah, good question. So, Opinion Dynamics completed an evaluation of EERE’s Solar Decathlon and Race to Zero collegiate competition programs. And one thing we found was that they were not collecting sociodemographic data of the participants, and I mean just very basic data, even their gender. And, critically, for many of the programs, they were also not collecting contact information. And so when we want to evaluate the outcomes at the end, it was really hard to find the student participants retroactively. And then we had to ask a whole lot of information that would have been good to know in advance. For instance, just thinking about gender, because we didn’t know what the population of participants looked like in terms of the split between males, females, and non-binary, we had no way of knowing how our representative, our survey results were of the population. And we didn’t know if we had non-response bias.

And so that really hampers our ability to judge the outcomes of the program when we don’t understand the population. So, this was one of the key pieces of value that we added for EERE was giving them advice on critical data to collect from the student participants. So, collecting things like basic contact information and basic things like gender, age, race, ethnicity, and then some additional points like first-generation college student. We also made recommendations to DOE EERE about important ways that they can measure diversity, equity, and inclusion within these programs. So Sarah, would you like to talk a little bit about how we recommended they measure equity in these programs?

SN (08:17):
Absolutely. So within these programs, as we’ve already mentioned or alluded to a little bit, we’re interested in understanding the breadth of students who participate. And one of the things that the program itself can do is collect administrative data or participation data. Can they make sure that they’re tracking who they’re reaching out to in terms of recruitment for the program, who’s actually signing up for the program? Looking at some of the demographics, Jen, that you just mentioned, race, gender, major, ethnicity, geographic location, and who is actually attracted to the program based on how it’s being marketed to or designed. And then, ultimately, who participates in the program and who even makes it to the finish line. There can be lots of barriers depending on people’s backgrounds that can inhibit folks from having positive experiences in some workforce programs. So, really having some of that participation tracking data to know when do people fall off?

If you’re doing a program that operates over time, over a couple weeks, over a couple months, over a year. Some of these programs, if I’m not mistaken, we’re two year programs, and that’s a long time to maintain some college student interests potentially. There’s lots of different things vying for attention or other responsibilities that come up. So I think that’s the first piece is just understanding who you’re reaching. But beyond the students, I think Jen, you had mentioned there’s different colleges or mentors that are involved. Similarly, trying to make sure that you’re able to track who you’re engaging with. Are we engaging with the Hispanic colleges or the HBCUs, the historic black colleges and universities, are they participating in the program? And if not, why not? And so beyond some of that thing that the program itself can do that it might be as simple as when people register, asking for a little bit of information at the start, some of that contact information, some background information, how they heard about the project or the program are all really critical.

And then beyond that, there are opportunities throughout the program without needing an external evaluator like us to be able to do some data collection, to implement a survey and ask people about their experiences. It doesn’t have to be very long, but just have opportunities to receive input or receive feedback live from the participants and understand where are the pain points, where are there opportunities for improvement perhaps within the cycle of the program, the future cycle of the program or otherwise. And I think the last piece is obviously all programs usually want to see what impact they have. Are they able to achieve what they wanted to, whether that is job placement, whether that is interest in a clean energy career, and really thinking about collecting data at the end to see how effective was the program. I think typically a lot of programs when we’re thinking about audiences of interest or how you might classify individuals, utilize geospatial data.

So, we see a lot across the US the use of the term disadvantaged community. This is a term that means different things in different states. The federal level has a definition. California has its own definition. New York has another definition of these are trying to capture historically underserved communities or marginalized communities that may not be receiving some of the benefits. And typically these measures focus on income and may include some other factors such as English being a second language, exposure to pollution, things like that. While it may be easy to say, oh, do you live in a disadvantaged community versus not, it doesn’t always give you the most accurate data because the scale is too high up. You’re looking at a whole area and generalizing the population, everyone in that population based off of these high-level criteria. So really trying to understand while those may be easy to use to try to identify if we’re reaching folks that may need some extra support or opportunities, that there are other more specific tailored ways to capture this information by interviews, focus groups, surveys, things like that.

JL (13:02):
Sarah, everything you said is so right on. And, we were able to make recommendations to EERE about different points in the participation process where they can collect important data to measure DEI among the participants in their programs. And so we found opportunities, like you mentioned, any enrollment period when people are first signing up and expressing interest, that’s a great chance to capture some data about sociodemographic information. Then a lot of the programs also have interim reports and final reports. So at that juncture, that’s another formal interaction between the participants in the program and could be another way to just have them update their information. And that allows us to track whether people did drop out along the way and at what points and have that information so we know who’s staying in and who’s falling out. And we can also see the makeup of the teams and how that might influence things.

SN (14:08):
Yeah, I’m really glad that you echoed that point of how do you integrate it into program design and already the activities that are planned or being executed or implemented, how do you tie in some of these data collection pieces so that they’re not too onerous on the implementer or even on the participants?

JL (14:27):
Right. And surveys are a sort of cheaper method of collecting this information or just adding a question to an enrollment form. But some of the more important measures of whether somebody wants to pursue a clean energy career or whether this experience was positive and beneficial for them, maybe collected qualitatively like through interviews or groups where students can really express in their own words how this program influenced them.

SN (14:57):
You get so much rich information from having conversations with people and about their experiences, but it is also important to make sure that people feel safe to share their real experiences and opinions of those programs. And thinking a little bit about who’s conducting those focus groups, do they have a rapport built with the participants potentially, or is it an external person who can say, I have no skin in the game. I am here to just listen and provide feedback to the program administrators?

JL (15:32):
Agreed.

SN (15:34):
So I think to close today’s podcast, we’ve talked a lot about different types of data to collect around diversity, equity, inclusion, accessibility for these collegiate programs, for the DOE. And I am interested in knowing a little bit more of what do we expect folks to do after they’ve collected this? If they are able to integrate survey questions into final reports or do interviews with some past participants to help them gain some knowledge or insights about diversity, equity, inclusion, where do we go from there and what can we expect folks to do with that information afterwards?

JL (16:21):
Yeah, a good question. So having this type of information can inform different parts of program design and implementation on the beginning end with outreach and spreading awareness and recruiting of participants. When you see maybe that dropout thing we were talking about about who’s interested but isn’t quite supported enough to make it through this data could tell you is it a certain type of college? Is it a certain type of student or what is it exactly where people might need more support or where people really seem to be able to be successful? An EERE has changed its outreach strategies to amplify its outreach to diverse colleges and changed some of the competition requirements so that it’s not just heavily STEM-focused majors that have an opportunity to participate. So it can inform recruitment in that way. It can also inform the support that student participants might need.

I know a lot of the programs offer students access to expensive software that college students can’t afford otherwise, but maybe there’s other soft skills and softer support mechanisms besides just tangible resources that they’ll need. And then really importantly too, when you collect this, it allows you to do an outcome or an impact evaluation. And that impact evaluation lets you know the benefits of your investment because the younger generations do tend to be more interested in the environment and sustainability. And so it’s like that naturally occurring market baseline. It’s really important to do impact evaluation so we can tease out the influence that the program had on students’ interest in clean energy careers, their clean energy outcomes and other outcomes of interest. So I think that’s one of the most important things, is just showcasing the accomplishments that the investment caused.

SN (18:29):
And I think a final piece I might add to that, Jen, is really thinking about future iterations of the program, and especially if these programs, I think some of them have been going on for close to a decade, if I’m not mistaken. And so they obviously have a template that they’re using year in year out, but how can they be flexible and adapt to changing needs or changing interests as well, and identifying opportunities to improve their program, improve the reach, but also maybe really highlight the things that people are getting out of the program. Those success stories can be super powerful as well. And so there’s so many opportunities once you collect data to take that in, think about it and see how the program can continue to evolve and continue to be relevant for decades to come.

JL (19:27):
Right? Or if some of the needs identified can’t be filled by the current program, it might inform future program offerings and spawn other programs that can fill really important gaps in generating the next clean energy workforce.

SN (19:45):
What a great point. We can’t expect one program to do it all. It’s always a great reminder.

JL (19:50):
True.

SN (19:51):
Any final thoughts that you’d like to share before we close out today? Jen, it’s been an honor taking some time to talk with you about this, and of course, you have a ton of experience working with these programs, so thank you so much.

JL (20:06):
Oh, sure. I always have more thoughts. Yeah, I guess maybe my final reflection is we spend a lot of our time in our day-to-day work working on either state level or utility level energy programs. And so this was really fun to work on and sort of get a national perspective and gain some visibility into what our federal Department of Energy is doing with its programs and reaching college students and doing investments in developing things. And so it’s exciting to see the activity going on at EERE, and it’s exciting to see the attention and interest being paid to diversity and equity, and I can’t wait to see what they do next.

SN (20:53):
Yeah, and I think I was going to iterate something that you said, I really enjoyed working on this project, and I think the collaboration of, there are times where folks say, I don’t know where to start with this in terms of how do we measure these things that we weren’t tracking before? Can you provide us some guidance practically? I think that’s my favorite part of this job is trying to figure out how do we make real time solutions that are feasible, that are accomplishable, and I think that’s super fulfilling, especially as we move into, I think more and more programs are integrating equity, diversity, inclusion, accessibility into their program designs or into how they want to measure success. So really excited to be involved in these discussions pretty early on.

JL (21:46):
Most definitely. Well, thank you so much for having me on the podcast today, Sarah.

SN (21:51):
Absolutely. Thank you all for listening in and hope to have you join us for our next installment of the Threads of Equity Podcast.

End music.