Introduction

The TECH initiative is a market transformation program designed to increase the adoption of heat pumps for space conditioning and water heating in single-family and multifamily residential homes in California. As part of the Initiative, TECH offers optional workforce development trainings designed to improve contractor comfort and familiarity with heat pumps as well as electrification as a business model.

Twenty-three optional trainings were offered between January and June 2022 across the three implementers, the National Comfort Institute (NCI), Electrify My Home (EMH), and the Association for Energy Affordability (AEA).[1] Trainings offered through NCI focused on the technical side of HVAC equipment, EMH’s focused on the contractors’ business model and whole-home electrification, and AEA’s trainings were centered around heat pump systems in multifamily buildings, covering central and individual systems for space conditioning and water heating. All of AEA’s trainings were short online webinars and therefore we did not evaluate these trainings to the extent we did the 19 in-person (and often multi-day) NCI and EMH trainings.  

A description of each NCI and EMH training offered in the first half of 2022 follows:

• • NCI Airflow Testing and Diagnostics: Intended for HVAC contracting firm owners, senior managers, sales consultants, installation mechanics, and service technicians, this eight-hour class provides technical training on performing static pressure testing, how to professionally install static pressure test ports, and how to measure and interpret static pressures.
  • NCI Refrigerant-Side Performance: This residential and commercial certification class provides students with real-world lessons and hands-on training. It is based on proven techniques on how to best approach refrigeration-side issues. Students learn to apply NCI’s performance-based, systematic approach to refrigeration-side diagnostics, including strategies for mitigating non-refrigerant faults prior to attaching refrigerant gauges.
  • NCI Residential System Performance and Electrification: This 20-hour certification course teaches students how to test, diagnose, and improve total residential HVAC performance. This course features numerous hands-on demonstrations that include how to use the test instruments, proper testing locations, and live testing and interpretation of readings. This class is offered in both in-person and live webinar formats.
  • EMH Residential Space Conditioning and Water Heating Electrification: Designed for construction trade personnel of all levels, this three-day class informs students of near-term and far-reaching changes in the home building industry and driving forces of such, including California’s regulatory and legislative framework. Students gain in-depth knowledge about heat transfer mechanisms, functionality, and benefits of heat pumps in residential electrification, as well as how to transition from traditional gas heating to modern electric heat pumps without negatively impacting their bottom line. This course is offered in live format only.

TECH offered contractors and their staff incentives for attending trainings, and they are described in Table 1. Contractor organizations were eligible to receive payouts for their staff participating in one of NCI or EMH’s multi-day trainings and subsequently submitting a minimum number of qualifying TECH applications. Technicians and sales staff were able to receive incentives for each day they participated in NCI training or EMH training. In our Level 1 Reaction survey following the training, 46% of respondents (26 of 56) said that one reason they participated in the training was to receive the TECH training incentive.

[1] AEA trainings were short, webinar-format trainings; we did not evaluate these trainings to the extent we did the in-person (and often multi-day) NCI and EMH trainings.

This memo presents findings from a post-training survey fielded approximately six months after contractors completed an NCI or EMH optional training through TECH.

Methods

In our evaluation, we use Kirkpatrick’s Adult Training Evaluation Model (Figure 1), the gold standard for evaluating adult learning interventions, to assess participant reactions to optional TECH training interventions.

As illustrated in Figure 1, Kirkpatrick’s model consists of four levels:

• • Level 1 – Reaction: Measures how participants feel about the learning experience. The value of Level 1 is that a good training experience improves knowledge transfer.
  • Level 2 – Learning: Measures the degree to which participants change attitudes, increase knowledge, or enhance skills as a result of the learning experience. The value of Level 2 is to demonstrate that learning occurs as a result of the training.
  • Level 3 – Behavior: Measures the degree to which participants apply what they have learned outside of the learning environment. This level seeks to demonstrate whether trainees take the information they learn and apply it.
  • Level 4 – Results: Measures the degree to which targeted outcomes are achieved system-wide. In this study, we measured the training’s results in terms of energy savings. The value of measuring Level 4 is to inform the return on training investment realized from the training endeavor.

To assess contractor knowledge and satisfaction after each training, implementers fielded a post-training Level 1 reaction survey closely following each course. To assess Level 2 Learning, the evaluation team is analyzing the knowledge tests administered by the implementers pre- and post-training. Approximately six months after trainings occurred, the evaluation team fielded a post-training Level 3 behavior survey to assess how contractors have been able to apply knowledge acquired during the trainings in their current work.

The objective of the Level 3 Behavior assessment was to gauge the extent to which contractors who attended an optional workforce training are applying what they learned in their work. To address this objective, our key considerations included contractors’ ability to apply learnings from the training in their work, the frequency in which they are using their new knowledge on the job, and what impact, if any, the training has had on their career.

The evaluation team programmed and administered the web-survey to all attendees who completed a TECH -sponsored training between January and June 2022. Survey invitations were delivered by email to trainees. We administered the survey in two batches to allow at least six months to pass since contractors completed their training. Survey invitations in batch one were delivered in September 2022, and batch two was delivered in November 2022.

According to rosters provided by Energy Solutions, the TECH implementer, 180 people attended at least one day of optional training, and were thus invited to complete the survey.[2] These 180 attendees were associated with 72 TECH-enrolled contractors, or about 8% of TECH’s enrolled contractors as of late May 2022 (72 of 917). Training attendees were prompted to provide survey responses about the specific training they completed. In cases where an attendee completed multiple optional trainings, the training with the lowest participation was prioritized to maximize survey responses across all courses.

We received a total of 42 survey responses. That resulted in a 11% response rate to the Post-Training Level 3 Behavior survey for NCI trainees, and 35% response rate for EMH trainees. Response rates at that level may introduce nonresponse bias to the survey data. This type of bias occurs when those who choose to respond to the survey differ from those who did not respond to the survey. This limitation is good to keep in mind when reviewing the findings.

Table 2 displays all NCI and EMH trainings that took place between January and June 2022, along with the count of those who attended each training versus how many completed the Post-Training Level 3 Behavior survey.

[2] Count excludes anyone the rosters indicated withdrew from training or was a no show for the entire length of the training.

Findings

This section presents findings related to how contractors have applied learnings from optional TECH-sponsored trainings in their current work.

1.1 Contractor Heat Pump Experience

Nearly half (18 of 42; 43%) of surveyed contractors installed 10 or fewer HVAC heat pumps since attending the training six months before (Table 3). The remaining contractors installed more than two on average per month. In comparison, respondents reported fewer instances servicing HVAC heat pumps, whereas the majority (24 of 42; 57%) mentioned they have serviced 10 or fewer.

Surveyed contractors who have installed or serviced at least one HVAC heat pump in the past six months are most confident in their ability to sell and install heat pumps since attending the training (Figure 2).

Fewer than half of surveyed contractors reported working with HPWHs in the six months since attending the training (Table 4). Of all the contractors who installed at least one HPWH since the training, they installed only one per month on average. Ten percent of surveyed contractors reportedly installed more than 10 per month on average. Further, a minority of contractors (9 of 42; 21%) mentioned servicing any HPHWs over the past six months.

Respondents who serviced or installed a HPWH in the past six months are most confident in their ability to size and install equipment (Figure 3).

1.2 Training Influence on Work Responsibilities

Some training attendees reported positive impacts to their career, in the form of promotions or raises. Most surveyed contractors (29 of 42; 69%) experienced a change in their work responsibilities since completing the training (Figure 4). The majority of them (18 of 29; 62%) considered the change to be a promotion (Figure 5). Nearly half of respondents who experienced a transition in their work responsibilities (13 of 29; 45%) subsequently also received a pay increase.

Trainings provided by TECH have been an important factor in surveyed contractors’ professional development and career advancements (Table 5). When we asked contractors to rate the importance of the heat pump training in helping them take on new work responsibilities using a scale from 1 to 10 where 1 was “not at all important” and 10 was “extremely important,” most respondents (19 of 29; 66%) provided a rating of 10, indicating the training was extremely important. Similarly, among respondents who reported they received a promotion since attending the training, two-thirds (12 of 18; 67%) rated the training a 10, indicating the training was also extremely important in helping them secure their new position.

1.3 Training Application in Work and Barriers

Nearly all contractors reported using what they learned in the TECH-sponsored training frequently on the job (Figure 4). All surveyed contractors use what they learned at least once a month, while half (21 of 42; 50%) use what they learned every day at their job.

Most surveyed contractors (36 of 41; 88%) are very likely to continue using knowledge they acquired from the training in their current work (Figure 5).

Moreover, nearly all surveyed contractors (40 of 41; 98%) intend to continue to apply what they learned in the training as a standard practice in their work moving forward. These findings suggest that the contractors find the lessons valuable and beneficial to their businesses. The single respondent who said they do not plan to apply their learnings in their work in the future explained this is due to lack of available rebates incentivizing them to sell heat pumps to customers.

More than half of contractors found that the complexity involved in converting gas systems to heat pumps and how that results in longer installations was the biggest challenge when trying to apply knowledge acquired during the training in their current work (Table 6). The higher upfront cost of heat pumps compared to gas equipment and the unreliability of customers’ electricity supply also factored into this challenge. Furthermore, six respondents mentioned a need for more equipment rebates, again highlighting customer costs as a barrier and demonstrating the positive impact TECH incentives had on the heat pump market by offsetting high upfront equipment costs. One contractor wrote in the survey about the barriers they have experienced:

“No one wants to do the necessary upgrades to upgrade [to a heat pump]. The cost of doing things like upgrading power from the main panel to the equipment location are what discourage most customers. They don’t want to pay the high costs, they don’t want the mess in their home, they don’t want to take the chance that the walls won’t look the same and they most of all don’t trust that going electric is the smart thing to do because of the lack of the utility companies ability to supply us all with power now without everything being electric.” -– Attendee of NCI’s Refrigerant-Side Performance Course

Additional challenges included lack of customer knowledge about heat pumps, the need for more rebate offerings to help reduce the cost for customers, the complicated TECH rebate application process, and issues with acquiring and retaining staff who are knowledgeable about heat pump equipment. Notably, nearly one-third of respondents reported no challenges when applying learnings from the training in their work so far.

Other challenges contractors mentioned included inconsistent heat pump equipment inventory (one respondent), more training on Manual J calculations (one respondent), and simply more customer opportunities to utilize their new knowledge (one respondent). One final respondent shared that they are considering starting their own company based on the knowledge they acquired through the training.

1.4 NCI Training Application

Most contractors, most or all of the time, applied what they learned from the NCI training on the job (Figure 6). For the two job tasks where at least one respondent indicated they only did them sometimes, respondents shared what prevented them from doing that task. One respondent said their employer does not allow them enough time at a site to use static pressure measurements to help assess the viability of an existing duct system for a heat pump retrofit. Of the three who only sometimes investigate why a heat pump might need to go beyond a simple changeout, one explained this is due to the method not aligning with their organization’s practices. The remaining two respondents cited customer-side issues, sharing that customers either don’t care or don’t have the time.

Note: Respondents were only asked about job tasks relevant to the specific training they attended, thus the number of contractors who provided a response for each statement varies.

1.5 EMH Training Application

Nearly all surveyed contractors who attended the EMH training now promote heat pumps to their customers, including those who had gas appliances, and explain the value of heat pumps most or all of the time (Figure 7).

Only two respondents reported carrying out all three job tasks less frequently than “most of the time.” One of them attributed this infrequency to the combination of a lack of customer demand and the high cost of heat pumps, while the other said the elimination of rebates left no incentive for them to explain the value of a heat pump to customers. Both responses point toward the importance of financial incentives to making a heat pump sale viable. One final respondent selected “unsure” for all three tasks.

1.6 Training Impact on Work

While a minority of surveyed contractors reported no change in their work since attending the training, the rest reported multiple positive changes (Table 7). More than half of contractors (23 of 42; 55%) shared they have experienced an increase in their confidence when working with heat pump equipment or presenting heat pump options to customers. Among the 13 contractors who described an increase in their confidence when working with heat pump equipment, two respondents added this has led to higher quality work and thus greater customer satisfaction.

One contractor who reported both an increase in their confidence presenting heat pump options to customers and working with equipment shared that the training has not only expanded their reputation in the industry but has also helped their company gain profitable work. Additionally, a few respondents who reported promoting heat pump options more frequently on jobs further explained their company has now transitioned to more of an overall electrification business model (three respondents). Some examples contractors shared in the survey included:

“I have definitely started to push and offer heat pump conversions on a regular basis now, whereas before it was very rarely.” -– Attendee of EMH’s Residential Space Conditioning and Water Heating Electrification Course

“Due to the training I have a much better knowledge base then I did previously. I am better prepared to explain the benefits of transitioning [to a heat pump] from what a customer currently has, even if it may be at a higher cost initially. I now have more tools and resources to expand my standing in the industry, while at the same time increasing my company’s ability to conduct meaningful and profitable work.” -– Attendee of EMH’s Residential Space Conditioning and Water Heating Electrification Course

Other changes contractors reported having occurred in their work since attending the training included assisting in the training of other technicians at their company (one respondent), and a better understanding of the TECH rebate submission process (one respondent).

1.7 Sharing Training Knowledge

It seems training attendees were able to internalize and remember what they learned. Over three-quarters of surveyed contractors (33 of 42; 79%) were either “very confident” or “moderately confident” in their ability to teach their colleagues about what they learned during the training (Figure 8). In fact, most respondents (39 of 42; 93%) reported they have already shared what they learned in the training with others, demonstrating the high perceived value of their new knowledge and skills. Below is what one contractor wrote in their survey response to the biggest change they noticed in their work since attending the training.

“[The biggest change noticed in my work since the training is a] deeper understanding of fundamentals and the ability to pass on to installers and technicians.” -– Attendee of NCI’s Refrigerant-Side Performance Course

1.8 Additional Training

Despite most trainees feeling more confident working with heat pumps and promoting them to their customers, more than half (23 of 42; 56%) shared they would like additional heat pump training (Table 8). Of those who indicated they would like to receive more training, many specified they would like more hands-on or field-training opportunities (eight respondents), servicing and repair training (three respondents), or training on various calculations (i.e., load, solar production, cost) (six respondents). Other support and/or resources respondents reported would be helpful included more TECH equipment rebate funding and better explanations of the TECH rebate application process.

The one contractor who provided the “other” response indicated it would be helpful for manufacturers to be more knowledgeable about the products they are selling. Two contractors also mentioned in their open-end responses that they never received the Visa gift card and diploma they were initially promised upon completion of the training.

Conclusions and Recommendations

We offer the following conclusions regarding TECH’s optional workforce trainings:

• • Conclusion: The delivery of the trainings was effective, and the content was valuable. Attendees valued the lessons they learned because they are frequently applying what they learned on the job and have already shared some of what they learned with others.
  • Conclusion: Training of the HVAC workforce on heat pumps is a good investment. The conversion of customers’ systems from gas to heat pump is more complex than like-for-like replacements and is a hurdle contractors must address if they are to sell more heat pumps. The training needed is not singular, it’s multifaceted. It needs to include sales strategies, technical training, and as we’ve seen in some cases, even training on rebate and incentive applications.
    • Recommendation: The TECH Initiative should continue to support optional WE&T trainings for contractors to improve their technical and customer-side sales skills.
  • Conclusion: The trainings improved contractor comfort recommending heat pumps to customers and their confidence working with heat pump equipment. Contractors gained technical knowledge on system sizing, system performance, and refrigerant management. They also improved their ability to explain the value of heat pumps to customers and promote electrification strategies.
  • Conclusion: Employers valued the heat pump knowledge and skills provided by the training. What the attendees learned via the trainings was important to their employers such that nearly half of surveyed trainees (18 of 42; 43%) received a promotion in the six months following the training. This conclusion implies the market is starting to demand workers with heat pump knowledge and skills, and that employers are willing to promote and pay more to employees with such skills.
  • Conclusion: Consumer-side barriers remain a challenge for contractors selling more heat pump equipment. Limited awareness or interest, higher upfront costs, and uncertainty in the reliability of their electricity supply are all customer-side barriers contractors must confront. The marketing and education that comes with incentives to reduce upfront costs can help to lower these customer barriers. This conclusion points to the need for continued customer-side investment in the form of incentives as well as marketing and education. Once the incentives are re-instated, they will have more opportunities to apply their knowledge.
    • Recommendation: We recommend that TECH and its subcontractor team invest in customer marketing, education, and awareness campaigns when incentives become available again.

Appendix:

Survey Instrument

Click on PDF icon to view.