Executive Summary

The TECH Initiative is a $120 million pilot program designed to help advance the state’s mission to achieve carbon neutrality by driving the market adoption of low-emissions space- and water-heating technologies for existing single-family and multifamily residential homes. The Initiative was created as part of California Senate Bill 1477 and is funded by revenues collected through California’s Cap-and-Trade program. Through a combination of market incentives, supply chain engagement, workforce development, consumer education, regional pilots, and Quick Start Grants, the initiative installs low-emissions space and water-heating technologies in existing California homes.

Opinion Dynamics is serving as the embedded evaluator for the TECH Initiative. This report is an interim process evaluation of the TECH Initiative, publicly known as TECH Clean California. We completed this report approximately eight months after TECH officially launched its incentives on December 7, 2021, and about a year since TECH implementation staff began program design and planning. The TECH Initiative was designed to be implemented over four years. However, incentive funding was largely claimed in its first six months.

The primary objectives of this interim process evaluation were to investigate the TECH implementer’s program processes; the contractor experience enrolling in TECH, applying for incentives, and attending optional trainings; the customer experience finding a contractor, completing the project, and paying for it; and finally, an assessment of single-family TECH project characteristics. We also sought to identify opportunities to optimize the program and inform program scalability. Opinion Dynamics used a mix of primary data and secondary data to answer the research objectives.

Findings

TECH Contractors: TECH had enrolled nearly 900 contractors by May 2022, when the single-family incentives had run out in most gas utility territories. The large majority of enrolled contractors (85%) held a C-20 HVAC Contractor license. More than half of enrolled contractors learned about TECH from a manufacturer or distributor, and more than half found it easy to enroll in TECH. Enrollment challenges included too much paperwork, length of time to complete paperwork, and slow response to questions.

Contractors found the TECH incentives to be generous and they exhausted much of the Initiative’s incentives in the first six to eight months. A majority of contractors marketed the TECH incentives to their customers. Nearly two-thirds of contractors found the customer and equipment eligibility criteria clear. Confusion related to difficulty verifying customer and equipment eligibility, often relating to the zip-code look-up tool. Just over half of contractors using layered incentives found it difficult to determine the customer’s incentive amount. Even so, nearly all (98%) contractors who layered incentives found it to be helpful in selling heat pumps to customers.

A minority of surveyed contractors who completed an application found it easy (43%), and 36% found it difficult. A sizeable portion (21%) found it neither easy nor difficult. Those who had challenges stated the application was asking for something they did not have. Others found the application too complex, that it required for too much documentation and photos, and that there was little support for helping them troubleshoot problems with the application. In an interview, an HVAC contractor explained that it took his staff at least “an hour or two” to complete each incentive application due to the volume of pictures and files they had to upload.

The volume of paperwork and frequent need for follow-up led to delays in reimbursements. It seems most contractors used the incentive as a point-of-sale rebate. The contractors paid for the heat pump equipment in full and discounted the customer’s invoice. While they awaited reimbursement from TECH, the contractors’ firms acted as a bank and carried the unpaid debt of the incentives. The four interviewed contractors explained that tying up that much cash flow waiting to be reimbursed was a risk to their firm. The delays in reimbursement have had negative effects for contractors’ businesses and have caused mistrust with the initiative. An interviewed contractor who sells both HVAC heat pumps and heat pump water heaters (HPWHs) reported they felt “powerless in this process.”

TECH Customers: Half of customers who purchased an HVAC heat pump found their contractor through an existing relationship while half of HPWH customers found their contractor online. Financial and environmental benefits were motivations for the purchase of a heat pump, including the TECH incentive and the ability to pair the heat pumps with rooftop solar. Most HVAC heat pump customers were looking to replace old equipment that was either functioning poorly or not functioning at all. On the other hand, HPWH customers were most likely to say they were exploring new options while their existing equipment was still functioning well.

Surveyed single-family customers generally reported positive experiences with their contractors. The majority of contractors explained how to use the equipment, answered questions satisfactorily, and showed up on time. Fewer contractors reportedly explained how to perform equipment maintenance, indicating this could be an area for improvement. A majority of customers reported that their contractors offered useful information about why an electric option was a good choice for their home (76% HVAC, 68% HPWH), although a sizable portion (34% HVAC, 69% HPWH) indicated they mentioned a heat pump as an option before the contractor did.

TECH Incentives: The TECH incentive stimulated heat pumps sales, contractor revenue, and market growth. Most surveyed contractors (55%) saw their heat pump sales increase either a great deal or a moderate amount since participating in TECH. With the TECH rebate being so influential, many contractors reported sales fell due to the abrupt loss of program funding. A majority of surveyed contractors (93 of 161; 58%) who had at least one TECH project reported the suspension of TECH incentives affected their sales “a great deal” or “a moderate amount” The TECH incentive also encouraged customers, as 93% reported that the incentive was either “very important” or “somewhat important” in their decision to buy the heat pump. In the follow-up interviews with single-family customers who were aware of the incentive, there was confusion regarding the amount of the TECH incentive. Financing was used by a minority of customers but was important for their ability to purchase the heat pump. Few households required electric panel upgrades; they were more common among HPWH projects than HVAC projects.

Households in PG&E territory (1,289 of 3,117; 40%) or SoCalGas territory (1,257 of 3,117; 40%) made up the bulk of TECH projects with the rest of households primarily residing in SDG&E territory (Table 1). This is proportional to the spending allocations required by the California Air Resources Board (CARB). There are, however, differences in the distribution of heat pump types installed across these territories. Almost half of households that installed an HVAC heat pump belonged to SoCalGas territory, while most households that installed a HPWH resided in PG&E territory.

TECH Projects: A large majority of heat pump installs were HVAC heat pumps, in single-family residences, and outside of disadvantaged communities. HPWH installs were commonly completed in one day, while HVAC projects were often likely to take more than one day. Generally, installed equipment passed city inspections on the first inspection. The fact that 12% of HVAC and 8% of HPWH installs did not pass inspection on the first visit potentially indicates room for improvement in contractor knowledge of heat pump-related code requirements.

HVAC Projects: The incentive application database omitted the previous AC type for more than half of the households that received an HVAC heat pump (56%). Of the rest, most households had central AC before (83%). TECH contractors installed more efficient HVAC equipment in disadvantaged communities (DACs) and when pairing the heat pump with another program’s incentive. TECH contractors were significantly more likely to have performed duct sealing at DAC households than non-DAC households. TECH contractors included smart thermostats on just under half of projects while they completed Manual J calculations on about 10% of projects. 

HPWH Projects: TECH-incented HPWHs were most likely to be installed in the garage. Just over half of both TECH-incented HPWHs and previous water heaters were 50-gallon capacity, but more of the new HPWHs were greater than 50-gallon capacity compared to the water heaters they replaced. TECH contractors most often set HPWHs on a hybrid operating mode (53%), though a heat pump-only mode (43%) was fairly common as well. TECH contractors recorded that 4% of HPWHs were set on electric resistance operating mode. It is unclear why the contractors would set HPWHs on electric resistance mode because the benefits of a heat pump are not realized when the equipment operates in this mode. TECH contractors installed thermostatic mixing valves in a majority of cases.

Workforce, Education, and Training: The training offerings in TECH’s first year appeared opportunistic and followed a somewhat scattershot approach of including a variety of topics, as opposed to a well-thought-out plan supported by a clear theory of change. While the implementation team attempted to be very responsive, it was challenging to get a full schedule of training events. Information about TECH-sponsored trainings have been housed in a variety of places, which can be advantageous for outreach and awareness. But there did not seem to be one site that clearly lists all of the TECH-sponsored trainings.

Almost a quarter of surveyed contractors (21%) had not heard about the optional trainings offered through TECH. Among contractors who had heard of them, over half (54%) had not attended any trainings. The most common reason contractors gave for not signing up for one was they had already completed a similar training (25%). Other reasons included inconvenience of the training, related to either time, duration, or location. About 12% of contractors who signed up for a training did not attend at all or dropped out early.

Those who completed trainings were satisfied with what they received and trainings improved their confidence working with heat pumps. Attendees reported higher satisfaction with instructors than the training overall. Nearly all surveyed trainees (98%) reportedly planned to share the information they learned from the training with their colleagues. Additionally, surveyed trainees were likely to recommend TECH trainings. Suggestions to improve trainings included lengthening trainings, using a more hands-on, interactive approach, and providing more in-depth explanations and evidence for calculations. The most commonly reported takeaways from TECH trainings were a better understanding of electrification and the important role it plays in decarbonization, quality control best practices, and the importance of proper equipment sizing.

Program Tracking Systems: TECH has a large implementer team and each organization on the team maintained separate databases with TECH contractor and project information. It was labor-intensive and challenging to combine data exports from the various organizations because there were not unique identifiers included across all datasets. Data tracking was also complicated by the fact that many contractor companies enrolled under one name but were “doing business as” a different company name.

There were other data challenges within each dataset that prevented additional analyses that could inform design changes or illuminate market dynamics. Documentation of changes to incentive amounts and eligibility qualifications existed across emails, webpages, and documents. Consequently, it was difficult to be sure that we had all of the information or the most up-to-date information. In addition, the datasets did not maintain consistency in the definition of multifamily, with many single-family projects labeled as multifamily. Further, the datasets did not include show exactly what incentives were applied to a project (for example, the panel upgrade incentive was not in the dataset).

Additionally, some data were omitted that would have been useful to include, such as the customer’s electric utility or whether an HVAC customer had central AC prior to their TECH project. We also found some gaps in quality assurance processes. For example, we found that 4% of HPWHs were recorded as being installed on electric-resistance mode. If Energy Solutions had included a review of this field in the QA procedures, they could have followed up with contractors to educate them about proper operational modes. We also found some cases where the recorded electrical panel amperage was higher before a panel upgrade than after. Energy Solutions indicated that panels could be marked as having an upgrade when the amperage did not change, but they said the amperage should not be reduced.

Conclusions and Recommendations

We offer the following conclusions and recommendations. We shared many of these recommendations with Energy Solutions as they emerged, and they have already addressed or are actively taking steps to address many of them.

Conclusion: California has set ambitious climate goals and heat pump technologies are key to reducing California’s GHG emissions that contribute to indoor and outdoor air pollution. The investment in the TECH Initiative and work by the Energy Solutions Team has led to a brand identity, infrastructure, and network that can be leveraged for additional heat pump deployment. The TECH incentive stimulated heat pumps sales, contractor revenue, and market growth and, without the incentive, contractors reported their installations of heat pumps have slowed.

• • Recommendation: Given the momentum established in the heat pump market through TECH, it is disruptive to the entire supply chain when a much-anticipated program runs out of incentives so unexpectedly. It is also detrimental to the market transformation goals of TECH. We recommend that additional funding be made available to support the continuation of the incentive portion of TECH.

Conclusion: The TECH incentive strategy, as originally designed, was too complex. The complexity of baseline and enhanced incentives caused confusion for contractors, customers, implementers, partner program staff, and evaluators. In some cases, it also led to contractors promising incentives to customers for which they were not eligible for, with negative financial implications to the contractor.

• • Recommendation: The Energy Solutions Team was already in the process of streamlining enhanced incentive offerings when the program largely ran out of funds. We support Energy Solutions’ effort to continue simplifying the incentive offerings. We recommend that Energy Solutions ensure the zip code lookup tool is accurate and up to date.

Conclusion: Initial research in California and nationally, suggested that heat pump markets are in a nascent stage of development. The TECH Baseline Market Assessment estimated that the installed base for air-source heat pumps was 4%, while HPWHs comprised 2% of water heaters in multifamily homes and 0.4% of water heaters in single-family homes in California. The incentive application data from TECH; however, suggests that the market is open to switching to heat pumps, especially for space conditioning when incentives are generous. This trend recently occurred in other heat pump programs with generous incentives. For example, Con Edison’s Clean Heat rebate program six-year program budget was oversubscribed by $528 million in just six months. Denver, Colorado’s electrification program also expended all of its heat pump incentives in three months.

• • Recommendation: Additional research needs to be conducted to identify the right incentive levels to promote market transformation while ensuring program and market stability. Anecdotal information in this report, that must be interpreted cautiously due to small sample sizes, indicates the reduction of the incentive to $1,000 for HVAC heat pumps in the Southwest Gas territory does not seem to have reduced applications. A few contractors reported that if incentives were lowered to $1,000; however, they would be unlikely to participate due to the perceived high effort necessitated by the application requirements.
  • Recommendation: Another key consideration when designing the next phase of TECH incentives is the recent passage of the Inflation Reduction Act. This bill invests in the reduction of carbon emissions in multiple ways, including a 30% tax credit up to $2,000 available to anyone who purchases and installs a heat pump, in addition to rebates of up to $8,000 for low- and moderate-income households earning less than 150% of the local median income. While the specifics of when and how the rebates will be dispersed is unknown, we recommend these benefits be factored in when optimizing the TECH incentive design.

Conclusion: There was an imbalance in the equipment incented by and installed through the TECH Initiative. A large majority of projects involved HVAC heat pumps, while a minority included HPWHs. Our baseline market assessment and this process evaluation found that the HPWH market is underdeveloped compared to the HVAC heat pump market, which elevates the need for interventions if TECH is to transform the HPWH market. At the same time, another California program, SGIP (Self-Generation Incentive Program Generation), will be making $84.7 million available for HPWH retrofit incentives in California beginning in 2023.

• • Recommendation: The CPUC and Energy Solutions should weigh the costs and benefits of using TECH Initiative funds to stimulate the HPWH market. Investment in the market is clearly warranted, but significant funds are already earmarked for incentives. As such, TECH may be better positioned to stimulate the HPWH market through WE&T activities and supply chain engagement rather than end-user incentives.

Conclusion: Customers reported that the majority of contractors explained how to use their new heat pump, answered questions satisfactorily, and showed up on time. However, fewer contractors reportedly explained how to perform equipment maintenance. Proper equipment maintenance is essential to ensuring equipment performs to manufacturer specifications and ensures persistence of maximum energy savings and GHG reductions.

• • Recommendation: Consider developing a customer leave-behind that contractors could use to help educate consumers on equipment maintenance.
  • Recommendation: Since many contractors have only been working with heat pumps over the past few years, assess if contactor training is needed on optimal equipment maintenance practices.

Conclusion: Some contractors communicated distrust for TECH, most specifically regarding delays in reimbursement, lack of clarity around incentive structure, poor communication, and lack of responsiveness to inquiries.

• • Recommendation: Earning back the trust of contractors needs to be a focus moving forward for the TECH implementation team. Establishing service level/response time goals and tracking these metrics over time will not only support rebuilding trust with contractors, but also help determine necessary staffing levels to ensure the implementation team is responding to contractor questions and inquiries and paying incentives in a timely fashion. Potential metrics include average time elapsed from question submission to the first response provided; time elapsed from the question submission to the time inquiry is closed, average time elapsed from initial incentive application to payment; and contractor satisfaction with TECH support and the incentive payment process.
  • Recommendation: Using the incentive application database coupled with contractor support experience, continue to update FAQs with answers to common questions and develop a document of contractor tips and tricks to streamline the incentive application process and accelerate reimbursement.
  • Recommendation: When communicating key initiative changes, such as incentive amount changes or changes in incentive availability, email communications and posting of email content to websites is insufficient. Consider using multiple channels to communicate changes that enable confirmation of receipt of information.

Conclusion: The TECH implementer has room for improvement in their data tracking systems and data management processes. Omissions in data tracked, inability to link data across different systems, and lack of a comprehensive data dictionary limited the ability for implementers to streamline the process of developing content for the public reporting website, perform more real-time QA/QC, and develop insights to inform program design modifications. This is of particular concern if additional incentive dollars are made available to the TECH Initiative.

• • Recommendation: Develop a unique identifier for all TECH databases to facilitate linking of the contractor enrollment database, the incentive application database, and the learning management systems. Develop a single database where information from the disparate databases is automatically uploaded on an established cadence. When we shared this early feedback with the Energy Solutions Team, prior to this report being published, they quickly began addressing this issue and are making great strides in making this a reality.

Conclusion: According to the TECH incentive application database, contractors indicated that in 4% of projects they set the HPWH on electric resistance mode. This negates many of the benefits of a HPWH.

• • Recommendation: The TECH implementer should add the HPWH operating mode to their Quality Assurance checklist staff use when reviewing incentive applications. The reviewer should confirm the HPWH mode is either set on heat pump or hybrid mode and follow up with contractors if it is recorded as being set to electric resistance mode. When following up with contractors, we suggest exploring the motivations for this choice to ensure this is not masking an equipment issue.

Conclusion: The incentive application requirements made it difficult for contractors to be reimbursed in a timely manner. Contractors found the incentive application was time-consuming and asked for too much information.

• • Recommendation: Since there are a limited supply of HERS raters, consider a pilot program to recruit and train HERS raters, potentially focusing on underrepresented populations. Increasing the number of HERS raters support both quality installation and market transformation goals of the TECH Initiative.
  • Recommendation: Reassess messaging around the documentation requirements. Make it clear why each data point is needed and how it is in service of reaching California’s ambitious clean energy goals.
  • Recommendation: Consider ways to speed up the reimbursement process. One possible option is to utilize mobile payment services instead of payment via paper check.

Conclusion: In the first eight months of the TECH Initiative, the majority of optional technical trainings focused on HVAC heat pumps. There was only a two-hour webinar that focused on the technical installation of HPWHs, offered three times between April and July.

• • Recommendation: Augment the training offerings to include more HPWH technical content. We communicated this early finding to the Energy Solutions team, and they are taking it into consideration as they are developing the training plans for year two of the TECH Initiative.

Conclusion: TECH-sponsored trainings increased contractors’ confidence working with heat pumps and improved their awareness of building electrification’s role in decarbonization. Yet, training attendees indicated they had the least confidence in sizing heat pumps, and post-installation findings indicated contractors have room for improvement in their sizing practices.

• • Recommendation: Proper equipment sizing is essential for customer satisfaction, efficient equipment performance, and the ability for heat pumps to act as a grid resource. We recommend the implementation team pay special attention to improving contractors’ skills and confidence around heat pump sizing in their plans for year two of TECH trainings.

Conclusion: The majority of surveyed contractors were aware of the optional training available. Only a minority of contractors took advantage of the free training offerings; however, because contractors had already attended similar training or found the locations and times to be inconvenient.

• • Recommendation: Given that most contractors have installed a relatively small number of heat pumps, contractors may not know what they don’t know and thus are unable to identify their own training needs. Consider providing a self-scoring assessment to help contractors identify knowledge and skill gaps. Areas of focus should include heat pump sizing best practices, effectively sizing heat pumps to maximize grid benefits, and installation best practices. This could help contractors better ascertain if these training offerings were redundant with past trainings they had taken.

Conclusion: Embedded evaluation has been shown to be a valuable approach as the Evaluation Team has been able to provide suggestions for modifications that the Implementation Team acted on quickly. Lack of involvement during the planning phase of the initiative however, meant we were playing catch up and trying to learn the strategy underlying the program theory as it was unfolding. In addition, not being involved in the contracting process for training subcontractors prevented us from discussing data collection requirements prior to the trainings. This led to delays in fielding surveys, which resulted in missed opportunities.

• • Recommendation: Contracting the Evaluation Team at the same time as the Implementation Team would maximize the benefits of embedded evaluation and potentially make better use of TECH funds.
  • Recommendation: Since we were meeting with the Implementation Team weekly and corresponding via email and ad hoc calls multiple times a week, the Evaluation Team did not conduct staff interviews. A structured data collection process, especially when starting work after the Implementation Team, is a necessary component of evaluation. We will be incorporating formal staff interviews every six months moving forward.
  • Recommendation: Consider consulting with the Evaluation Team when contracting with implementation subcontractors so that evaluation data needs and data collection methods are optimized and planned for prior to implementation activities.

Conclusion: Embedded evaluation enabled real-time participant research that resulted in above-average response rates. Collecting insights from larger sample sizes closer to the time of the intervention gave us more confidence in the validity of our findings and greater ability to uncover variations among participants. Having robust data is particularly important for pilot programs to inform the most effective strategies for scaling to a state as diverse as California, which aligns with the vision of SB 1477. The timely and robust insights from embedded evaluation accelerate the ability to scale pilots into full-scale programs.

• • Recommendation: Consider expanding the use of embedded evaluation in pilot programs outside of BUILD and TECH.

Conclusion: Some contractors mentioned a challenge with the definition of multifamily buildings (five or more units) and that the definition does not align with the BUILD Program’s multifamily definition.

• • Recommendation: Work with the BUILD Program Implementation Team to align the definition of multifamily for both programs.

1. Introduction

The State of California has established ambitious climate goals. In Senate Bill (SB) 1477 (Stern, 2018), the legislature finds that the “electricity and heating fuels used in buildings are responsible for a quarter of California’s greenhouse gas emissions and contribute to indoor and outdoor air pollution.”[1] As part of meeting the State’s climate goals, SB 1477 requires the California Public Utilities Commission (CPUC) to develop two programs, designed to test two specific approaches to building decarbonization. The two SB 1477 pilot programs are the Building Initiative for Low Emissions Development (BUILD) program and the Technology and Equipment for Clean Heating (TECH) Initiative. The BUILD Program is designed to provide incentives to eligible applicants for the deployment of near-zero-emission building technologies in new construction to significantly reduce the emissions of greenhouse gases from buildings. The TECH Initiative is designed to advance the state’s market for low-emission space and water-heating equipment for existing residential buildings.

As stated in D. 20-03-027, the decision “Establishing Building Decarbonization Pilot Programs,” “the BUILD Program and TECH Initiative are building decarbonization pilot programs intended to raise awareness of building decarbonization technologies and applications, test program and policy designs, and gain practical implementation experience and knowledge necessary to develop a larger scale approach in the future.”[2] The BUILD Program and TECH Initiative are funded through the California Air and Resources Board (CARB) Cap-and-Trade Program funds. BUILD and TECH spending must conform to the proportions of gas utility service territories from which the funds are derived.

In April 2020, D. 20-037-027 directed the CPUC’s Energy Division to select the implementer for the TECH Initiative. Following a rigorous competitive bidding process, the CPUC, via a scoring committee, selected Energy Solutions as the prime contractor of the TECH Initiative. Energy Solutions entered into a Master Consulting Services agreement with Southern California Edison Company (SCE), the administrator of the BUILD Program and TECH Initiative, on May 20, 2021.

Similarly, in April 2020, D. 20-03-027 directed the CPUC’s Energy Division to select a single evaluator for the TECH Initiative and the BUILD Program. The role of the program evaluator is to measure the impact of program activities, as well as qualitatively assess the success and scalability of the programs’ strategies. This should occur in as close to real time as possible so that timely, substantive feedback can be used to change course when and if appropriate, and to ensure the success of these pilot programs. The scoring committee selected Opinion Dynamics as the embedded evaluator for BUILD and TECH.[3] Opinion Dynamics entered into a Master Consulting Services Agreement with SCE on June 28, 2021.

This report is an interim process evaluation of the TECH Initiative, publicly known as TECH Clean California, conducted approximately eight months after TECH officially launched its incentives on December 7, 2021, but about a year since TECH implementation staff began program design and planning.[4] The TECH Initiative was designed to be implemented over four years; however, incentive funding was largely exhausted in the first six months of the initiative. This interim process evaluation assesses program design and implementation with a focus on the contractor experience, the customer experience, program operations, and whether program activities have been implemented as intended. The objective of this interim process evaluation is to illuminate opportunities for program optimization and identify lessons learned to inform program scalability.

[1] SB 1477 was codified as Public Utilities (Pub. Util.) Code Section 748.6, Section 910.4, and Sections 921-922.

[2]https://docs.cpuc.ca.gov/PublishedDocs/Published/G000/M331/K772/331772660.PDF

[3] Embedded Evaluation is based on the principles of Developmental Evaluation as defined by Dr. Michael Quinn. It is an approach to insert evaluation within the program design, implementation, and reporting processes to support evidence-based decision-making as well as informed ongoing decision-making.

[4] TECH also partnered with heat pump programs already in place for the soft launch of TECH incentives in September 2021 to leverage the other programs’ momentum.

1.1 Program Description

The TECH Initiative is a $120 million program designed to help advance the state’s mission to achieve carbon neutrality by driving the market adoption of low-emissions space and water-heating technologies for existing single-family and multifamily residential homes. The initiative was created as part of California Senate Bill 1477 and is funded by revenues collected through California’s Cap-and-Trade program. Through a combination of market incentives, supply chain engagement, workforce development, consumer education, regional pilots, and Quick Start Grants, the initiative installs low-emissions space and water-heating technologies across California homes.

The initiative’s overall goal is full-scale market transformation of the heat pump market in California to ensure a thriving market for clean heating technologies in the next 10 years. In D. 20-03-027, Ordering Paragraph 32 states, “The TECH Initiative implementer shall approach the initiative with a menu of tactics… but shall implement the upstream and midstream market approach as well as provide consumer education, contractor training, and vendor training, to drive market development.” The Decision also goes on to state that “market development initiatives involve phases that require development and testing of strategies and approaches to arrive at impactful market intervention efforts.” The Decision gives the TECH implementer flexibility to determine and test specific tactics while also addressing the statutory mandates in SB 1477. In accordance, Energy Solutions, the TECH implementer, employed a combination of market incentives, supply chain engagement, workforce development, consumer education, regional pilots, quick start grants and public data reporting.

1.2 Current Program Status

In spring of 2022, Energy Solutions staff recognized that TECH incentives were being claimed at a much higher rate than anticipated and would soon be fully subscribed. At the time of this report, all TECH single-family incentives had been claimed except for those in Southwest Gas territory, which represents a tiny portion of TECH activity (limited to 1.63% of TECH incentive funds based on CARB funding rules) and some heat pump water heater (HPWH) funds in SoCalGas territory (49.26% of TECH incentive funds). The California Energy Trailer Bill, released on August 31, 2022, allocated an additional $50 million for TECH in 2023. At this critical juncture, Opinion Dynamics and the CPUC recognized the need to conduct interim research. This interim process evaluation examined program processes, project characteristics, as well as the customer and contractor experience to inform any modifications or improvements TECH could make if it were to receive additional incentive funds. While single-family incentives had largely been claimed, other TECH activities are ongoing, such as the pilots, Quick Start Grants, consumer education on electrification, contractor engagement, contractor training, and multifamily incentives. Upcoming evaluation work will examine the ongoing TECH activities.

1.3 Research Objectives

The primary objectives of this interim process evaluation were to investigate program processes in the first eight months of the TECH Initiative as they related to the contractor and customer experience. Together, these lines of investigation illuminate how TECH has unfolded and describe how the California markets have responded to the TECH incentive offering thus far. Specific objectives included:

• • Evaluating the TECH implementer’s program processes for contractor enrollment, contractor communication, incentive design, incentive processing, database management, and quality assurance procedures.
  • Investigating contractor experiences related to enrolling in TECH, applying for incentives, promoting the initiative, communicating with TECH staff, and attending optional workforce development trainings.
  • Understanding single-family customer experiences related to finding a TECH contractor, deciding on a heat pump, getting the equipment installed, and paying for the equipment.
  • Conducting an assessment of TECH projects to understand their locations, costs, incentive layering, and characteristics such as HVAC heat pump efficiencies, water heater capacities, or whether a panel upgrade was also completed.

2. Methods

Opinion Dynamics is serving as the embedded evaluator of the TECH Initiative. We have been staying in close communication with Energy Solutions as they implement TECH. We have weekly meetings with key implementation staff to stay up to date on changes and activities related to contractors, projects, trainings, and other activities, such as the pilots. These meetings allow us to ask questions and get clarification on processes and metrics. We established secure file transfer protocols and receive regular data exports on these topics. Having up-to-date data on the initiative allows us to conduct our evaluation activities alongside initiative implementation, so we may deliver insights and feedback in a timelier manner than happens in typical retrospective evaluations.

For this interim process evaluation, Opinion Dynamics used a mixed method approach that relied on a combination of surveys and interviews and a review of the TECH incentive application database of closed projects. We conducted online surveys with enrolled TECH contractors, contractors who attended TECH-sponsored trainings, and single-family property owner customers who received an incented heat pump. We also interviewed a small sample of heat pump manufacturers, enrolled TECH contractors with a high volume of projects, and customers who indicated some issue with their heat pump installation in their survey response. All data collection instruments can be found in Appendix A. Below we explain each of these methods in more detail.

2.1 TECH-Enrolled Contractor Survey

Opinion Dynamics surveyed contractors enrolled in TECH Clean California to gather information about their experience with the initiative. The objectives of the contractor feedback survey included the following:

• • Understand how contractors first learned about TECH and ease of the enrollment process
  • Describe contractors’ understanding of various TECH eligibility criteria and availability of eligible product types
  • Characterize the ease of applying for incentives through TECH, including completion of Multifamily Incentive Reservation forms and process of layering TECH incentives with those offered through other programs
  • Assess contractors’ awareness and rate of enrollment in optional trainings
  • Determine influence TECH has had on heat pump sales
  • Understand TECH communication methods contractors find most useful

Our sample frame included all 907 California contractors who were enrolled in TECH as of June 13, 2022. We invited these market actors to take the survey via email, contacting them up to three times. The online survey was open from June 21, 2022, to July 5, 2022. We received a total of 184 survey completes which translates to a response rate of 21%. Respondents received a $30 e-gift card as a token of appreciation for taking the time to participate in this research.

2.2 Post-Training Contractor Surveys

All TECH-enrolled contractors participated in an onboarding training to learn about TECH initiative and the incentive processing system, called IRIS for single-family submissions. TECH also offered optional workforce development trainings designed to improve contractor comfort and familiarity with heat pumps and electrification as a business model. In our evaluation, we use Kirkpatrick’s Adult Training Evaluation Model (Figure 1), the gold standard for evaluating adult learning interventions, to assess participant reactions to optional TECH training interventions.

As illustrated in Figure 1, Kirkpatrick’s model consists of four levels:

• • Level 1 – Reaction: measures how participants feel about the learning experience. The value of Level 1 is that a good training experience improves knowledge transfer.
  • Level 2 – Learning: measures the degree to which participants change attitudes, increase knowledge, or enhance skills as a result of the learning experience. The value of Level 2 is to demonstrate that learning occurs as a result of the training.
  • Level 3 – Behavior: measures the degree to which participants apply what they have learned outside of the learning environment. This level seeks to demonstrate whether trainees take the information they learn and apply it.
  • Level 4 – Results: the degree targeted outcomes are achieved system-wide. In this study, we measured the training’s results in terms of energy savings. The value of measuring Level 4 is to inform the return on training investment realized from the training endeavor.

The objectives of the survey fielded to the contractors who attended an optional workforce development training were to gauge how they felt about their learning experience. Topics included the quality of the material, the instructor, what they learned, and how valuable they thought the training was.

Twenty-three optional trainings were offered between January and June 2022 across the three implementers, the National Comfort Institute (NCI), Electrify My Home (EMH), and the Association for Energy Affordability (AEA).[5] To assess contractor knowledge and satisfaction after each training, implementers fielded a post-training Level 1 reaction survey following each course. One challenge we experienced as embedded evaluators was that the trainings were being implemented before we could design our survey and before data collection methods had been agreed upon with the training implementers. One best practice that we identified out of this experience is that data collection methods, expectations and roles should be discussed as part of the contracting process with the evaluator reviewing that portion of the contract language. We tailored each survey to the learning objectives of each training. Therefore, the evaluation team created four different survey instruments.

Administration of the surveys differed by the training implementer. The evaluation team programmed and hosted the NCI-tailored survey and provided survey links to NCI who then delivered them by email to trainees. Our NCI-tailored surveys were not ready for fielding until May; therefore, those who attended a training earlier in the year were not sent the survey immediately after their training. Due to the extended period of time that occurred between the completion of courses in January and February and when the survey link was delivered, we observed a lower response for surveys related to the earliest trainings.

EMH developed and programmed a short survey they implemented to trainees who attended a course between January and March. In April, EMH programmed the EMH-tailored survey developed by Opinion Dynamics into AEA’s learning management system (LMS). The advantage of this approach was that the survey was available immediately after the training and attendees could take the survey on the same computer as the training. Unfortunately, EMH hosted only one training after programming the full Opinion Dynamics survey into their LMS. Therefore, any survey data we have for EMH trainees prior to April comes from the shorter survey EMH developed.

According to rosters provided by Energy Solutions, 180 people attended at least one day of optional training.[6] We received a total of 118 survey responses, but due to the inconsistencies across the two EMH surveys (i.e., survey developed by EMH versus survey designed by evaluation team), 62 completed surveys include responses to only a handful of questions. Consequently, the evaluation team used a limited number of total survey responses to run analyses for corresponding questions. Overall, we observed a 28% response rate to the post-training survey for NCI trainees, and 86% response rate for EMH trainees.

Table 2 displays all NCI and EMH trainings that took place between January and June 2022, along with the count of those who attended each training versus how many completed the post-training survey.

[5] AEA trainings were short, webinar-format trainings; we did not evaluate these trainings to the extent we did the in-person (and often multi-day) NCI and EMH trainings.

[6] Count excludes anyone the rosters indicated withdrew from training or was a no show for the entire length of the training.

2.3 Contractor Interviews

To capture more detailed, nuanced feedback, we selected a sample of TECH-enrolled contractors to complete phone interviews. In early June 2022, we identified the 21 contractors who had the greatest number of TECH projects for the sample. We ensured the contractors we spoke with represented both HVAC and water-heating projects and some had experience layering TECH incentives with another program’s incentives. We reached out to 21 contractors to invite them for an interview. We completed interviews with four contractors and provided them each a $60 gift card. Two contractors were in SDG&E territory, one was in BayREN/PG&E territory, and the other was in SoCalGas territory. Two of the four had experience with TECH-incented multifamily projects in addition to single-family projects.

2.4 Single-Family Customer Post-Install Survey

Opinion Dynamics conducted an online survey with single-family homeowners who received a TECH-incented HPWH or HVAC heat pump. The objective of the survey was to capture feedback on their motivations for the installation, factors that influenced their decision-making, and their experience with their TECH contractor. Respondents received a $5 gift card for completing the survey.

Opinion Dynamics sent 3,675 emails and received 968 completes, resulting in a response rate of 26%. We sent 2,975 emails to HVAC customers and 700 to HPWH customers, leading to 728 HVAC completes (24% response rate) and 240 HWPH completes (34% response rate). These survey findings reflect the experiences of customers who had a heat pump installed between December 8, 2021, and July 10, 2022.

As an embedded evaluator, we administered this survey on a rolling basis every two weeks as more customers participated, which has resulted in higher-than-normal response rates. This approach allowed us to make needed adjustments to the survey, which we have done three times. We added questions about whether the single-family property was owner- or renter-occupied, whether the customer had rooftop solar or had plans to get it, and whether the project passed inspection on the first visit. Since these questions were not asked of every single surveyed respondent, the sample size varies when using the results from these three questions.

2.5 Customer Follow-Up Interviews

When reviewing the customer survey data, we noted customers who indicated a problem or concern about their installation in the survey. We selected 41 of these customers and invited them to complete a phone interview where they could elaborate on the installation issue and offered a $25 gift card as a token of our appreciation. During the month of June 2022, we conducted phone interviews with 21 of these TECH customers. The interviews lasted an average of 30 minutes and were recorded with the respondent’s permission.

Ten of the customers received an HVAC heat pump and 11 received a HPWH. Their regional locations in California are shown in Table 3.

2.6 Incentive Application Database Review

Opinion Dynamics conducted an analysis of the TECH incentive application database of single-family projects. Opinion Dynamics assessed these projects from the first six months of the TECH Initiative to understand the following:

• • How many contractors are enrolled in TECH?
  • How many unique contractors are responsible for what percentage of TECH projects?
  • Where were TECH projects in California?
  • To what extent were TECH projects layered with another program’s incentives?
  • How many projects required panel upgrades?
  • What were the project costs and project lengths?
  • What additional tasks did the contractor perform when installing heat pumps?
  • Where in the home were HPWHs installed and did this differ by climate zone?
  • What were the capacities of the installed HPWHs?
  • What were the efficiencies of the installed HVAC heat pumps?

Energy Solutions provided Opinion Dynamics with a database of heat pump projects that received TECH incentives from December 7, 2021, through May 23, 2022. The database contained closed projects for which the incentive had been paid to the contractor; it did not include any open projects where the contractor was awaiting reimbursement for a heat pump installation. The database represented a total of 3,361 TECH-incented heat pumps: 2,711 HVAC heat pumps, and 650 HPWHs, which were installed in 3,117 California households. While 179 households received more than one HVAC heat pump, none received more than one HPWH; 2% of households received both an air source heat pump and HPWH.

In August 2022, Energy Solutions provided Opinion Dynamics a more comprehensive incentive application database that also contained open projects, where the contractor was still awaiting reimbursement for a TECH project. Table 4 presents the proportion of TECH claims that were paid and remain unpaid at the time of the data extract, with a breakdown of how long unpaid claims have been open. To determine the length of time a claim has been open, we calculated the length of time between the date the incentive application was submitted and the date of this data extract, which was August 10, 2022.

Nearly a quarter (22%) of submitted claims remain unpaid as of August 10, 2022. The majority of unpaid claims (1,757 of 2,246; 78%) had been open for more than six weeks; the oldest claim submission dated back to January 2022.

Our analysis examined results at both the equipment and household level. We also assessed installations in and out of disadvantaged communities (DACs) and by climate zone. Where appropriate, we assessed for statistically significant differences between groups using a t-test. Statistically significant findings are indicated with an asterisk, which represents a p-value below or equal to 0.05.

2.7 Manufacturer Interviews

The TECH Initiative staff engage with heat pump manufacturers by holding regular meetings with them. Early in the TECH Initiative, TECH staff collaborated with manufacturers to recruit contractors from the manufacturer’s contractor network. Later in the TECH Initiative, these meetings focused on upcoming policy or code changes that could affect what heat pump equipment is eligible for incentives and how much incentive funding is still available in TECH.

We spoke with three HVAC heat pump manufacturers and two HPWH manufacturers supporting the TECH Initiative. The goal of these interviews was to capture the manufacturer’s perspective of the impact TECH has had in the market and collect insights about effective program design. The phone interviews lasted between 20 and 30 minutes and were conducted in late June and early July of 2022.

3. Program Theory and Design

Opinion Dynamics is serving as the embedded evaluator for the TECH Initiative. One of our first evaluation activities was to work with the prime program implementer, Energy Solutions, to create a Program Theory Logic Model (PTLM) that explicates the TECH Initiative’s activities, outputs and intended market and program outcomes. We also developed key performance indicators (KPIs) that, when measured, can demonstrate whether the intended outcome was achieved. Examples of KPIs include measures of how contractors’ heat pump promotion strategies have changed, or measures of improvements in customer familiarity with heat pump benefits.

Our evaluation approach is grounded in the PTLM of the TECH Initiative to ensure the program and market metrics are in direct alignment with the theory and logic of the TECH Initiative.

3.1 Program Theory and Logic Model

The PTLM activities, shown in Figure 2, are listed from left to right and the outcome categories are listed in chronological order from top to bottom. The PTLM features arrows labeled with numbers that represent linkages between activities, outputs, and outcomes.

3.2 TECH Team and Roles

The TECH Initiative’s prime implementer, Energy Solutions, assembled a team to help them implement the initiative’s activities. Table 5 shows a list of organizations who support TECH and the role they play in the Initiative.

3.3 Contractor and Customer TECH Eligibility

In this section, we discuss the eligibility requirements for contractors and customers in the TECH Initiative.

Contractor Eligibility

TECH eligibility for California contractors included licensure and insurance requirements. Any contractor receiving incentives through TECH, whether single-family or multifamily incentives, or those offered through the pilots and Quick Start Grants, must enroll in TECH and meet the requirements. Initially the requirements were as follows:

• • Licensure requirements
    • The General B, C20, and C36 are eligible for HPWH incentives.
    • General B and C20 are also eligible for HP HVAC incentives.
  • Minimum insurance requirements included as explained in the contractor application form
    • Commercial general liability insurance with coverage of at least $1 million each occurrence/$2 million aggregate for bodily injury, property damage and personal injury (such coverage to be afforded utilizing one or more commercial general liability and/or umbrella liability policies).
    • Workers’ compensation insurance with limits no less than as required by applicable laws and employers’ liability insurance no less than $1 million per event of injury or death each accident.
    • Coverage at least as broad as the Insurance Services Office (ISO) covering automobile liability with limits no less than $1 million each accident for bodily injury and $500,000 each accident for property damage.

On August 17, 2022, TECH changed its insurance requirements for participating contractors to align with the Contractors State License Board’s (CSLB’s) insurance requirements. TECH staff found that the initial insurance requirements, which were more than required by the CSLB, were a financial burden on smaller contractors. The insurance requirements post-August 17, 2022, are below:

• • Liability insurance, with an aggregate limit of $1 million, for licenses with five or fewer persons listed as personnel; plus, an additional $100,000 for each additional personnel listed on the license; not to exceed $5 million total (LLC licenses only)
  • Workers’ compensation insurance as dictated by CSLB requirements
  • In addition, all [insurance] policies have a financial rating of at least “A-” and a financial size category of at least “VII” on the most current edition of AM Best’s Key Rating Guide or a secure rating by another generally recognized rating agency

This change means contractors no longer have to increase their insurance coverage in order to participate in TECH, which should allow more small contractor companies to participate moving forward. The license requirements did not change.

Customer Incentive Eligibility

Customers were eligible for TECH incentives if they lived in one of the gas IOU territories. Customers served by the City of Long Beach, Channel Islands Gas, Palo Alto or residing outside of IOU gas service territories were not eligible for TECH Incentives. Contractors looked up customers by their zip code to determine if they were eligible. The zip code look-up tool, housed on the Switch is On website, identified baseline and enhanced incentives for which the customer was eligible.

3.4 Budget

Building decarbonization pilot program funding is authorized and financed pursuant to SB 1477. SB 1477 makes available $50 million annually for four years,[7] for a total of $200 million, derived from the revenue generated from the GHG emission allowances directly allocated to gas corporations and consigned to auction as part of the California Air Resources Board’s (CARB’s) Cap-and-Trade program.[8] D. 20-03-027appropriates 40 percent of the $200 million budget for the BUILD Program and 60 percent for the TECH Initiative.

Since TECH is funded through CARB’s Cap-and-Trade Program funds, TECH spending must be proportionally directed to the gas corporation service territories from which the funds are derived. The proportion allocation of TECH Initiative spending must be consistent with each gas corporation’s allocation of Cap-and-Trade allowances:

• Southern California Gas Company (SoCalGas): 49.26%
• Pacific Gas and Electric Company (PG&E): 42.34%
• San Diego Gas and Electric Company (SDG&E): 6.77%
• Southwest Gas Corporation (SWG): 1.63%

Any spending for TECH with statewide or cross-territory benefits, including but not limited to administrative and evaluation spendings, must be attributed to the gas corporation service territories in proportion to their original funding contribution.

TECH Implementation Budget

By design, TECH is a multi-faceted initiative which includes many activities (Table 6). Budgets are split out between funds for implementation of the initiative’s activities and those for incentives. The TECH staff re-evaluate these budgets on an ongoing basis depending on actual spends and needs.

[7] Fiscal Year (FY) 2019–2020 to FY 2022–2023.

[8] Four gas corporations currently participate in California’s Cap-and-Trade program: Southern California Gas Company, Pacific Gas & Electric Company, San Diego Gas & Electric Company, and Southwest Gas Corporation.

3.5 Incentive Design

TECH began the incentive portion of the Initiative with a soft launch in September 2021 and a full-scale launch on December 7, 2021. The soft launch targeted areas with existing HPWH programs (the Bay Area) and areas with recently ended HPWH programs (Southern California) to maintain market momentum. Initially, TECH did not require a contractor to pass the rebate through to the customer. This requirement changed in March 2022 and was communicated to contractors via email.

The full launch of the TECH Initiative employed a complex incentive structure. The structure was designed to encourage incentive layering and manage the budget according to CARB allocations. For single-family retrofit projects, baseline incentives were available in all TECH-eligible zip codes; and enhanced incentives were available in areas where another program was already offering heat pump incentives and partnering with TECH. We refer to projects that received enhanced incentives as those with layered incentives because two programs’ incentives were layered together. The TECH Initiative also offered quality installation incentives if contractors performed additional tasks. For new construction and retrofit multifamily projects, incentives were also available. In addition, TECH offered bonus incentives for attending an optional workforce training and subsequently submitting qualified applications. We discuss each of these incentive types in more detail below.

Single-Family Baseline and Enhanced Incentives

TECH offered residential incentives for existing single-family properties (1–4 units). Table 7 lists out single-family HVAC baseline and enhanced incentives. The TECH-incented HVAC heat pump had to replace an existing non-heat pump heating source or replace just the heating element of that system. The non-heat pump heating source could have included propane, natural gas, wood burning, electric resistance, or solar-heated appliances. Furnaces eligible for replacement included but were not limited to, central furnaces, wall furnaces, forced-air furnaces, electric baseboard furnaces, and wood-burning stoves.

Table 8 describes single-family HPWH baseline and enhanced incentives. These varied depending upon the existing equipment and the size of the HPWH.

Single-Family Quality Installation Incentives

There were also incentives available if contractors performed additional tasks to ensure quality installation. On HVAC projects, these were completing Manual J calculations, duct testing and sealing, and field measured performance (Table 9).

Beginning May 14, 2022, quality installation HVAC incentives were lowered to $300 each and the HVAC heat pump incentive was lowered to $1,000 per unit.

On HPWH projects, quality installation incentives were offered for enrolling a customer in a demand response program, adding a thermostatic mixing valve, and completing a panel upgrade (Table 10).

Multifamily Incentives

TECH offered incentives for multifamily properties with five or more units. Incentives were provided for HPWH and heat pump HVAC equipment installed in multifamily properties for both retrofit and new construction applications through December 31, 2021. After that date, multifamily incentives were available for only retrofit scenarios because new construction projects could apply for incentives through the BUILD Program in 2022. Incentives were available for various equipment types that serve residential apartments and communal spaces used by residents and multifamily building staff. Projects could range from a single unit changeout for a single apartment unit, to the replacement of a large central system that served multiple units at once. Multifamily incentives could be utilized as stand-alone incentives, layered with other non–TECH Initiative multifamily incentives for which the project may be eligible, or used alongside incentives from TECH pilots targeting multifamily properties.

TECH used an incentive reservation system for multifamily projects, given the long pre-construction and construction timelines. TECH’s multifamily offering limited participating contractors and property owners to a maximum of $3 million of incentives aggregated across all multifamily incentive programs and pilots. This cap was in place to ensure the incentive budget was spread adequately across enough participants to facilitate market transformation.

Table 11 defines the multifamily HVAC incentive structure.

Table 12 describes the multifamily HPWH incentive offerings.

Some multifamily projects were eligible for an additional incentive for electrical panel upgrades. This incentive required that the apartment unit received a TECH-funded HVAC heat pump or HPWH and was all-electric after upgrades were made (Table 13).

Contractor Bonus Incentives

Contractor organizations were eligible to receive separate bonus payouts for their staff participating in one of NCI or EMH’s multi-day trainings and subsequently submitting a minimum number of qualifying TECH applications. When funds were largely exhausted on May 13, 2022, the bonus incentives were no longer offered.[9] Technicians were able to receive training for participating in NCI training or EMH training. If they attended both, they received incentives for each day of training attended. Sales staff were only eligible to receive a single bonus as only the EMH training was applicable to their role. Bonus incentives are described in more detail in Table 14.

[9] The contractors who had previously signed up for NCI’s training on 5/17–5/18 still received the bonus since they committed before the suspension.

Incentive Uptake and Changes

At the end of March 2022, the TECH implementation team noticed rapid uptake of single-family incentives. As of April 5, 2022, contractors in SDG&E territory had claimed 67% of incentives available to them. As shown in Table 15, between the end of April and mid-May, the incentives claimed more than doubled in PG&E territory and nearly doubled in SoCalGas territory. The TECH team reportedly began checking the incentive budget twice daily to stay informed of how many funds remained.

They acted quickly and instated numerous changes to the incentives. The changes and when they were communicated to enrolled contractors are summarized in Figure 3. These changes are explained in more detail following the figure.

On April 20, 2022, TECH staff emailed contractors announcing changes to the incentive structure, some of which would take effect on May 31, 2022, and others to take effect on June 20, 2022. The most important information in this announcement was the suspension of all TECH incentives in SDG&E territory by May 31, 2022, or earlier if funds were exhausted before then. Changes that would go into effect June 20, 2022, included the suspension of the $200 Thermostatic Mixing Valve incentive for HPWHs in the Bay Area, the start of flat-rate incentives instead of the baseline and enhanced combination, and the elimination of the panel upgrade incentive. TECH also announced a new $50 incentive for enrolling a customer in a demand response program, which would go into effect June 20, 2022. 

Two and a half weeks later, on May 6, 2022, contractors received another email from TECH letting them know that all incentive funds for SDG&E were exhausted. If contractors had discussed the incentive with customers but not yet reserved funds, that customer would then not be able to get the incentive. This email message also introduced a new incentive reservation system that required contractors to upload a signed customer contract with an installation date to the incentive processing portal to reserve the incentive funds for clients. The deadline to reserve incentives at the original amounts was moved up from June 20, 2022, to May 13, 2022. Hence, contractors received a message on May 6, 2022, telling them they had to reserve incentives for all their existing projects within a week in order to receive the larger original TECH incentive amounts. To reserve an incentive for a project, the contractor must create an incentive application and enter the customer and product information but waits to submit the application until the equipment has been installed and all other application requirements are met.

Less than a week later, on May 12, 2022, TECH staff sent a final email to contractors announcing the exhaustion of funds for PG&E territory and HVAC heat pump funds in the SoCalGas Territory (HPWH funds remained in SoCalGas territory). As a result, all incentives for PG&E and HVAC heat pump incentives for SoCalGas were immediately suspended and any project submitted after 5:22 p.m. PDT on May 12, 2022, was waitlisted. The submission deadline for all other incentive funds was set to May 13, 2022, after which no funding would be available, except in Southwest Gas Territory. At the same time, TECH also revised the incentive amounts for single-family customers in Southwest Gas territory as shown in Table 16. The TECH staff worked with the CPUC to repurpose program funds to pay additional incentives and ensure contractors with in-process projects were made whole.

The email language regarding incentive and program changes was posted on a contractor-facing website. Best practice would be to distill the information into a table or figure to visually present the information in a more easily digestible way on the website and not just post email message content.

Single-Family Incentive Application Process and Reimbursement

TECH-enrolled contractors complete an incentive application online, via a portal called IRIS. The application form requires information about the customer, the prior equipment, the heat pump equipment, additional project work such as panel upgrades, pictures of nameplates, and a copy of the permit and Home Energy Rating System (HERS) report. See Appendix B for a complete list of incentive application requirements. The process for incentive application and reimbursement is as follows:

3.6 Workforce, Education, and Training

One important aspect of the TECH Initiative is to provide contractor and vendor training to support the advancement of the state’s market for low emission space and water-heating equipment as called for in SB 1477. Workforce development via contractor training is a TECH activity designed to improve contractor familiarity and comfort with heat pump technologies. TECH partnered with NCI, EMH, and AEA to offer optional trainings.[10] The NCI trainings focused on the technical side of HVAC equipment. EMH’s training focused on the contractors’ business and whole-home electrification. AEA’s trainings focused on   heat pump systems in multifamily buildings, covering central and individual systems for space conditioning and water heating . A description of each training offered in the first half of 2022 follows:

• • NCI Airflow Testing and Diagnostics: Intended for HVAC contracting firm owners, managers, and technicians, this eight-hour class provides technical training on performing static pressure testing, how to professionally install static pressure test ports, and how to measure and interpret static pressures.
  • NCI Refrigerant-Side Performance: This residential and commercial certification class provides students with real-world lessons and hands-on training. It is based on proven techniques on how to best approach refrigeration-side issues. Students learn to apply NCI’s performance-based, systematic approach to refrigeration-side diagnostics, including strategies for mitigating non-refrigerant faults prior to attaching refrigerant gauges.
  • NCI Residential System Performance and Electrification: This 20-hour certification course teaches students how to test, diagnose, and improve total residential HVAC performance. This course features numerous hands-on demonstrations that include how to use the test instruments, proper testing locations, and live testing and interpretation of readings. This class is offered in both in-person and live webinar formats.
  • EMH Residential Space Conditioning and Water Heating Electrification: Designed for construction trade personnel of all levels, this three-day class informs students of near-term and far-reaching changes in the home building industry and driving forces of such, including California’s regulatory and legislative framework. Students gain in-depth knowledge about heat transfer mechanisms, functionality, and benefits of heat pumps in residential electrification, as well as how to transition from traditional gas heating to modern electric heat pumps without negatively impacting their bottom line. This course is offered in live format only.
  • AEA Electrification 101 Multifamily: This webinar reviews the benefits of electrification and policy and code considerations. It also provides an overview of what should be considered in an all-electric multifamily project (both new and retrofit), and explores electrification technologies for all end uses, as well as the role of solar PV and electric vehicles.
  • AEA Multifamily Electrician Retrofits and Assessments for Building Professionals: This webinar explores how to approach an electrification retrofit, what the constraints and opportunities there are for multifamily buildings with different configurations, and how projects are shaped by electrical infrastructure and available technology. Attendees work through example projects in class.
  • AEA Multifamily Electrification Retrofits for Property Owners: This course is aimed at educating multifamily property owners about what they need for a successful retrofit project. Attendees learn how to approach an electrification retrofit, what the team can look like, the constraints and opportunities for different property configurations, and how projects are shaped by electrical infrastructure and available technology. Attendees work through example projects in class.
  • AEA Individual Heat Pump Water Heater Installation: This two-hour webinar is aimed at educating contractors, consultants, and engineers on design considerations for individual HPWHs, particularly for multifamily developments. This course provides an overview of HPWHs, code requirements, and takes a deep dive into design considerations discussing sizing, venting configurations, load shifting, and installation best practices.

[10] Since the AEA trainings were short, webinar-format trainings, we did not evaluate them to the extent we did the NCI and EMH trainings.

4. Findings

This section presents findings related to contractor enrollment and their experience with the TECH Initiative; marketing efforts to support heat pump awareness and TECH incentives; customer decision-making and project characteristics; customer and contractor satisfaction with the TECH Initiative; as well as findings related to contractors’ experience with optional TECH-sponsored trainings. 

4.1 TECH-Enrolled Contractors

There was high interest in TECH among California contractors. According to Energy Solutions, TECH had enrolled 923 contractors by June 1, 2022 (Figure 4). TECH had enrolled nearly 900 contractors by May 2022, when the single-family and multifamily incentives had run out in most territories. It was announced incentives were exhausted in SDG&E territory on May 6, 2022, and gone in PG&E on May 12, 2022; incentives that remained after mid-May were HPWH incentives in SoCalGas territory and both HPWH and HVAC for SWG territory.

As of August 2022, the California regions serviced by the most TECH-enrolled contractors included the Greater Los Angeles area (236 of 965; 24%) and the Bay Area (111 of 965; 12%); although notably, a few contractors serviced multiple regions in additions to these two primary areas (Table 17). Please note, the received enrollment data was missing service regions for over a quarter of contractors (258 of 965; 27%) so information for these individuals is unknown.

Contractors with a C-20 HVAC Contractor license represented most TECH-enrolled contractors as of August 2022 (824 of 965; 85%) (Figure 5). It is important to note that the TECH enrollment report was missing licensure data for 7% of contractors (67 of 965), which is surprising because this a key eligibility requirement for enrollment in TECH.

The majority of TECH-enrolled contractors (645 of 965; 67%) worked for a corporation, while the next most common type of business was a partnership (Figure 6).

With the rapid uptake in incentives and future incentives reserved, TECH implementation staff had a flood of projects to process and review for quality assurance (QA) before issuing payment. This led to a backlog of incentives, which left contractors awaiting payment. Figure 7 displays the percent of contractors (y-axis) who had paid and unpaid TECH single-family incentive applications as of August 24, 2022. In the figure, please note that the number of projects (x-axis) is presented as categories, some covering a range (e.g., 1–5, 6–20).

Looking at the proportion of unpaid projects in the “0” and “1–5” categories on the x-axis, nearly three-quarters of contractors (482 of 655; 74%) who had submitted at least one incentive application to TECH as of August 24, 2022, had five or fewer TECH incentive applications that remained unpaid. Additionally, looking at the proportion of paid projects under the “0” category on the x-axis, one-fifth of contractors (134 of 655; 20%) had been paid for all submitted projects. Approximately one-third of TECH-enrolled contractors (310 of 965; 32%) had not submitted an incentive application to TECH by the date claim data provided to the evaluation team (August 24, 2022).

The TECH implementation team worked as quickly as they could to process the single-family applications. As of September 19, 2022, a majority of incentive applications had been closed and the contractors paid (Table 18).

The application database analysis showed that between the start of TECH in December 2021 and May 23, 2022, 328 unique contractors completed single-family TECH projects and were paid their incentive(s). On average, each contractor completed about 10 projects, though 96 contractors completed only one TECH project. The highest-volume contractor completed nearly a tenth of all projects in our database (260 of 3,361; 8%). The top 25 contractors completed half of all closed projects as of May 23, 2022 (1,701 of 3,361; 51%).

4.2 Contractor Satisfaction

Contractors who were enrolled in TECH as of June 13, 2022 were surveyed about their satisfaction with the initiative. The findings in this section come from the survey fielded from June 21, 2022, to July 5, 2022. During this time period, the TECH team was processing the large number of incentive applications and when many contractors were awaiting incentive payout(s). As such, many of their responses reflect their experience in this situation.

Overall, surveyed contractors reported mixed satisfaction with TECH. More than a quarter of surveyed contractors (48 of 180; 27%) said they were “not at all satisfied” with their experience (Figure 8).

We found that contractors with fewer projects were less satisfied than those contractors with more TECH projects. Of the 30 contractors who took our survey and had no TECH projects recorded in the incentive application database, half of them were “not at all satisfied” or “slightly satisfied” with TECH overall. Another way to look at this data is that, of the 37 surveyed contractors who said they were “not at all satisfied” with TECH, nearly four-fifths (29 of 37; 78%) completed five or fewer TECH projects. Comparatively, almost two-thirds of those who were “moderately satisfied” or “very satisfied” with TECH, had completed more than five projects. Of the 12 surveyed contractors who had experience with a multifamily TECH project, ten of them were at least “somewhat satisfied” with their overall TECH experience.

Furthermore, we found that contractors who layered a TECH incentive with another program’s incentive were also less satisfied with their TECH experience, though this was a small portion of survey respondents. As shown in Figure 9, nearly two-thirds of contractors who layered TECH incentives said they were “not at all satisfied” or “slightly satisfied” with TECH (11 of 18; 61%). Almost all satisfied contractors had not layered a TECH incentive with another program’s incentive (51 of 54; 94%). Finally, we found a larger proportion of satisfied contractors (18 of 54; 33%) who had enrolled in TECH in 2021 compared to 20% dissatisfied contractors (12 of 61).

When given the opportunity to share final thoughts about their experience with TECH in the survey, a minority of respondents (9 of 180; 5%) reflected positively on their participation in the initiative. Most respondents (102 of 180; 57%); however, used this survey section to provide critical feedback, including many suggestions for improvement. The remaining respondents did not provide optional comments.

Many contractors invested great effort to become involved in TECH and promote it, only to experience detrimental effects to their firm and its cash flow. A notable proportion of contractors communicated distrust for TECH, specifically regarding delays in reimbursement (53 of 180; 29%), lack of clarity around incentive structure (28 of 180; 16%), and poor communication (26 of 180; 14%). A collection of open-ended responses from contractors are provided verbatim below as examples of the feedback received. A sample of various suggestions for improvement are also included below, commonly regarding eligibility and structural changes to consider for the initiative moving forward.

• • “The fact that I had to include the rebate in my quote and trust in TECH Clean to pay me is very unnerving. So far, I am out thousands of dollars.”
  • “I sent my tech to 3 days of classes out of town I promoted the program and did 4 jobs. When the rebates never came, I quit recommending the program and now I feel used.”
  • “Don’t expect contractors to act as bankers until you folks get around to cutting checks. People calling us “where is their rebate?” drove us nuts. It ruins the satisfaction of getting a rebate if clients (or contractors) have to wait more that 2 or 3 weeks for their payment.”
  • “They offered a contractor bonus that was suddenly suspended as I was enrolled and had started a TECH sponsored class. They cut off the bonus literally in the middle of one of their sponsored/approved two-day trainings. This is shady to me. I took off work and spent two days of my time and fuel to go to the class for nothing.”
  • “I have called and left several voicemails for my rep & am still waiting on a reply weeks later …. no reply to voicemails or emails.”
  • “I am not going to get reimbursed for $21,000 in rebates I offered but were still in process at the time of the shutdown. It is unconscionable to do this to small businesses owners who are diligently assisting is state programs.”

The suggestions for improvement contractors mentioned at the close of the survey included:

• • “The payment side to the contractor has to be fixed and corrected moving forward. Contractors are being forced to front thousands of dollars of cash flow to participate in this program. There has to be a balance between all parties for a program like this to work long term.”
  • “Make things clearer so nobody is confused and offers the wrong incentives to customers, correctly anticipate demand so that both rebates are available throughout the year, and [ensure] there are enough trained employees with Energy Solutions to process the applications in a timely manner. Do NOT make a promise to contractors then go back on your word and plan roll-outs better.”
  • “Increase incentives for higher performing systems. And decrease incentives for minimum required efficiency equipment. Modulating and variable capacity heat pumps truly provide the utility reductions that this program is trying to achieve. Providing information to homeowners on the benefits of variable capacity systems over multi-stage systems would help them wisely make choices in the equipment they pick.”

As embedded evaluators, Opinion Dynamics shared this feedback with the TECH implementation team soon after receiving it. TECH staff acknowledged the feedback and have planned changes to improve the contractor experience. One improvement to the incentive structure Energy Solutions will implement is simplifying the incentives, which will make the amounts easier to determine.

4.3 Program Processes

This section presents findings about TECH Initiative processes related to contractor enrollment and TECH’s communication with contractors. It also explains how TECH projects came about from the customer perspective, customer satisfaction, characteristics of the TECH projects, and how incentives were applied to project.

4.3.1 TECH Contractor Outreach, Awareness, and Enrollment

TECH began its outreach to support contractor enrollment in July 2021. TECH collaborated with heat pump manufacturers and distributors to leverage their contractor networks to facilitate awareness and enrollment in the initiative. The manufacturers and territory managers at the distributors informed their contractor networks about the TECH Initiative, its goals, and encouraged contractors to sign up. One interviewed HVAC manufacturer reportedly spent $400,000 on social media campaigns to drive interest in the TECH Initiative. As one of the HVAC manufacturer representatives we spoke with explained “We made that investment because we wanted to leverage what we thought was a good program to help our distributors and contractors grow their businesses.” Manufacturer engagement by TECH was an effective strategy, as survey findings indicate more than half (99 of 184; 54%) of enrolled contractors first learned about the initiative through a manufacturer or a distributor (Figure 10).

Interviewed contractors explained they wanted to get involved with TECH because it looked like a good opportunity for their businesses, mostly because of the size of the rebates available. To enroll in TECH, a contractor submits an application and demonstrates they meet the eligibility requirements, such as proper licensure and insurance. Then, the contractor completes required trainings, including an introduction to the online incentive application portal. The interviewed contractors found it easy to enroll, though one said it took longer than they thought it would (two weeks) because TECH staff kept coming back to them for more details. Most surveyed contractors (112 of 184; 61%) found enrolling in TECH to be “somewhat” or “very” easy, although nearly one-fifth (32 of 184; 18%) reported some difficulty with the process (Figure 11).

Of the 32 respondents who reported some difficulty enrolling in TECH, half mentioned not having any support from TECH staff or extremely long delays when trying to communicate with staff members. As one contractor wrote in the survey, “No one would ever answer the phone and they took 10 days to 2 weeks to answer an email. That is completely unacceptable.” In addition to the lack of support and communication, nine respondents (of 32; 28%) mentioned that the enrollment process had too much paperwork and asked for too much information they had to gather that wasn’t easily available. Table 19 shows the breakdown of open-end responses that were grouped into buckets for all 32 respondents.

4.3.2 Understanding of Customer and Equipment Eligibility Criteria

Most surveyed contractors found customer and equipment eligibility criteria to be at least “moderately clear” (Figure 12). About one-fifth found the criteria to be “a little clear” or “not at all clear.” Nearly half of contractors who found one type of eligibility “not at all clear” (12 of 25; 48%) found all three types of eligibility criteria were “not at all clear.”

In total, 49 respondents thought the following aspects of TECH were either “not at all clear” or “a little clear”:[11]  customer eligibility criteria (34 respondents), HVAC heat pump incentive eligibility criteria (38 respondents), or heat pump water heater incentive eligibility criteria (30 respondents), shown in Table 20. For all three criteria, equipment qualifications (e.g., product type, sizing, type of replacement) and project verification to receive the rebate (e.g., HERS testing, location by zip code) were most commonly mentioned to be unclear. Contractors continued to mention poor support from TECH in resolving questions around eligibility criteria.

[11] Respondents were asked to rate all three of these eligibility criteria and the responses were not mutually exclusive.

Many contractors who noted an issue with verification or qualification highlighted the complicated process for verifying eligible zip codes via the zip code lookup tool hosted on the Switch is On website. Contractor responses suggested TECH lacked organization, completeness, and overall preparation when compiling the list of eligible zip codes, which resulted in a time-consuming verification process for participating contractors. Some contractors also disclosed that due to the lack of clarity around zip code eligibility, some customers were promised rebate amounts both greater and less than they were eligible for based on their location. Below, we have included representative quotes from two contractors who explained their experience verifying customer zip codes in open-ended survey answers:

“Many zip codes were not in the database and had to be manually verified pre-verification. Once submitted they had to be re-verified via phone and email. No response from any department via email or phone. It took 60-80 phone calls for our last 2 rebates to verify the zip code variation issue. We just don’t have the resources to follow up with this inefficient process.”

“I had asked if we could go off of the incentive matrix that PG&E’s CHR had sent out, and we were told yes, when in reality, that was not the case. We didn’t understand anything about Enhanced vs. Baseline and at the start the zip code list wasn’t complete or correct, so that threw things into even more confusion.  We ended up promising some customers more than they should have gotten and others less.”

Energy Solutions staff noted that there is not one publicly available list of all zip codes and their cities in California. The staff had to combine multiple lists, none of which were comprehensive, which led to the unintentional omission of some zip codes. TECH staff also explained that if a contractor came forward with information that contradicted what they were paid, TECH staff had approval to make up the difference. They received approval for a $100,000 contingency fund for this situation and reportedly used it “liberally” to ensure contractor satisfaction. It is unclear how many contractors knew they could challenge an incentive amount paid to them or how many took advantage of this situation.

Overall, surveyed contractors reported having sufficient product options within the qualified product list to choose from for both their HVAC (96 of 172; 56%) and water-heating (48 of 79; 61%) heat pump projects (Figure 13). All four interviewed contractors reported that all the heat pumps they wanted to install were eligible under TECH. A sizeable portion of contractors, however, did have some difficulty finding qualified equipment for their TECH projects (44% of contractors working on HVAC projects and 39% of contractors working on HPWH Projects). Those who paired TECH incentives with another program (31 of 44; 70%) were more likely to report they always had enough product options, compared to those who used only TECH incentives (14 of 26; 53%).

4.3.3 Project Generation

This section presents findings related to contractor and manufacturer marketing to spur interest in heat pumps projects, how single-family customers found their TECH contractor, and factors that influenced customer decision-making to purchase the heat pump.

Marketing

A large majority (145 of 180; 81%) of surveyed contractors’ firms increased their promotion of heat pumps after enrolling in TECH (Figure 14). Of contractors who did not increase the promotion of heat pumps, one-third (12 of 35; 34%) had not yet filled out an incentive application, indicating they had just begun their participation.

Most of the high-volume interviewed contractors (3 of 4) invested in marketing to promote the TECH Initiative. All three promoted the TECH incentive to their existing customer base via electronic newsletters and mailers to homes. Some promoted the incentive through TV and radio commercials, social media, their website, and via search engine optimization. These advertisements successfully garnered attention. One interviewed contractor explained:

“We went all out across all media platforms, social media, and were really pushing this because it was such a great incentive to homeowners. And it definitely triggered an incredible response. We got a ton of people calling about the program. It really helped stimulate the market for us. And we were able to generate a lot of revenue during that time.”

This contractor reported that the investment in marketing and short time frame when incentives were available meant that he “probably broke even” between the increased revenue and amount invested in marketing. The interviewed contractors also noted that marketing materials are made well in advance of their distribution so any changes to incentive eligibility or amounts would force them to pull their marketing.

The interviewed manufacturers also invested in marketing to promote the TECH Initiative. They reported marketing to contractors to promote enrollment and to consumers to generate leads for their contractor network. They were similarly disappointed to have invested in marketing and promotion only to have TECH incentives run out shortly after.

Finding a Contractor

About half of surveyed HVAC heat pump customers (357 of 728; 49%) found their TECH contractor through existing personal relationships, from either a previous experience with the contractor or recommendations from a friend or family member (Figure 15). Most other HVAC customers (255 of 728; 35%) found a contractor online. Examples of “other” responses included finding their contractor through social media, TV/radio ads, mailers/pamphlets, and home adviser groups.

Results for HWPH respondents are almost reversed from those of HVAC heat pump customers, with nearly half of customers finding their contractor online either through a web search or their utility’s website (Figure 16). Notably, 9% of surveyed HPWH customers (22 of 240) found their TECH contractor through the Switch is On website.

Results for HWPH respondents are almost reversed from those of HVAC heat pump customers, with nearly half of customers finding their contractor online either through a web search or their utility’s website (Figure 16). Notably, 9% of surveyed HPWH customers (22 of 240) found their TECH contractor through the Switch is On website.

Most HVAC heat pump customers were looking to replace old equipment that was either functioning poorly or not functioning at all (Table 21). On the other hand, HPWH customers were most likely to say they were exploring new options while their existing equipment was still functioning well. About a quarter of HPWH customers (58 of 240; 24%) reached out to a contractor specifically to electrify their home, while only 6% of HVAC customers (44 of 728) had this reason. Compared to earlier waves of the single-family post-install survey, customers have become more likely to reach out to contractors with the intention of electrifying their homes or reducing their carbon footprint. About 18% of HPWH customers from the first surveys conducted, back in April 2022, wanted to electrify their homes along with less than 1% of HVAC customers. In comparison, 24% of HPWH customers and 6% of HVAC customers from cumulative survey data up to July 10, 2022, reached out to contractors with this intention.

About one-third of both HVAC heat pump and HPWH customers hired the first contractor they spoke to (33 HVAC, 36% HPWH). Most of the others found their contractor after speaking with two contractors (23% HVAC, 23% HPWH) or three contractors (27% HVAC, 26% HPWH). Seven of the interviewed customers who were aware of the TECH incentives reported they contacted contractors and found that they were unaware of the TECH Initiative. In some instances, this led to the participant needing to interview five or more contractors in their area before finding one offering TECH incentives. This situation arose more often for HPWHs (6) than for HVAC heat pumps (1).

A majority of customers reported that their contractors offered useful information about why an electric option was a good choice for their home (76% HVAC, 68% HPWH), although a sizable portion (34% HVAC, 69% HPWH) indicated they mentioned a heat pump as an option before the contractor did. As data from more waves of this survey have been included, customers (particularly HPWH customers) have become more likely to mention a heat pump as an option before their contractor does. Around 32% of HVAC customers and 53% of HPWH customers from the first wave of surveys in April 2022 mentioned a heat pump before their contractor.

Customers had varying levels of knowledge about their new heat pumps, though of our survey respondents, HPWH customers were more knowledgeable than HVAC customers (68% and 52%, respectively, reported having at least some knowledge). Customers in later survey waves reported more knowledge about heat pumps than those in the earliest waves, and the disparity between the more-knowledgeable HPWH customers and the less-knowledgeable HVAC customers also grew larger. 

We found minimal differences in single-family customer knowledge of heat pumps among the gas IOU territories (Figure 17). About half of surveyed HVAC customers had little to no knowledge of heat pumps prior to talking to their TECH contractor, regardless of IOU territory. HPWH customers, on the other hand, were more likely to be very knowledgeable about heat pumps, suggesting they researched HPWHs prior to contacting their TECH contractor.

Single-Family Customer Decision-Making

Solar panels can be a catalyst for customer interest in heat pumps; the two systems are complementary because the heat pumps can use the electricity produced by the solar panels, typically driving significant interest in electrification. Half of the interviewed customers with solar or plans for solar (6 of 12) indicated that having solar panels was a major motivation for their purchase of TECH-incented heat pump. Many surveyed TECH customers either already had solar panels or were considering getting them (Figure 18). Water heater customers were more likely to have solar or plans to get solar than HVAC customers.

Surveyed customers who had solar panels installed (53%) or were considering getting them (56%) were slightly more likely to have purchased their heat pumps without the incentive than customers who had no plans to get solar panels (48%) (Table 22). These trends indicate the incentive was more influential for those who did not already have solar panels or plans for solar panels.

Financial benefits (such as the TECH incentive, energy savings, and energy efficiency) and environmental reasons (reducing carbon footprint, environmental impact) were the main reasons customers decided to move forward with a heat pump instead of an alternative type of equipment (Table 23). Another important factor was a desire to complement existing or future solar installations by converting as many appliances as possible to electric. Other, less common responses for HVAC heat pump participants, included things like ease of use, indoor comfort, and safety when compared to gas. This information was provided unaided, in open-ended answers.

When given specific factors to score, HVAC customer responses had similarities and differences compared to their previous open-ended descriptions. Financial benefits such as energy savings, operating costs, and incentives were still rated highly, but other factors such as equipment reliability, indoor comfort and indoor air quality were also rated as very important in their decision to purchase a heat pump (Figure 19).

HPWH customers found it important that the water heater used electricity (Figure 20) in addition to their financial and environmental motivators.

According to surveyed homeowners, contractors often mentioned both the TECH Clean California incentive, (70% HVAC, 72% HPWH), and the specific dollar amount (93% HVAC, 96% HPWH). A much smaller number of respondents could remember the dollar amount specified by their contractors (HVAC 51%, HPWH 41%), and they reported that the contractor generally mentioned the amount as $3,000 although answers ranged from $430 to $13,000 and had an average of $3,117. Of the interviewed customers, only two-thirds (14 of 21) indicated they had prior awareness of the TECH incentive.

In the follow-up interviews with single-family customers who were aware of the incentive, there was considerable confusion regarding the amount of the TECH incentive. Before TECH instated the incentive pass-through requirement, it seems many contractors kept the incentive or only passed on part of the incentive to the consumer. In these cases, there was no annotation of the projected incentive on the installation bid paperwork. Even in the case of later installations, when the incentive amount was included on the bid paperwork, it seems the incentive was not always communicated to the consumer.

The TECH incentive encouraged customers to purchase their heat pumps, as 93% (898 of 971) reported that the incentive was either “very important” or “somewhat important” in their decision to get the heat pump. Of those customers, 40% of HVAC customers (266 of 665) were either unlikely to or would not have purchased a heat pump without the incentive. An even larger portion of HPWH customers reported feeling the same way, at 60% (143 of 233).

It seems the TECH incentive was most influential for HVAC customers in SoCalGas territory. These customers were least likely to have still purchased a heat pump without the incentive (Figure 21). The sample sizes for the different IOUs for HPWH customers were too small to draw any meaningful conclusions. Quotes from interviewed customers who were aware of the incentive offered these insights:

“The incentive was a big difference for us because we are on a fixed income.”

“We are really grateful for these programs that help us to reduce our greenhouse gas emissions.”

Financing was generally not used by single-family customers for their TECH projects (Table 24). When customers did finance the equipment, both HVAC heat pump customers and HPWH customers chose most often to finance through the contractor that installed their equipment (55% HVAC [n=182] and 45% HPWH [n=9]). Other financing included putting it on a Credit card, using home refinancing, or financing through GoGreen Home. GoGreen Home financing is available for energy efficiency retrofits and is administered by the State of California with support from the IOUs. The TECH Initiative has been working to expand residential customers’ access to GoGreen Home financing for heat pump projects.

A majority of both HVAC (109 of 176; 62%) and HPWH customers (12 of 20; 60%) who used financing were unlikely to or would not have purchased their heat pumps without the financing (Figure 22). This finding indicates the availability of financing helps make heat pump projects a financial reality for some customers and expanding access to financing is one way to reduce barriers to heat pump installs.

Few households required panel upgrades (155 of 3,117; 5%) to accommodate their heat pump projects per the application database analysis. Panel upgrades were significantly more common among households that installed a HPWH (62 of 627; 10%) compared to those that installed an HVAC heat pump (117 of 2,539 5%).[12] Not surprisingly, panel upgrades were also significantly more common in households that had either a room/window air conditioner (AC) (4 of 38; 11%) or no AC at all (17 of 193; 8%) than in households that previously had central AC (42 of 1,176; 4%). Panel upgrades were about equally as likely for households in DAC communities (11 of 200; 6%) as households in non-DAC communities (126 of 2,917; 4%).

More than half of households that got a panel upgrade (74 of 137; 54%) previously had an electrical panel with 200 amps and a small amount (7 of 137; 5%) had electrical panels with capacities over 200 amps (Figure 23). All records showed a panel capacity of 200 amps post-upgrade, so 5% of panel upgrade projects resulted in a decrease in amperage.  

[12] Statistically significant at the P-value <= 0.05 level.

The need to upgrade their electrical panels did not appear to negatively affect customers’ decision to move forward with the heat pump. While nearly half of surveyed HPWH customers (112 of 240; 47%) and over a quarter of HVAC heat pump customers (190 of 728; 26%) reported their electrical panel needed to be upgraded as part of the installation, relatively few (17 of 240; 7% HPWH, 25 of 728; 3% HVAC) reported that it caused them to rethink their decision (Table 25). Please note that, even though the need to upgrade their electrical panel caused these few customers to rethink their heat pump purchase, they ultimately did move forward with a TECH-incented heat pump.

4.3.4 Project Implementation

This section describes the single-family TECH-incented heat pump projects. It reviews where in California the projects occurred, the project lengths, additional installation work the contractors performed, and how layered incentives offset project costs. We begin with findings comparing HVAC heat pumps and HPWH and then discuss specifics for each of those equipment types.

Project Characteristics

Per the application database analysis with closed projects, TECH-enrolled contractors installed a total of 3,361 heat pumps between the start of the TECH Initiative, December 7, 2021, and May 23, 2022.[13] HVAC heat pumps accounted for the majority of installed equipment, with 2,711 HVAC heat pumps installed and 650 HPWHs installed. These 3,361 heat pumps were installed in 3,117 households. A small proportion of households (2%) purchased both HPWHs and HVAC heat pumps (Table 26). There were 2,539 households that installed at least one HVAC heat pump and 627 households that installed a HPWH (Table 26).

[13] A closed project means that the contractor has been paid the incentive for that project. This database contained single-family projects, defined as one to four units. Nearly all projects in the database were single-family homes (98%) while 2% of projects were at sites with two to four units, which TECH classifies as single-family.  

Most single-family TECH projects were completed in non-DAC areas (2,917 of 3,117; 94%) which was true for both HVAC and HPWH projects (Table 27).

Table 8 describes single-family HPWH baseline and enhanced incentives. These varied depending upon the existing equipment and the size of the HPWH.

Table 8 describes single-family HPWH baseline and enhanced incentives. These varied depending upon the existing equipment and the size of the HPWH.

The distribution of the surveyed customers reflects the distribution of the projects described above: HPWH customers were concentrated in PG&E territory while nearly half of HVAC customers were in SoCalGas territory (Figure 25).

At the time of this report, TECH baseline incentives for HVAC heat pumps and HPWH were still available in SWG territory (along with the HPWH incentives available in SoCalGas territory). The reduction in incentive amounts for SWG territory do not seem to have slowed project uptake in that area, as there were more incentive applications after the reduction on May 14, than before it (Table 29).

Nearly all homes represented in our survey of single-family customers were owner-occupied (Table 30).

HPWH installs were commonly completed in one day, while HVAC projects were often likely to take more than one day (Figure 26). Contractors took an average of 3.2 days to install a HPWH, ranging from one day to over a month and a half (44 days) per the incentive application database analysis of closed projects. HVAC heat pump installations took an average of 3.6 days, ranging from one day to nearly two months (55 days). Nearly three-quarters of HPWH projects took just one day (74%), whereas less than half of HVAC heat pump projects (41%) were completed within a day.

There was no significant difference in the installation time between HPWH projects installed with only TECH incentives and those installed with enhanced incentives. Contractors took an average of 3.4 days to complete a project that only had TECH incentives, and 3.8 days to complete a project with layered non-TECH incentives. The distribution of days between TECH-only HPWH projects and layered incentive projects was similar.

HVAC heat pump projects with layered incentives took approximately a full day longer than those with only TECH incentives, a difference that was statistically significant. Contractors took an average of 3.6 days to complete a project that only had TECH incentives, and 4.7 days to complete a project with layered non-TECH incentives. TECH-only HVAC projects were more likely to take just one day (43%) compared to layered incentive HVAC projects (31%) and less likely to take three or more days (Figure 28).

The majority of TECH installs without layered incentives had a total incentive amount of exactly $3,000 (90%; 2,258 of 2,519). Figure 29 combines HVAC heat pump and HWPH projects. The average incentive amount for these installs with only TECH incentives was $3,139 and ranged from $1,000 to $7,600.

Heat pump installs with layered incentives had a significantly higher average incentive amount ($4,310) and were also much more spread out in incentive distribution ($1,500 to $12,000) (Figure 30). Iin these layered project installs; however, TECH paid an average of just $1,852 per project, meaning incentives paid by non-TECH sources made up 58% of the total incentive amount, on average.

Of the HVAC heat pump projects that only received TECH incentives, the average total cost of a project before incentives was $18,503. In these cases, the TECH incentive offset an average of around 17% of costs ($3,160) (Figure 31). In comparison, HVAC heat pump projects that had additional non-TECH incentives layered on cost an average of $19,362 prior to incentives being applied. TECH incentives offset an average of 14% ($2,806) of these costs, while non-TECH incentives offset an additional 10% ($1,931). HVAC heat pump projects with layered incentives both offset a higher portion of costs than HVAC projects with TECH only incentives and were more expensive overall.

Of the HPWH projects that only received TECH incentives, the average total cost of a project before incentives was $7,119. In these cases, the TECH incentive offset an average of roughly 44% of total costs ($3,163.66) (Figure 32). In comparison, HPWH projects that had additional non-TECH incentives layered on cost $7,831 prior to any incentives. TECH incentives offset an average of 17% ($1,325) of these costs, while non-TECH incentives offset an additional 27% ($2,126). HPWH projects with layered incentives did not offset a higher portion of costs than HPWH projects with TECH only incentives, although projects with layered incentives were more expensive overall.

Heat Pump Water Heater Projects

In this section, we review characteristics of HPWH projects. These findings reflect the projects for which contractors had received their incentives (closed projects) through May 23, 2022, as represented in the incentive application database analysis. TECH-incented HPWHs were most likely to be installed in the garage (474 of 616; 77%) (Figure 33). In most California climate zones, the garage is an ideal place for a HPWH because it has sufficient space and temperature to function well.

HPWHs installed in climate zone 3 were far more likely to be installed in the basement when compared to other climate zones (Table 31). This is likely due to the colder winters in climate zone 3 (northern inland regions including Sacramento).

[14] https://www.energy.ca.gove/media/3560

A majority of both TECH-incented HPWHs (297 of 533; 56%) and previous water heaters (284 of 533; 53%) were 50-gallon capacity, but more of the new HPWHs were greater than 50-gallon capacity compared to the water heaters they replaced (Figure 34). The new HPWHs were a larger capacity than the previous water heater in 60% of cases, the same capacity in 36% of cases, and a smaller capacity in just 4% of cases. The fact that 40% of TECH-incented HPWHs installed were larger than 50-gallon capacity is likely a reflection of a 2015 update to the National Appliance Energy Conservation Act. The update increased the minimum water heater energy factor, such that any electric storage water heater 55 gallons or above must use heat pump technology. As such, if a household requires a water heater tank capacity of 55 gallons or more, the contractor should install a HPWH.

TECH contractors most often set HPWHs on a hybrid operating mode (348 of 650; 53%), though a heat pump–only mode (279 of 650; 43%) was fairly common as well. TECH contractors recorded 4% (24 of 650) of HPWHs were set on electric resistance operating mode. It is unclear why the contractors would set HPWHs on electric resistance mode because the benefits of a heat pump are not realized when the equipment operates in this mode. It is possible that it could be set to electric resistance mode because of a problem with the unit itself (e.g., the heat pump part of the unit is not functioning). Understanding the motivation of this decision is an important factor moving forward.

TECH contractors installed thermostatic mixing valves in a majority of cases, and a bit more often in DACs (9 of 34, 74%) than outside of DACs (383 of 616; 61%), but it is worth noting the small sample size for HPWHs installed in DAC households (Figure 35). On June 20, 2022, TECH began requiring a thermostatic mixing valve to be installed on all HPWHs. Prior to that, there was a $200 bonus incentive in the Bay Area for installing the valve. The fact that 76% of HPWH projects were layered with either SMUD or BayREN incentives, and those programs require the valve, is likely the reason the valve was common on TECH projects. TECH contractors connected 14% of HPWHs to the grid (n=650). Grid connectivity was somewhat more common in DACs (24% of 34) than outside of DACs (14% of 616), but again, the small sample size for households in DACs makes it difficult to draw a clear conclusion. A small proportion of HPWHs required a condensate pump.[15]

[15] As the surrounding air is drawn across an HPWH’s evaporator coil and cooled, moisture in the air will condense on the coil.  This condensed water must be routed to a drain.  A condensate pump may be needed to move condensate to the appropriate drainage piping.

HVAC Heat Pump Projects

When examining the space conditioning heat pump projects, we found most HVAC customers had central AC before their TECH project. TECH contractors installed more efficient HVAC equipment in DACs and when pairing the heat pump with another program’s incentive.

The incentive application database omitted the previous AC type for more than half of the households that received an HVAC heat pump (1,123 of 2,539; 56%).[16] Of the rest, most households had central AC before (1,175 of 1,416; 83%) (Figure 36) and 14% used a room AC prior to their heat pump installation.

[16] As of July 2022, based on evaluation feedback, this is now a required field in the application database.

Households in DAC communities received, on average, more efficient HVAC heat pumps than those outside of DACs (Figure 37). Households in DAC communities (65 of 173; 38%) also received a significantly higher proportion of HVAC heat pumps with a SEER rating of 19 or higher than those in non-DAC areas (595 of 2,538; 23%).

The prevalence of more efficient equipment in DACs is likely due to the type of HVAC heat pump installed. A significantly higher percentage of non-DAC households purchased central ducted heat pumps (1,701 of 2,538; 67%) than DAC households (70 of 173; 40%), which have lower average SEER ratings than single zone/ductless heat pumps, often found in DAC housing (Figure 38).

Contractors, on average, installed more efficient HVAC heat pumps when pairing the TECH incentive with another incentive (Figure 39). Over half of HVAC heat pumps installed with layered incentives (190 of 343; 55%) had a SEER rating of 19 or higher compared to under a quarter (470 of 2,368; 20%) for heat pumps installed with just a TECH incentive (Figure 39). While 14 SEER heat pumps qualify for TECH incentives, PG&E’s Comfortable Home Rebates program required 15 SEER minimum with higher incentives for 16 and 18 SEER.  

TECH offered contractors a Quality Installation Incentive for completing each of following: Manual J calculations, duct testing and sealing, or field measured performance. At first, this bonus incentive was $600 through May 14, 2022, but was then lowered to $300. Contractors noted whether they performed any of these on the incentive application. TECH contractors were significantly more likely to have performed duct sealing at DAC households than non-DAC households (Figure 40). TECH contractors included smart thermostats on just under half of projects while they completed Manual J calculations on about 10% of projects. 

Project Challenges

We asked contractors what difficulties, if any, they have encountered when installing heat pumps in homes. Many surveyed contractors reported issues with technical aspects of the installation, such as electrical panel or wiring upgrades (28 respondents), permitting processes (10 respondents), or insufficient space for the installation of new equipment (5 respondents). Additionally, a large proportion of contractors reported issues related to the implementation of TECH. Surveyed contractors called out aspects of the TECH Initiative as factors that hindered their ability to install heat pumps in customer homes and have a positive outcome, such as delays in incentive distribution (16 respondents), overall lack of program funding (8 respondents), and complexity of the application process (8 respondents). Table 32 presents a complete breakdown of issues reported by surveyed contractors.

Single-Family Customer Satisfaction

Surveyed single-family customers generally reported positive experiences with their contractors (Figure 41). Fifty-eight HVAC HP customers and 26 HPWH customers specifically mentioned in open-ended responses that they were very satisfied with the experience working with their contractors. This satisfaction was not universal; however, 27 HVAC HP customers and 12 HPWH customers mentioned negative experiences with their contractors. In particular, 15 HVAC HP customers and 4 HPWH customers specified that their issues with the contractors were not remedied quickly. The issues related to equipment that stopped functioning, defective equipment, noise issues, tepid air blowing from the vents, damage to home (in one case, from an unbalanced water heater that leaked), overloaded electrical panels (in one case the breaker switch would flip when the microwave was used), and a higher interest rate on the financing than was promised.  

The majority of contractors explained how to use the equipment, answered questions satisfactorily, and showed up on time (Figure 42). Fewer contractors reportedly explained how to perform equipment maintenance, indicating this could be an area for improvement.

Generally, installed equipment passed city inspections on the first inspection for both HPWH and HVAC customers (Figure 43). Five HVAC HP customers and two HPWH customers indicated in the survey that their equipment had not yet passed a city inspection. The fact that 12% of HVAC and 8% of HPWH installs did not pass inspection on the first visit potentially indicates room for improvement in contractor knowledge of heat pump-related code requirements.

Three interviewed HVAC customers mentioned issues related to permitting. One customer said he did not want work performed on his home without a city permit and that most contractors he reached out to (five total) did not want to take the job as they reportedly felt a permit was an unnecessary burden. Another HVAC customer reported that the contractor had failed to meet a code that required an electricity supply disconnect at the unit, which was located in the attic. After the failed inspection, the contractor scheduled a return trip and installed the correct equipment, and the work subsequently passed the second inspection. Another customer reported that it had been five months since installation, and they have been unable to have the city code inspector perform the close-out inspection of their mini-split HVAC unit.

Four interviewed TECH customers mentioned non-permitting issues they had with their newly installed equipment (two HVAC and two HPWH). Two customers with mini-split HVAC systems reported physical issues with their installed products, one of which required immediate replacement of the entire unit. The other required an additional fitting to be installed to provide adequate heat from the unit during the winter heating season. Both of these issues were addressed in a timely fashion and resolved to the full satisfaction of the customers.

An interviewed customer who purchased a split unitary HVAC heat pump system had it installed by a non-local contractor who normally would install similar units in a much warmer climate zone. The interviewed customer lived at a high altitude (~5,400 feet elevation) in the Tehachapi Mountain Range and found his unit was insufficient for their winter heat demand. The customer reported that the lack of qualified contractors in their area forced them to go with an installation provider from the town of Lancaster, 60 miles away. Upon installation and duct inspection, the unit was unable to raise the temperature of the house above 65 degrees. After eight post-installation service trips, conducted over a two-month period, the contractor informed the customer that their equipment was “likely” faulty and offered to replace it with a larger sized unit. By this point the customer had become so frustrated with the experience they opted to replace their heat pump system with a natural gas furnace and air conditioner system, at considerable additional cost. 

Two other mechanical issues involving HPWH installations were identified by customers. One indicated that an abnormal vibration occurred during normal operations and that he was able to remedy it himself. The other customer had extensive knowledge of HPWH operations and observed that his unit was not performing according to expectations, partly because energy savings did not materialize on his energy bill. After extensive investigation, including discussions with manufacturer engineering staff and installing energy-monitoring equipment, he learned that this unit was not designed to run only as a heat pump. Instead, the HPWH was designed to operate in a “hybrid” function with the resistance heater. His characterization of the engineers’ response was that they did not believe that “anyone would wait for just the heat pump to heat their water” and thus did not design it to function in heat pump-only mode. Since this HPWH unit, “Premier” made by State, may not be performing as advertised and may not be producing the designed energy efficiencies that are expected when receiving a TECH-incented unit, further exploration should be conducted to determine whether this unit should be on list of incented products.

4.3.5 TECH Incentives

TECH-enrolled contractors quickly exhausted much of the TECH incentive funding in the initiative’s first six to eight months. Interviewed contractors and manufacturers found the single-family incentive amounts to be “generous,” “huge,” and “enormously helpful.” They said that amount made it a “no brainer” for customers to go with a heat pump instead of a gas alternative. They agreed that if the incentive were lowered to $1,000, then it would still be impactful, but not as much and, at that amount, some customers will not be convinced to go with a heat pump. One interviewed water heater manufacturer noted that the combination of recent inflation (which caused equipment prices to rise) and the expiration of federal tax credits for HPWHs, has placed more importance on the amount of the incentive in customer decision-making.

An interviewed contractor explained that initially, TECH implementation staff were not clear about whether the incentive was a point-of-sale rebate to be taken off the customer bill at the time of install or whether it should be a check that was later provided to the customer after paying in full. He explained that some contractors thought it was an instant rebate off the project cost while others charged the customer in full and provided a check after receiving payment from TECH. This particular contractor reported they were “apprehensive” about whether he would be reimbursed by the initiative, due to the amount of paperwork and “hoops to jump through.” For that reason, he chose to offer the rebate to customers as a check he would mail later. TECH staff indicated that contractors are allowed to discount the customer at either time but must reference the incentive on the invoice at the time of the sale.

A minority of surveyed contractors who completed an application found it easy (70 of 164; 43%), and 36% (58 of 164) found it difficult as seen in Figure 44. A sizeable portion (35 of 164; 21%) found it neither easy nor difficult.

Of the 58 surveyed contractors who found the application difficult, a majority stated the application was asking for something they did not have. As one respondent mentioned, “The application would ask for things we did not have. Such as photo of electrical panel or we couldn’t find the model number listed.” In addition, 18 respondents found the application too complex, and 13 respondents once again mentioned that there was no support for helping them troubleshoot problems with the application. It is also important to note that 11 surveyed respondents mentioned problems with the website/portal either being too difficult to navigate, “wonky,” or having constant glitches. Table 33 has the total breakdown of responses grouped into six buckets.

Interviewed contractors found the application asked for redundant information. The process for contractors to pull the permit requires a variety of forms and the HERS report. They described the TECH requirements as “excessive” and “overkill” because the permit ensures they would be completing those forms anyway and TECH staff should just need to see the permit. TECH does not require a closed permit to complete the incentive application; however, it does require the HERS CF-3R form. Waiting on the HERS report form and spending the additional time uploading the forms created additional burden for the contractors.

In an interview, an HVAC contractor explained that it took his staff at least “an hour or two” to complete each incentive application due to the volume of pictures and files they had to upload. Two of the four interviewed contractors (representing a total of 107 closed projects at the time of sample selection) reportedly hired a full-time staff person solely to work on the rebate applications. Compared to other rebate programs interviewed contractors had worked with, TECH required more paperwork and pictures. One contractor added, if TECH were to re-start with an HVAC incentive of $1,000, then they would choose to not participate, stating it is “not worth the time it takes for us to manage the paperwork through the online portal. The fact that they need all the information they do makes no sense.”

TECH staff sent updates or clarifications about application requirements to contractors via email. Per Energy Solutions, open rates of emails to TECH-enrolled contractors about program rules ranged from 57% to 61%. If a contractor did not see the emails, then they learned of new requirements after an incentive application was rejected for missing a required item. One interviewed contractor explained their experience:

They kept changing what required documents they wanted. And kept adding information that wasn’t originally requested in the beginning. There were no specifics on the types of electric heat pumps that we needed to install in terms of AHRI numbers – then all of a sudden they wanted to them. They’d notify us, oh we need this now. It was a launch and adjust situation which was challenging.

The volume of paperwork and frequent need for follow-up led to delays in reimbursements. It seems most contractors used the incentive as a point-of-sale rebate. The contractors paid for the heat pump equipment in full and discounted the customer’s invoice. While they awaited reimbursement from TECH, the contractors’ firms acted as a bank and carried the unpaid debt of the incentives. The four interviewed contractors explained that tying up that much cash flow waiting to be reimbursed was a risk to their firm. They noted smaller companies cannot take on that risk. When contractor companies have tens of thousands of dollars tied up, they cannot buy more inventory, they cannot stock parts on trucks, and may be reluctant to hire staff because they do not know when the money will be coming. TECH initially communicated a goal of paying contractors the incentive amount two weeks after approval. An interviewed manufacturer noted, however, that the turnaround time had grown to seven or eight weeks. Such a situation is a hardship for contractor firms.

A manufacturer reported one of their contractors had accumulated $300,000 in incentives they were owed to be paid by TECH, largely due to delays caused by waiting on HERS raters. The manufacturer had to advocate for this contractor, and it reportedly took TECH staff two months to resolve the issue. One interviewed contractor explained his experience with the reimbursement process:

“According to Energy Solutions, there was a timeframe that they put on it, that the homeowner could expect to see their rebate within, after the claim got approved. Well, that timeframe obviously was not accurate. And we had this issue with having to field calls from our clients who were disgruntled about timeframes and whatnot. So, the paperwork part of it was really the part that made it difficult for us to submit.”

A surveyed contractor echoed this sentiment and mentioned how much he was owed by TECH:

“We have floated almost $50,000 of incentive to our customers and so far have only been paid $3,000.  Original promise of a two week turn around is a complete lie. Very poorly organized program which is better in marketing than execution in my opinion.”

The delays in reimbursement have had negative effects for contractors’ businesses and have caused mistrust with the initiative. An interviewed contractor who sells both HVAC heat pumps and HPWHs reported they felt “powerless in this process.” If TECH were to reinstate incentives in his area, he said he would approach participation “more cautiously” than he had before.

Layered Incentives

TECH-incented HPWHs were much more likely than HVAC heat pumps to be layered with another program’s incentive. TECH contractors layered the TECH incentives with another program in over three-quarters of HPWH projects (492 of 650; 76%) while HVAC heat pump installs were layered with other incentives in a minority of cases (343 of 2,711; 13%) per the incentive application database analysis. TECH contractors layered incentives more frequently in DACs (63 of 207; 30%) than outside of DACs (772 of 3,154; 24%), but the difference was not statistically significant.

More than half of the paired TECH incentives were layered with SMUD incentives (438 of 835; 53%). Heat pump incentives from BayREN (228 of 835; 27%) and PG&E made up most of the rest (Table 34). Notably, BayREN provided incentives for HPWHs only, and PG&E provided incentives for HVAC heat pumps only, while SMUD provided both types of heat pump incentives.

Per the incentive application database analysis, 32% (104 of 328) of contractors had at least one closed project with a layered incentive. The TECH contractors in our survey had more experience layering TECH with other incentives, as 41% (67 of 163) of them layered the TECH incentive with another program’s incentive on at least one project. Half of surveyed contractors (34 of 67; 51%) who layered incentives found it “not easy” to determine the customer’s total incentive (Figure 45).

Of the 21 contractors who found it “somewhat difficult” or “very difficult” to determine the customer’s total incentive amount when layering TECH incentives, 12 suggested TECH provide clearer information around the incentive, including eligibility, amount for different regions and equipment types, and instructions necessary for layering rebates from TECH with those from other programs. One contractor described in the survey how inconsistency across TECH resources caused confusion and resulted in a monetary loss:

“Your webinar and print out from your website showed our zip codes as “enhanced” or layered but your web page to start the input of information said our zip codes were NOT “enhanced” so I had to eat the enhanced part of the rebate I promised the consumer.”

Other suggestions contractors provided that would help them in layering TECH incentives included processing rebates more quickly, as many are still stuck waiting to see if their rebates will be approved after months of waiting and providing contractors with better support to resolve pending questions and issues (Table 35).

Even though it was not always easy to determine the total incentive amount, nearly all (66 of 67; 98%) contractors who layered incentives found it to be helpful in selling heat pumps to customers (Figure 46). In fact, most said that having two incentives to offset the upfront cost helped “a great deal.”

For multifamily projects, which are usually planned in advance, contractors fill out a form to reserve incentives to cover all the heat pumps needed for the multifamily upgrade project. The property owner also signs this reservation form. Few surveyed contractors (12 of 184; 7%) had filled out a Multifamily Incentive Reservation form (Figure 47). Of those that have completed the form, more than half (7 of 12; 58%) found the process to be at least “somewhat” or “very” easy (Figure 47).

Four of the 12 contractors coordinated with TECH staff to complete the form. Three of those four were “very satisfied” with their experience, and the last respondent reported they were “somewhat satisfied.”

Two surveyed contractors answered that it was either “Somewhat difficult” or “Very difficult” to fill out the Multifamily Incentive Reservation form. One said there was no support while the other mentioned it was unclear which projects were classified as multifamily and thus required the form. They suggested that the definition of multifamily should be anything more than one unit, which is the criterion used in the BUILD Program. They wrote in the survey that “Working in four-unit buildings with shared water heating, it was unclear if we should fill out the reservation or not.”

Incentive Influence on Heat Pump Sales

The TECH incentive stimulated heat pumps sales, contractor revenue, and market growth. Most surveyed contractors (99 of 182; 55%) saw their heat pump sales increase either a great deal or a moderate amount since participating in TECH (Figure 48). Sixty-two percent (32 of 52) of contractors who said participating in TECH increased their sales “a great deal” reported fewer than five HVAC heat pump installations on average per month in 2021, and 98% (51 of 52) reported fewer than five water-heating heat pump installations. For contractors who said TECH increased their sales a great deal, the number of TECH-incented heat pumps installations ranged from 1 to 133; one-third of which (17 of 52; 33%) completed more than 30 heat pump installations by August 10, 2022. These findings suggest TECH had a larger impact on heat pump sales for contractors who previously sold few heat pumps.

Interviewed contractors (2 of 4, 50%) noted that prior to TECH, incentives were not available for installing a heat pump to replace a gas furnace due to fuel-switching regulations. With revised rules permitting them to replace gas furnaces with heat pumps, they could install many more heat pumps through TECH than they could previously.[17]

[17] CPUC Decision 19-08-009 adopted the fuel substitution test, which replaced the previous “three-prong test.” The fuel substitution test allows energy efficiency incentives to be applied to projects where the old equipment was natural gas-powered and the new equipment is electric, provided the project meets two qualifications: (1) The measure must not increase total source energy, and (2) The measure must not adversely impact the environment.

The high rebate amounts offered through TECH influenced customers to purchase heat pumps. One interviewed contractor who sold HVAC heat pumps in San Diego described how TECH influenced his sales: “From January through the end of April, we sold more heat pumps during that time than during the past two to three years combined.” He added that, when the incentives were suspended, his heat pump sales “died off” and he went back to selling gas furnaces and central air conditioners.

“Once TECH came and threw a lot of money with the rebate, it made a huge impact with people switching over. Most of those people probably weren’t going to buy heat pumps. TECH’s been very successful.”

“The rebates and marketing have been instrumental in boosting sales of heat pump technology. Prior to the campaign we were at zero heat pumps and now we install over 5 a month.”

With the TECH rebate being so influential, many contractors reported sales fell due to the abrupt loss of program funding. A majority (93 of 161; 58%) of surveyed contractors who had at least one TECH project reported the suspension of TECH incentives affected their sales “a great deal” or “a moderate amount” (Figure 49). Notably, over four-fifths (49 of 59; 83%) of contractors who reported the suspension of TECH incentives affected their heat pump sales “a great deal,” were contractors who also said that participation in TECH had increased their sales “a great deal” or “a moderate amount;” indicating that those contractors TECH helped the most were also among those most negatively impacted. We did not find any significant differences in how the TECH incentive suspension affected contractors across different utility territories.

Table 8 describes single-family HPWH baseline and enhanced incentives. These varied depending upon the existing equipment and the size of the HPWH.

4.3.6 Communication

TECH staff communicate information to contractors electronically, via emailed communication or via information posted on websites. TECH staff communicated about the incentive suspension via email. As one interviewed contractor in the Bay Area described,

“We got one email in May saying, ‘uh oh, we’re running out of money.’ I had a thousand questions and concerns. […] In early June we got another email saying, ‘Actually, we’re totally out of money. You have five days to reserve incentives for your existing clients.’ So, we dropped what we were doing and reserved all funds for our remaining clients, because we had nine months of clients scheduled counting on this money. It was an insanely small window of time [between notification and the deadline to reserve incentives for our already-scheduled projects].”

Contractors noted that something so fundamental to the program and so important to their businesses should be communicated in a way where the TECH staff could confirm receipt of information. Although most surveyed contractors were satisfied with TECH’s email communication (104 of 184; 57%), a notable quarter (45 of 184; 24%) of surveyed contractors mentioned some level of dissatisfaction with the emailed communication (Figure 50).

Among the 67 contractors who were not satisfied, most suggested TECH could improve its communication by responding to inquiries in a timelier manner and providing program updates quicker, particularly regarding changes to incentives (43 respondents). Ten contractors also highlighted how more program staff would be beneficial, enabling TECH to hopefully improve response time to inquiries and provide contractors with better support overall.

Many contractors (26) mentioned how TECH’s communication, or lack thereof, caused them to be unaware of these changes when they took place, and therefore resulted in detrimental effects to their business. A few contractors (3) suggested that information of high importance and urgency such as incentive changes should be communicated through modes other than email; a mode that allows contractors to easily confirm receipt may be beneficial to consider (e.g., phone call, text message). Examples of contractor responses provided below:

• • “We found out that the program was running out of funds through our contact at SMUD before Tech sent out an email days later. The communication from TECH has always been horrible.”
  • “Reach out via text message as well for important updates – such as reducing the initial incentive from $3,000 to $1,000”
  • “To shut down the program early with no communication to the contractors puts us in a very bad and dangerous position. Massive communication efforts should have been made prior to shutting anything down.”
  • “Email just doesn’t seem like a good way to communicate something as drastic as an incentive being cut completely short. What if we didn’t see that email until it was too late. Not a great way to communicate urgent items.”

One interviewed contractor said he received a phone call from TECH staff notifying him that the incentive was being suspended, because he was told he had submitted more claims in their region than any other contractor. An interviewed manufacturer provided a point of contrast, saying they have open lines of communication with TECH staff and have experienced no barriers to getting their questions answered.

TECH also provides information to contractors on websites. There were two main contractor-facing websites: The TECH Clean California website hosted at Energy-Solution.com and the Switch is On, hosted by the Building Decarbonization Coalition, a subcontractor of Energy Solutions. According to the TECH Clean California Work Plan, the Switch is On hosted a “Contractor Hub” that was “designed as a one-stop resource that consolidates all the disparate information from different programs, supply chain actors and trade allies so that participating contractors can easily find the resources they need to both participate in the program and build their heat pump business. […] It is also a place where prospective contractors can find information about the initiative and explore and learn about how it can benefit their business and a link to an enrollment form.”[18]

Contractors visited the TECH Clean California website more often than The Switch is On website. In fact, nearly one-fifth (33 of 184; 18%) of surveyed contractors said they had never visited The Switch is On. Overall, contractors found information provided on the TECH Clean California website to be slightly more useful than The Switch is On website, although ratings were similar across the two (Figure 51).

[18] Energy Solutions. TECH Clean California Work Plan. Page 105. (Unpublished). November 5, 2021.

4.4 Data

Proper data collection, storage, and tracking is important when implementing any program, but particularly important for an initiative the size and scale of TECH. In this section, we describe the data tracking systems used by the TECH implementation team and implications for the ability to do quality assurance and quality control (QA/QC), understand program trends, identify issues, and make course corrections.

Data Tracking Systems

TECH has a large implementer team and each organization on the team maintained separate databases with TECH contractor and project information. It was labor-intensive and challenging to combine data exports from the various organizations because there were not unique identifiers included across all datasets. Data tracking was also complicated by the fact that many contractor companies enrolled under one name, but were “doing business as” a different company name. Additionally, many contractor company names look similar to others, usually using some permutation of “heating and air” in their titles, necessitating a look at the address to confirm whether it is the same company, just entered differently in different databases.

Several research activities and analyses for this interim process evaluation relied on secondary data. Opinion Dynamics attempted to leverage three datasets for our evaluation activities:

• • Contractor Enrollment Database: This dataset was maintained by AEA and included the contractor company name, main contact name, email, phone number, billing address, license types, and Tax ID number.
  • Incentive Application Database: This dataset was maintained by Energy Solutions and is an export from the IRIS incentive processing website. This data included the contractor’s company name (but not an individual contractor’s name), the contractor’s address (which sometimes did not match the billing address), the contractor’s email, customer information, and equipment information.
  • Rosters of contractors and their staff who attended an optional workforce development training: These rosters were maintained by the training organizations (NCI, EMH, AEA). These datasets included the name of the attendee (which in some cases was the TECH-enrolled contractor and in other cases was a technician employed by a TECH-enrolled contractor), the attendee’s phone and email, and the contractor’s company name.

It would be valuable to be able to match project information from the incentive application database to the contractor’s company data in the enrollment report or training rosters. The business name and contractor (or training attendee) name varied enough; however, that matching on those fields resulted in very few matches. We brought this issue to the attention of Energy Solutions and suggested they identify a variable that can be maintained across all the datasets to facilitate matching and aggregation. As of August 2022, they have been able to include the TECH-enrolled contractor’s CSLB license number in each dataset, which will allow for future analyses that include data maintained across the three databases.

There were other data challenges within each dataset that prevented additional analyses that could inform design changes or illuminate market dynamics. Not having key data points limited analyses and prevented full exploration of TECH’s impacts. Key data points or processes the implementer has not instated, and which hampered our analyses, included:

• • No unique identifier to allow us to match enrolled TECH contractors with WE&T training attendees. This prevented us from being able to tie project data to those contractors who received training. Reportedly, the TECH training implementers (NCI, EMH, and AEA) each use a different contractor characteristic to structure their databases and the inconsistency in addresses, business names, and contractor names makes linking them labor-intensive and fraught with the potential for error.
  • Documentation of changes to incentive amounts and eligibility qualifications was not maintained in a single location. The information about how and when incentive amounts and deadlines changed exist across emails, webpages, and documents. Consequently, it is difficult to be sure that one has all of the information or the most up to date information. The evaluation team requested a summary from the TECH implementer multiple times because each iteration provided was incomplete. This is likely what lead to some of the contractor confusion reported in this study.
  • The application database with customer, site, and equipment data was either not structured in a way to support analysis or lacked proper quality assurance review. Issues included:
    • Classification of single-family projects as multifamily projects. The application database had a field for “building type” and the values were single-family or multifamily. All TECH documentation defined multifamily buildings as those with five or more units. It was only in the review of this report by subcontractors on the TECH implementation team, that we learned the projects labeled multifamily were all sites with two to four units. Therefore, they were single-family projects according to TECH’s definition. It also came to light that Energy Solutions had not provided Opinion Dynamics with any information about the completed multifamily projects.
    • Lack of specific data showing what incentives were applied to a project. The database showed whether a project received a layered incentive from another program and included a total amount of incentive paid outside of TECH. However, it did not contain indicators of whether the project received bonus incentives, such as those for completing Manual J calculations or completing a panel upgrade. Being able to know how many and which projects received specific incentives is a standard practice for energy program tracking and would help the implementation team see trends and illuminate possible program optimizations while also supporting energy program evaluation.
    • Omission of the customer’s electric utility. Energy Solutions staff expressed they chose to not require this field from the contractor because they were more confident in their ability to match an address with an electric utility than they were in the contractors’ ability to correctly enter it on the form. They indicated that they expect to append the electric utility to the customer record in early 2023. Not having this information at hand prevents comparisons of customers in PG&E territory who might be SMUD territory, or comparisons of SoCalGas customers with those in the territory of the Los Angeles Department of Water and Power.
    • Omission of whether the site had solar PV. Whether a customer has solar PV is an important factor in understanding consumer demand, market potential, appropriate incentive levels, as well as energy and GHG impacts. We identified this as an important field to add into the database, but the implementation team indicated this request would require significant resources to fulfill. Instead, the evaluation team was able to add a question to the customer survey to requesting this information. The implementation staff informed us they plan to add this data into the existing application database in 2023 by using AMI-data to determine solar PV ownership. The latter method, however, would miss a key nuance to understanding if the customer plans to add solar, so we recommend adding a question about solar to the incentive application form.
    • Not requiring prior air conditioning type. This field was missing for 50% of closed projects we examined in the application database analysis. Energy Solutions reported they have since changed this to be a required field. This information is important for understanding the benefit of how heat pumps providing both heating and cooling (for households that did not have cooling before) and for understanding energy impacts and the need for electrical panel upgrades.
    • Lack of quality assurance review related to the mode on which a HPWH was set. Our analysis uncovered that 4% of HPWH were recorded as being installed on electric-resistance mode, which forfeits any energy savings associated with purchasing a heat pump. If Energy Solutions had included a review of this field in the QA procedures, they could have followed up with contractors to educate them about proper operational modes.
    • Limited tracking of when staff had to follow-up with a contractor to correct issues in the database. The database contains fields for “first submission” and “second submission.” The TECH implementer informed us that whether the date in those columns was the same or different is not an indicator of whether the project required follow-up with the contractor to correct something. Energy Solutions staff reported they do not track specific revisions to incentive applications but do keep a record of common issues so that they can address recurring problems.[19] However, this practice prevents understanding of the time elapsed between each step in the application process.

Quality Assurance/Quality Control

The way the data are being tracked and maintained raises concerns for whether the implementation team can execute proper QA/QC. It seems the team performs limited QA/QC on TECH data. Table 36 lists the QA steps Energy Solutions staff perform when reviewing incentive applications.

[19] Energy Solutions staff reported the top five main issues that caused contractor follow up in the application process include (1) Invoice Total includes rebate amount taken from project costs (the invoice total should be prior to any incentives being applied); (2) No Name Plate Photo; (3) Attaching a CF1R/CF2R form instead of the CF3R; (4) Putting multiple units with the different model numbers on one claim (should be 2 claims); and (5) (Not shown on claim) Installed system is not an AHRI-certified combination, meaning that it isn’t eligible for TECH incentives.

In our review of the Incentive Application Data, we found some of these steps were not followed or there were inconsistencies or missing data that should have been included in the QA screening steps:

• • We found cases where the total project cost after incentives was a negative value, which would indicate the contractor received incentives in excess of the project cost. We notified Energy Solutions of these cases, which they indicated they would follow-up on.
  • We found some cases where the recorded electrical panel amperage was higher before a panel upgrade than after. Energy Solutions indicated that panels could be marked as having an upgrade and the amperage will not change, but they indicated the amperage should not be reduced. The evaluation team has also noted that there is no definition of panel upgrade in the application, so we are not sure how contractors are interpreting the meaning of panel upgrade. We recommend adding a definition in the application to help ensure contractors are using that data field consistently and adding a step to the post-processing QA to check these data.
  • We found cases where the contractor reported the HPWH was installed on electric resistance mode instead of heat pump mode or hybrid mode (hybrid uses the heat pump and only switches to electric resistance if needed). If a HPWH is set on electric resistance mode, most benefits of the heat pump water heater are not realized. We recommend this setting be added to the QA/QC process to maximize program impact.
  • We found cases where the contractor entered their email address instead of the customer’s email address. For one high-volume contractor we noticed that their first 30 projects had the contractor’s email address in the customer email address field. We informed Energy Solutions of this practice and the contractors who were doing it. Energy Solutions was able to follow up with the contractors to correct this habit. The application data shows that the high-volume contractor mentioned earlier changed their practice and their subsequent projects included the customer email address. This email QA procedure is important because without the customer email address, we cannot survey the property owner to hear about their experience with their TECH contractor. This is an example of the benefit of embedded evaluation as this issue was corrected mid-implementation as opposed to at the end of a program cycle.

4.5 Workforce, Education, and Training

This section reviews our experience with working with the TECH Team on the evaluation of training as well as findings from surveyed contractors who attended an optional workforce development training offered through EMH or NCI. EMH trainings were three days in length, while NCI trainings ranged from one to three days.

Evaluation Experience with TECH Training

In Opinion Dynamic’s weekly meetings with the Energy Solutions team, the topic of trainings was frequently on the agenda. While the implementation team attempted to be very responsive (as they have been with anything we have requested), it was challenging to get a full schedule of training events. Even when we were able to get a full schedule of events on April 11, another training not on that schedule would be mentioned in passing, which resulted in confusion, the loss of opportunity, and wasted resources. Moving forward, we request a more detailed plan and schedule that is updated in real-time, so all parties are aware of what training is being offered, the mode of the training, the instructor, and the intended audience.

Another challenge we encountered was that the implementation team has not been able to articulate the strategy behind why they were choosing one set of trainings over another or communicate the suggested training path for specific contactor roles (e.g., installer, contractor business owner, salesperson, etc.) to contractors. Currently, the training offerings appear opportunistic and followed a somewhat scattershot approach of including a variety of topics, as opposed to a well-thought-out plan supported by a clear theory of change.

As mentioned earlier in the report, information about TECH-sponsored trainings have been housed in a multitude of places, which can be advantageous for outreach and awareness, but there does not seem to be one site that clearly lists all of the TECH-sponsored training or training opportunities relevant to TECH but not sponsored by TECH. We recommend that the creation of this site, with suggested training paths, a training strategy, and a clear schedule, should be a focus in the last quarter of 2022.

Contractor Training Experience

As shown in Table 37, most individuals (contractors or their technicians) who signed up for an optional training attended the whole training. There were, however, some people who signed up for a training who dropped out early or never showed up at all.

Even though a minority of surveyed TECH contractors (54 of 184; 29%) attended an optional training offered through TECH, this is a larger proportion than attended an optional training out of the total TECH-enrolled contractor population. Almost a quarter of surveyed contractors (38 of 184; 21%) had not heard about the optional trainings offered through TECH (Figure 52). Among contractors who had heard of the optional trainings, over half (79 of 146; 54%) had not attended any trainings (Figure 53).

The most common reason contractors gave for not signing up for an optional training was they had already completed a similar training (20 of 79; 25%), although over a third (29 of 79; 37%) of respondents provided a reason regarding inconvenience of the training, related to either time, duration, or location (Table 38).

Most surveyed trainees had installed at least one HVAC heat pump in their career (44 of 50; 88%), although almost half (20 of 50; 40%) had limited experience with the equipment and had installed 20 or fewer. Over half of contractors (26 of 50; 52%) had never installed a heat pump water heater (Table 39).

The majority of surveyed trainees (48 of 93; 52%) first heard about TECH trainings through their employer (Figure 54). A quarter of trainees (25%) first heard about training opportunities through one of TECH’s websites (11 of 93; 12%) or a distributor or manufacturer (12 of 93; 13%).[20]

[20] See Section 4.3.6 for more explanation of the two websites: TECH Clean California hosted at Energy Solutions and The Switch is On hosted by the Building Decarbonization Coalition.

Most surveyed trainees participated in TECH trainings to gain new knowledge and stay up to date on the latest industry practices and advancements (Figure 55).

Among the two Airflow Testing and Diagnostics trainees who completed the post-training survey, both agreed with all statements assessing their knowledge and understanding of training concepts (Figure 56), although the small sample size is a challenge for making inferences from this data.

Refrigerant-Side Performance trainees reported the lowest level of agreement with concepts assessing their knowledge and understanding of course concepts compared to other training attendees. Of the concepts we asked about, Refrigerant-Side Performance trainees had the least understanding about how to assess whether other trades would need to be involved in an electrification project (Figure 57).

Attendees of the Space Conditioning and Water Heating Electrification course reported high levels of agreement overall with concepts covered in the training. In particular, over four-fifths of attendees strongly agreed that electrification poses a potential benefit to their business (22 of 26; 85%) and moving forward, they plan to present an electric solution to their customers (23 of 26; 89%) (Figure 58).

All six HVAC System Performance trainees reported they either “agree” or “strongly agree” with all statements assessing their knowledge and understanding of concepts covered in the training (Figure 59).

Overall, prior to attending the training, EMH attendees had less confidence in their ability to complete various heat pump job tasks than NCI attendees. Both EMH and NCI attendees were least confident in their ability to size heat pumps prior to the training. Nearly one-third of EMH attendees (7 of 26; 27%) reported they were “not confident,” and 10% of NCI attendees said they were “not confident” with this task (Figure 60).

Surveyed trainees were also asked to rate how much the TECH training affected their confidence in completing the same various heat pump tasks. NCI attendees reported the training had the greatest effect on their confidence to install heat pumps, 79% (23 of 29) reported they were “moderately more confident” or “a great deal more confident.” The largest impact EMH attendees reported was on their confidence to sell heat pumps, over four-fifths (21 of 26; 81%) said they were at least moderately more confident about this task after completing the training (Figure 61).

Although both NCI and EMH attendees reported higher satisfaction with instructors than the training overall, nearly all surveyed trainees (52 of 55; 95%) reported they were either “satisfied” or “very satisfied” with the training (Figure 62).

Overall, surveyed trainees who completed a training through EMH reported higher agreement than NCI attendees that the training met various standards related to clarity of information, structure and length of training, and relevance to their industry. Trainees who completed a training through NCI reported lowest satisfaction with the format and structure of the training (Figure 63).

When asked to rate their level of agreement with factors related to instructor(s) and the way they delivered course material, both EMH and NCI attendees provided positive ratings; suggesting instructors from both organizations were found to be knowledgeable and effective in the way they presented information (Figure 64).

TECH trainings exceeded most surveyed trainees’ expectations, such that nearly all EMH attendees (82 of 88; 93%) and nearly three quarters of NCI attendees (19 of 27; 70%) reported the training somewhat or far exceeded their expectations (Figure 65).

Nearly all surveyed trainees (52 of 53; 98%) reportedly planned to share the information they learned from the training with their colleagues.

Additionally, surveyed trainees were likely to recommend TECH trainings, particularly EMH attendees who all said they were “very likely” or “extremely likely” (Figure 66). Additionally, a majority of NCI attendees (23 of 27; 85%) also said they were at least “very likely” to recommend the training.

Twenty-eight trainees provided suggestions for how TECH can improve trainings. The most common suggestions were to lengthen trainings (six responses), use a more hands-on, interactive approach (five responses), and more in-depth explanations and evidence for calculations (three responses). Fifteen trainees provided other suggestions to improve trainings, see Table 41 in Appendix C for these remaining responses.

Surveyed trainees provided a variety of additional topics that would have been helpful to cover in the trainings that were not, including common heat pump issues (2 respondents). The remaining topics provided by trainees are presented below in Table 40.

The most commonly reported takeaways from TECH trainings were a better understanding of electrification and the important role it plays in decarbonization (14 responses), quality control best practices (11 responses), importance of proper equipment sizing (3 responses), and benefits of heat pump technology (2 responses). Other takeaways are shown in Table 42 in Appendix C. Ten trainees provided other suggestions to improve trainings (see Table 41 in Appendix C for these remaining responses).

Overall, surveyed trainees expressed satisfaction with TECH trainings. Many trainees mentioned gratitude for the opportunity to enhance their professional growth, complimented the instructors, and were generally pleased with their new knowledge about electrification and heat pump benefits. Representative quotes from surveyed trainees include:

“It helped me on my professional growth and greatly expanded my knowledge and understanding of the available system’s alternatives and future industries opportunities!”

“Extremely valuable knowledge, I don’t feel like the homeowner will ask me a question I can’t answer or a question where I won’t have the known knowledge to give an educated guess.”

5. Conclusions and Recommendations

We offer the following conclusions and recommendations. We shared many of these recommendations with Energy Solutions as they emerged, and they have already addressed or are actively taking steps to address many of them. In September 2022, Energy Solutions embarked on an internal project to redesign systems that will allow them to process the high volume of applications they expect to receive when additional incentive dollars are made available in 2023. These improvements are intended to ensure timely payment to contractors and streamline validation of equipment and customer eligibility.

• • Conclusion: California has set ambitious climate goals and heat pump technologies are key to reducing California’s GHG emissions that contribute to indoor and outdoor air pollution. The investment in the TECH Initiative and work by the Energy Solutions Team has led to a brand identity, infrastructure, and network that can be leveraged for additional heat pump deployment. The TECH incentive stimulated heat pumps sales, contractor revenue, and market growth and, without the incentive, contractors reported their installations of heat pumps have slowed.
    • Recommendation: Given the momentum established in the heat pump market through TECH, it is disruptive to the entire supply chain when a much-anticipated program runs out of incentives so unexpectedly. It is also detrimental to the market transformation goals of TECH. We recommend that additional funding be made available to support the continuation of the incentive portion of TECH.
  • Conclusion: The TECH incentive strategy, as originally designed, was too complex. The complexity of baseline and enhanced incentives caused confusion for contractors, customers, implementers, partner program staff, and evaluators. In some cases, it also led to contractors promising incentives to customers for which they were not eligible for, with negative financial implications to the contractor.
    • Recommendation: The Energy Solutions Team was already in the process of streamlining enhanced incentive offerings when the program largely ran out of funds. We support Energy Solutions’ effort to continue simplifying the incentive offerings. We recommend that Energy Solutions ensure the zip code lookup tool is accurate and up to date.
  • Conclusion: Initial research in California and nationally, suggested that heat pump markets are in a nascent stage of development. The TECH Baseline Market Assessment estimated that the installed base for air-source heat pumps was 4%, while HPWHs comprised 2% of water heaters in multifamily homes and 0.4% of water heaters in single-family homes in California. The incentive application data from TECH; however, suggests that the market is open to switching to heat pumps, especially for space conditioning when incentives are generous. This trend recently occurred in other heat pump programs with generous incentives. For example, Con Edison’s Clean Heat rebate program six-year program budget was oversubscribed by $528 million in just six months. Denver, Colorado’s electrification program also expended all of its heat pump incentives in three months.
    • Recommendation: Additional research needs to be conducted to identify the right incentive levels to promote market transformation while ensuring program and market stability. Anecdotal information in this report, that must be interpreted cautiously due to small sample sizes, indicates the reduction of the incentive to $1,000 for HVAC heat pumps in the Southwest Gas territory does not seem to have reduced applications. A few contractors reported that if incentives were lowered to $1,000; however, they would be unlikely to participate due to the perceived high effort necessitated by the application requirements.
    • Recommendation: Another key consideration when designing the next phase of TECH incentives is the recent passage of the Inflation Reduction Act. This bill invests in the reduction of carbon emissions in multiple ways, including a 30% tax credit up to $2,000 available to anyone who purchases and installs a heat pump, in addition to rebates of up to $8,000 for low- and moderate-income households earning less than 150% of the local median income. While the specifics of when and how the rebates will be dispersed is unknown, we recommend these benefits be factored in when optimizing the TECH incentive design.
  • Conclusion: There was an imbalance in the equipment incented by and installed through the TECH Initiative. A large majority of projects involved HVAC heat pumps, while a minority included HPWHs. Our baseline market assessment and this process evaluation found that the HPWH market is underdeveloped compared to the HVAC heat pump market, which elevates the need for interventions if TECH is to transform the HPWH market. At the same time, another California program, SGIP (Self-Generation Incentive Program Generation), will be making $84.7 million available for HPWH retrofit incentives in California beginning in 2023.
  • Recommendation: The CPUC and Energy Solutions should weigh the costs and benefits of using TECH Initiative funds to stimulate the HPWH market. Investment in the market is clearly warranted, but significant funds are already earmarked for incentives. As such, TECH may be better positioned to stimulate the HPWH market through WE&T activities and supply chain engagement rather than end-user incentives.
  • Conclusion: Customers reported that the majority of contractors explained how to use their new heat pump, answered questions satisfactorily, and showed up on time. However, fewer contractors reportedly explained how to perform equipment maintenance. Proper equipment maintenance is essential to ensuring equipment performs to manufacturer specifications and ensures persistence of maximum energy savings and GHG reductions.
    • Recommendation: Consider developing a customer leave-behind that contractors could use to help educate consumers on equipment maintenance.
    • Recommendation: Since many contractors have only been working with heat pumps over the past few years, assess if contactor training is needed on optimal equipment maintenance practices.
  • Conclusion: Some contractors communicated distrust for TECH, most specifically regarding delays in reimbursement, lack of clarity around incentive structure, poor communication, and lack of responsiveness to inquiries.
    • Recommendation: Earning back the trust of contractors needs to be a focus moving forward for the TECH implementation team. Establishing service level/response time goals and tracking these metrics over time will not only support rebuilding trust with contractors, but also help determine necessary staffing levels to ensure the implementation team is responding to contractor questions and inquiries and paying incentives in a timely fashion. Potential metrics include average time elapsed from question submission to the first response provided; time elapsed from the question submission to the time inquiry is closed, average time elapsed from initial incentive application to payment; and contractor satisfaction with TECH support and the incentive payment process.
    • Recommendation: Using the incentive application database coupled with contractor support experience, continue to update FAQs with answers to common questions and develop a document of contractor tips and tricks to streamline the incentive application process and accelerate reimbursement.
    • Recommendation: When communicating key initiative changes, such as incentive amount changes or changes in incentive availability, email communications and posting of email content to websites is insufficient. Consider using multiple channels to communicate changes that enable confirmation of receipt of information.
  • Conclusion: The TECH implementer has room for improvement in their data tracking systems and data management processes. Omissions in data tracked, inability to link data across different systems, and lack of a comprehensive data dictionary limited the ability for implementers to streamline the process of developing content for the public reporting website, perform more real-time QA/QC, and develop insights to inform program design modifications. This is of particular concern if additional incentive dollars are made available to the TECH Initiative.
    • Recommendation: Develop a unique identifier for all TECH databases to facilitate linking of the contractor enrollment database, the incentive application database, and the learning management systems. Develop a single database where information from the disparate databases is automatically uploaded on an established cadence. When we shared this early feedback with the Energy Solutions Team, prior to this report being published, they quickly began addressing this issue and are making great strides in making this a reality.
  • Conclusion: According to the TECH incentive application database, contractors indicated that in 4% of projects they set the HPWH on electric resistance mode. This negates many of the benefits of a HPWH.
    • Recommendation: The TECH implementer should add the HPWH operating mode to their Quality Assurance checklist staff use when reviewing incentive applications. The reviewer should confirm the HPWH mode is either set on heat pump or hybrid mode and follow up with contractors if it is recorded as being set to electric resistance mode. When following up with contractors, we suggest exploring the motivations for this choice to ensure this is not masking an equipment issue.
  • Conclusion: The incentive application requirements made it difficult for contractors to be reimbursed in a timely manner. Contractors found the incentive application was time-consuming and asked for too much information.
    • Recommendation: Since there are a limited supply of HERS raters, consider a pilot program to recruit and train HERS raters, potentially focusing on underrepresented populations. Increasing the number of HERS raters support both quality installation and market transformation goals of the TECH Initiative.
    • Recommendation: Reassess messaging around the documentation requirements. Make it clear why each data point is needed and how it is in service of reaching California’s ambitious clean energy goals.
    • Recommendation: Consider ways to speed up the reimbursement process. One possible option is to utilize mobile payment services instead of payment via paper check.
  • Conclusion: In the first eight months of the TECH Initiative, the majority of optional technical trainings focused on HVAC heat pumps. There was only a two-hour webinar that focused on the technical installation of HPWHs, offered three times between April and July.
    • Recommendation: Augment the training offerings to include more HPWH technical content. We communicated this early finding to the Energy Solutions team, and they are taking it into consideration as they are developing the training plans for year two of the TECH Initiative.
  • Conclusion: TECH-sponsored trainings increased contractors’ confidence working with heat pumps and improved their awareness of building electrification’s role in decarbonization. Yet, training attendees indicated they had the least confidence in sizing heat pumps, and post-installation findings indicated contractors have room for improvement in their sizing practices.
    • Recommendation: Proper equipment sizing is essential for customer satisfaction, efficient equipment performance, and the ability for heat pumps to act as a grid resource. We recommend the implementation team pay special attention to improving contractors’ skills and confidence around heat pump sizing in their plans for year two of TECH trainings.
  • Conclusion: The majority of surveyed contractors were aware of the optional training available. Only a minority of contractors took advantage of the free training offerings; however, because contractors had already attended similar training or found the locations and times to be inconvenient.
    • Recommendation: Given that most contractors have installed a relatively small number of heat pumps, contractors may not know what they don’t know and thus are unable to identify their own training needs. Consider providing a self-scoring assessment to help contractors identify knowledge and skill gaps. Areas of focus should include heat pump sizing best practices, effectively sizing heat pumps to maximize grid benefits, and installation best practices. This could help contractors better ascertain if these training offerings were redundant with past trainings they had taken.
  • Conclusion: Embedded evaluation has been shown to be a valuable approach as the Evaluation Team has been able to provide suggestions for modifications that the Implementation Team acted on quickly. Lack of involvement during the planning phase of the initiative however, meant we were playing catch up and trying to learn the strategy underlying the program theory as it was unfolding. In addition, not being involved in the contracting process for training subcontractors prevented us from discussing data collection requirements prior to the trainings. This led to delays in fielding surveys, which resulted in missed opportunities.
    • Recommendation: Contracting the Evaluation Team at the same time as the Implementation Team would maximize the benefits of embedded evaluation and potentially make better use of TECH funds.
    • Recommendation: Since we were meeting with the Implementation Team weekly and corresponding via email and ad hoc calls multiple times a week, the Evaluation Team did not conduct staff interviews. A structured data collection process, especially when starting work after the Implementation Team, is a necessary component of evaluation. We will be incorporating formal staff interviews every six months moving forward.
    • Recommendation: Consider consulting with the Evaluation Team when contracting with implementation subcontractors so that evaluation data needs and data collection methods are optimized and planned for prior to implementation activities.
  • Conclusion: Embedded evaluation enabled real-time participant research that resulted in above average response rates. Collecting insights from larger sample sizes closer to the time of the intervention gave us more confidence in the validity of our findings and greater ability to uncover variations among participants. Having robust data is particularly important for pilot programs to inform the most effective strategies for scaling to a state as diverse as California, which aligns with the vision of SB 1477. The timely and robust insights from embedded evaluation accelerate the ability to scale pilots into full-scale programs.
    • Recommendation: Consider expanding the use of embedded evaluation in pilot programs outside of BUILD and TECH.
  • Conclusion: Some contractors mentioned a challenge with the definition of multifamily buildings (five or more units) and that the definition does not align with the BUILD Program’s multifamily definition.

Recommendation: Work with the BUILD Program Implementation Team to align the definition of multifamily for both programs.

Appendix:

Data Collection Instruments

Click on PDF icon to view.