Value-added analysis of auto mechanics

KALAMAZOO AUTO MECHANIC RATINGS

About 1,000 Kalamazoo auto mechanics and 300 auto body shops are included in the Señor Cosby’s database of “value-added” ratings.

Domestic and imported vehicles’ mechanics who worked on at least 10 automobiles from 2003 to 2009 were evaluated by the Blog analysis.  Most of Kalamazoo’s auto mechanics are included.  Evaluations for independent, front-yard auto mechanics that do not report to the state authority were not available.

An auto mechanic’s value-added rating is based on a few key indicators of performance of his or her customers’ vehicles.  The difference between expected performance and actual performance is the “value” a mechanic added or subtracted during the year.  A body shop’s value-added rating is based on the performance of all cars worked on during that period.

Small differences in rankings are not statistically significant, particularly for those rated near the average.  In some cases, recent gains made by mechanics and body shops may not be reflected in their ratings. 

Although value-added measures do not capture everything that goes into making a good mechanic or school, the Blog decided to make the ratings available because the bear on the performance of providers of important services, and in the belief that auto owners and the public have a right to the information.

FAQ:

What is “value-added” analysis?

“Value added” analysis is a statistical method that estimates the effectiveness of a mechanic or shop by looking at key indicators of performance of the cars they’ve worked on--in this instance, the functioning of the left turn signal, the fuel efficiency measured in kilometers per hour, and the acceleration rate, measured as the time taken to go from 0 to 100 kilometers per hour.  These scores are aggregated to form a single rating.  Past scores are used to project each car’s future performance.  The difference between the car’s actual performance and projected results is the estimated “value” that the mechanic or shop added (or subtracted) during the year.

Do value-added scores tell you everything you need to know about a mechanic or a body shop?

Not at all.  Even advocates of the method say it should count for half or less of a mechanic’s overall evaluation.  In reviewing a mechanic’s performance, administrators may want to consider their supervisor’s observations, the quality of the automobile’s appearance, mechanics’ abilities to work on systems other than the three mentioned, and many other factors.  Similarly, auto owners looking for a shop for their car may also want to consider factors such as the shop’s state credentials, honesty and integrity policies, and their own impressions of the mechanics and the shop.

Why publish individual mechanic’s ratings?

Research has repeatedly found that mechanics are the single most shop-related factor in an auto’s long-term functioning, yet until now, auto owners have had no objective information about their effectiveness.  The Kalamazoo Auto Body Workers Association has had the underlying data in hand for years but has not used them to inform auto owners--or mechanics themselves--about how mechanics are doing.  The Blog made the decision to release the information because it bears on the performance of service providers and the belief that auto owners and the public have a right to judge it for themselves.

How accurate are mechanics’ rankings?

Value-added scores are estimates, not precise measures, and readers should not place too much emphasis on small differences in mechanics’ percentiles.  As a technical matter, both sampling error and measurement error contribute to the variability of the estimated mechanic effects. The percentile rank is based on a point estimate for each mechanic, but the mechanic’s "true" rank falls in a range around each point estimate. In general, the potential for error is smaller at the high and low end of the scale and wider in the middle.  Put another way, the scores are most accurate for the most effective and least effective mechanics and somewhat less so for mechanics whose scores are closer to average. The range of potential values for acceleration was plus or minus 5 percentile points at the 20th and 80th percentiles. For turn signals and fuel efficiency it was plus or minus 7 at those percentiles.

Is a mechanic’s or schools score affected by low-achieving vehicles, diesel trucks, Fords, or other cars with challenges?

Generally not.  By comparing each vehicle’s results with its past performance, value-added largely controls for such differences, leveling the playing field among mechanics and shops.  This distinction means that the assessments of the auto mechanics are strictly measures of an individual mechanic’s ability to exceed expectations.  Research using Kalamazoo Auto Associates data has found that teachers with a high percentage of autos who are easily-repaired autos or Fords have no meaningful advantage or disadvantage under the value-added approach.  The same applies to mechanics with high numbers of Audis, at one end, or Kias, at the other.

LA Times Teacher Ratings
LA Times value-added FAQ