THE Federal Motor Carrier Safety Administration’s original Compliance, Safety and Accountability (CSA) program attempted to ascertain a motor carrier’s odds of experiencing a future crash.
But the program’s Safety Measurement System (SMS) was inherently flawed, leading to unremitting disagreement about the correlation between the system’s safety categories and crash risk, and abuse of its results by plaintiff attorneys, insurance brokers, and even drivers and their supervisors.
“Throw that out the window,” said Steve Bryan, executive vice president and general manager of SambaSafety Transportation. “That is no longer the mission of the CSA system. The new system is designed and intended to identify deficient safety culture in a motor carrier, to answer the question ‘How safe are you?’”
FMCSA plans to conduct a full-scale test of its new system in April 2019.
The revamped CSA will be powered by Item Response Theory (IRT) methodology, as recommended by the National Academy of Sciences (NAS) in its CSA review “Improving Motor Carrier Safety Measurement,” which FMCSA commissioned after former President Barack Obama signed into law the Fixing America’s Surface Transportation (FAST) Act in December 2015.
And SambaSafety already prepared a new CSA/IRT Scorecard for carriers and drivers.
“When CSA was first being tested in 2008, Vigillo (now a SambaSafety company) was the industry leader in analyzing what fueled CSA scores and made the first CSA Scorecards available to the entire industry, nearly two years before the FMCSA program went live in December 2010,” Bryan said.
“Today, SambaSafety is leveraging Vigillo’s expertise to help customers understand the new CSA scoring model long before it will be released.”
CSA ‘Classic’
FMCSA’s original CSA scoring system was taken offline in 2015, soon after Obama signed the FAST Act, which prompted the 18-month-long review by NAS experts. Bryan said his company went to work developing its new CSA/IRT scorecard as soon as NAS released its findings in June 2017.
The NAS report made six recommendations, with the first being to “investigate a new statistical model, within the existing structure of SMS, over the next two years,” leading to FMCSA’s interest in IRT science, which was designed to answer psychometric questions in a variety of areas.
In this case the question is, “Is there a violation?” and the answer is yes or no.
“The starting pistol has been fired and FMCSA is working on this,” Bryan said.
The “classic” CSA was driven by the roughly 3.5 million roadside inspections law enforcement, in partnership with FMCSA, conducts annually, revealing, on average, two violations per inspection, resulting in 7 million violations feeding the system, with 945 possible infractions.
Severity subjectively was weighted by what NAS calls “expert” opinions, with additional points assigned for out-of-service violations, and severity multiplied by a certain number based on timeliness.
The process created, in the last two years, 43 million CSA points, the data that fueled everyone’s CSA scores, with points allocated to each carrier, theoretically helping compare similarly sized companies. After carriers were grouped by inspection count, they were placed into safety event groups, which were compared, resulting in a percentile in each of the seven CSA BASICs.
Science vs. opinion
The new methodology is much different.
“Nobody is talking about predicting future crashes,” Bryan said. “We are trying to evaluate the safety culture that exists at each individual motor carrier.”
To do so more accurately, expert opinions are out, and data science is in.
That also means no more severity or time weights, no CSA points, no safety event groups, and no BASIC percentile score.
“The good news here is the national academies gave recognition to the fact that we have nearly a decade of training and education and understanding in doing all of this based on the seven BASICs of CSA,” Bryan said. “So … they said leave those alone.”
The new CSA still will generate scores in each BASIC category, but without percentiles or constraints, and violations will be grouped, so FMCSA will look at types of violations instead of individual infractions. After violations are placed in one of the groups, IRT queries, “Is there a violation in the group?”
A yes indicates the presence of at least one violation in the group, and then the group is assigned a weight severity that is not affected by the number of unique violations within the group, allowing for the disparity in motor carrier operations and in how different states conduct inspections.
The caveat, Bryan said, is carriers must trust “the IRT magic machine.” He said SambaSafety licenses the software that handles the complex scientific calculations involved.
“We don’t have to understand exactly what goes into computing the scores, we just have to know that this is science that is time- and battle-tested across the world,” Bryan said.
New CSA frontier
The new CSA system still will rely on the same old data, which often is flawed, but Bryan said he likes the direction CSA is heading.
“CSA will now compute a single score for every motor carrier based on their safety culture,” Bryan said.
The revamped model also is “time proof,” he said, allowing for the injection of more data in the future, and the potential addition of an eighth FMCSA BASIC, called Beyond Compliance, rewarding carriers who go above and beyond requirements, like investing in advanced safety technologies.
The new system won’t go live until late next year, at the earliest, and even then Bryan anticipates plenty of issues popping up, including how to address “unfair” scores and IRT’s dislike of violation “variety.”
But he’s advising carriers to get ahead of the system now—with SambaSafety’s help.
The company began providing a preview of its CSA/IRT Scorecard in October and planned to release the new product in November.
Visit vigillo.com/products/csa-solutions/csa-irt-scorecard for more information.
