The regime for assessing the safety of self-driving car systems is under scrutiny after consultants discovered errors in an analysis of Tesla crashes by a US government agency, which then delayed release of the data for two years in an apparent attempt to divert attention from the issue.

Quality Control Systems Corp (QCSC) made a deep-dive into data that purported to back up an impressive claim by NHTSA, the powerful US road safety body, and Tesla that the Autosteer system in Autopilot had reduced serious accidents by 40% — a game-changing improvement.

Tesla boss Elon Musk tweeted the results as NHTSA’s findings were released in January 2017.

But QCSC found that the NHTSA analysis failed to take into account all the mileage driven by the 43,781 vehicles studied. In fact, NHTSA only used mileage data for 14,791 vehicles. As a result, the crash rate before Autosteer was inflated, leading to the wrong conclusion, says QCSC.

“The importance of this research goes well beyond the specific issues addressed in our statistical analyses,” said QCSC. “The larger question is whether the field experience of autonomous vehicles and advanced driver-assistance systems will be fairly and transparently assessed by public officials.”

When QCSC, which specialises in data analysis as a business tool, approached NHTSA for access to the publically-funded research, they were refused. The log-jam took two years to break, only after QCSC threatened court procedings to obtain the data via a freedom of information request.

In response to a request for a comment from Autocar UK, the Washington-based NHTSA released a limited statement: “The agency is reviewing the report released by Quality Control Systems Corp. with interest and will provide comment as appropriate.”

QCSC’s report is titled “NHTSA’s Implausible Safety Claim For Tesla’s Autosteer Driver Assistance System” and looked at data for airbag deployments of MY2014 – 2016 Model S and MY2016 Model X models equipped with Tesla’s Autopilot.

QCSC’s analysis of the data, contained in a 24-page report, found that airbag deployments actually increased from 0.76 million to 1.21 million, a 59% rise, rather than decreased.

“Before and after comparisons of the resulting crash rates are unbiased by missing data for exposure mileage because there are no missing data in this subset of the data,” said QCSC in its report.

When NHTSA analysed airbag deployment and mileage travelled data supplied by Tesla, NHTSA came to the conclusion that there were 1.3 deployments per million miles before Autosteer and 0.8 million after – a 40% reduction.

NHTSA’s findings were widely reported in January 2017, partly because of the fatal accident in Florida six months earlier when a Model S operating with Autopilot ploughed into a truck, having failed to detect the vehicle making a turn across the car’s path.

Autopilot is a Level 2 advanced cruise-control self-driving system that allows the driver to temporarily relax their grip on the steering wheel. Autosteer is the lane-keeping part of the Autopilot system, which also facilitates lane changes when the indicator is operated.

Introduced in 2014, Autosteer was enhanced in 2016 with an upgrade. Significantly, it means a lane change can be executed with less oversight by the driver.

Ultimately Tesla aims for automatic lane-changing manoeuvres in Level 3 or 4 autonomy.