Tuesday, April 12, 2016

Driving to Safety: How Many Miles of Driving Would It Take to Demonstrate Autonomous Vehicle Reliability?

 One proposal to assess safety is to test-drive autonomous vehicles in real traffic, observe their performance, and make statistical comparisons to human driver performance. This approach is logical, but it is practical? In this report, we calculate the number of miles that would need to be driven to provide clear statistical evidence of autonomous vehicle safety. Given that current traffic fatalities and injuries are rare events compared with vehicle miles traveled, we show that fully autonomous vehicles would have to be driven hundreds of millions of miles and sometimes hundreds of billions of miles to demonstrate their safety in terms of fatalities and injuries. Under even aggressive testing assumptions, existing fleets would take tens and sometimes hundreds of years to drive these miles — an impossible proposition if the aim is to demonstrate performance prior to releasing them for consumer use. Our findings demonstrate that developers of this technology and third-party testers cannot simply drive their way to safety. Instead, they will need to develop innovative methods of demonstrating safety and reliability. And yet, it may still not be possible to establish with certainty the safety of autonomous vehicles. Therefore, it is imperative that autonomous vehicle regulations are adaptive — designed from the outset to evolve with the technology so that society can better harness the benefits and manage the risks of these rapidly evolving and potentially transformative technologies.

However, the myth of i.i.d (independent identical distribution) is clearly there.
We can answer this question by reframing failure rates as reliability rates and using success run statistics based on the binomial distribution (O’Connor and Kleyner, 2012).
Easiest way for Google to pass the test, is to build a miles long circle in no-man's land, and line dozens AVs bumper to bumper. Then like them run and log the miles. Of course, in the report Rand mentioned:
Perhaps the most logical way to assess safety is to test-drive autonomous vehicles in real traffic and observe their performance.
However, what is "real" traffic. Is the empty high way at 4:00 am real? Is the icy road in Rocky mountain real?
A good Texas driver is almost certainly road hazard close to Ski resort. (Just speaking for myself :) )

It frustrates me that the RAND report does not even bother to list i.i.d as their fundamental assumption.

Will write another longer post to discuss this topic. Some previous thoughts:

No comments:

Post a Comment