Saturday, February 27, 2016

Why You Shouldn’t Be Allowed to Drive | TIME

Wow, this is earlier than I thought. My inner contrarian is yelling.


Spreadsheets: The Original Analytics Dashboard · Simply Statistics

Steven Levy wrote the following about the original granddaddy of spreadsheets, VisiCalc. 

Already, the spreadsheet has redefined the nature of some jobs; to be an accountant (statistician) in the age of spreadsheet (big data) program is — well, almost sexy. And the spreadsheet (big data) has begun to be a forceful agent of decentralization, breaking down hierarchies in large companies and diminishing the power of data processing.

There has been much talk in recent years about an “entrepreneurial renaissance” and a new breed of risk-taker who creates businesses where none previously existed. Entrepreneurs and their venture-capitalist backers are emerging as new culture heroes, settlers of another American frontier. Less well known is that most of these new entrepreneurs depend on their economic spreadsheets as much as movie cowboys depend on their horses.
Simply Statistics suggested that you replace "accountant" with "statistician" and "spreadsheet" with "big data" and "you are magically teleported into 2016."

Of course, the combining of presentation with computation comes at a cost of reproducibility and perhaps quality control. Seeing the description of how spreadsheets were originally used, it seems totally natural to me. It is not unlike today's analytic dashboards that give you a window into your business and allow you to "model" various scenarios by tweaking a few numbers of formulas. Over time, people took spreadsheets to all sorts of extremes, using them for purposes for which they were not originally designed, and problems naturally arose.
So now, we are trying to separate out the computation and presentation bits a little. Tools like knitr and R and shiny allow us to do this and to bring them together with a proper toolchain. The loss in interactivity is only slight because of the power of the toolchain and the speed of computers nowadays. Essentially, we've brought back the Data Processing department, but have staffed it with robots and high speed multi-core computers.
Other tools include IPython Notebook, and GitHub. Also importantly, we may need an alternative to the ubiquitous Excel. More and more computational powers are added to it in each update. What we need is a strip-down spreadsheet editor with some basic data validation functionalities, but no data analysis / visualization  built-in. A software package force us to make distinction between data input and processing/analysis/presentation.
VisiCalc running on Apple IIc, 1983. Photo by Mark Mathosian.

Job opening - for a data graphics editor!

Agree with Andrew Gelman that there should be more this sort of jobs. More elegant visualizations, kill the junk charts.

Quote from the job posting:

We have a job opening for a new position we’re calling “data graphics editor.” I’ve been having trouble attracting the right kind of candidate. In my search, I may have been leaning too heavily toward graphic design skill sets when perhaps I should be looking in your world of statistical graphics skills. My search led me to your door. Would you have any advice for me on where I might go to get this job posting in front of the right audience of students, grads, or practitioners?
Some of there recent graphs.

Friday, February 26, 2016

Autonomous cars in the snow... and more

“The maps we create contain useful information about the 3D environment around the car, allowing it to localize even with a blanket of snow covering the ground.”

The autonomous vehicles create the maps while driving the test environment in favorable weather. Technologies automatically annotate features like traffic signs, trees, and buildings later. Then, when the vehicles cannot see the ground, they detect above-ground landmarks to pinpoint themselves on the map, which they then use to drive successfully."

via Michigan Today

An average Homo Sapiens driver from Texas would perform horribly on the snowy mountain road in Colorado? However, there is no law to keep him/her off the road. How about a robot that has passed the safety test in Sunny California. (My adviser, Dr. Mosleh, loved to tell us how he was saved by an 18-wheeler on snowy road in Maryland when he first took the job at UMD. I secretly think that is why he moved back to UCLA.)

I support safety regulation that hold a high bar for AV, especially in the beginning years. True unknown risks are introduced by new technology. ( I hate to say this, but still, unknown unknown.) Also the public's perception of the risk would be high, only because they are not familiar with the technology. As discussed in previous post, the regulation/test should address the risky scenarios, not just how many miles traveled. The methodology discussed in my Ph.D. dissertation might be expanded to generate the risky scenarios.

Thursday, February 25, 2016

Boston Dynamics’ Marc Raibert on Next-Gen ATLAS: “A Huge Amount of Work” - IEEE Spectrum

Yes. And poking future overlord?

via IEEE Spectrum

I sincerely hope the future self-aware robots won't have access to these videos.

Saturday, February 20, 2016

Robot Art Raises Questions about Human Creativity

via MIT Technology Review

I am more concerned about the AI vs. Homo Sapiens in the survival game. It has nothing to do with sense of self, creativity or Turning Test. To survive, the only thing that counts.

Friday, February 19, 2016

Tufte in R!

Thank Lukasz Piwek, we can create Tufte-like graphs in R

Wednesday, February 17, 2016

Tuesday, February 16, 2016

Deep Learning Makes Driverless Cars Better at Spotting Pedestrians - IEEE Spectrum

No previous algorithms have been capable of optimizing the trade-off between detection accuracy and speed for cascades with stages of such different complexities. In fact, these are the first cascades to include stages of deep learning. The results we're obtaining with this new algorithm are substantially better for real-time, accurate pedestrian detection.

Deep Learning Makes Driverless Cars Better at Spotting Pedestrians - IEEE Spectrum

Sunday, February 14, 2016

Andrew Ng: Driverless Shuttle Bus - IEEE Spectrum

We believe the approach of creating a car that can autonomously drive everywhere and be safe everywhere is beyond today’s technology. Instead, we are looking initially at shuttle routes and bus routes, routes that are, perhaps, a modest 20 miles, driven in a big circle, or back and forth.
We think if all you are doing is driving a 20-mile route, the technology is indeed within striking distance of making that safe. We plan to commercialize this in three years and will be moving aggressively to get this to market.

Checking in with Andrew Ng at Baidu’s Blooming Silicon Valley Research Lab - IEEE Spectrum:

It is an (un)interesting approach that use shuttle bus as a start point. The environment is more predictable, and cost saving is fairly easy to materialize. Just think about the mass transit that happens daily in the giant manufacturing facilities, such as Foxconn campus.

Several such tests already started. The first self-driving bus to operate on fully public roads debuted in the Netherlands in Feb 2016. A pilot project by French Easymile is scheduled to bring two driverless shuttles to an office park in Bay Area in summer 2016.

I took a lot courses from Coursera, which Andrew Ng co-founded, including the famous Machine Learning he taught. Best wishes to him, and no matter who wins the race to get first truly autonomous driving application, it is win for the technology, and a major advancement on the journey of Homo Sapiens.

Friday, February 12, 2016

Gravitational Waves Discovered from Colliding Black Holes - Scientific American

The LIGO experiment has confirmed Albert Einstein’s prediction of ripples in spacetime and promises to open a new era of astrophysics

Tuesday, February 9, 2016

With Driverless Cars, How Safe Is Safe Enough? - Thoughts with Bayesian flavor [Draft]

 Well, that's the news from Lake Wobegon, where all the women are strong, all the men are good looking, and all the drivers are above average, including the robots driving the driverless. 

The Myth of i.i.d

And yet, it could be impossible to accurately gauge safety until many, many autonomous vehicles hit the roads. In the U.S., approximately one fatality occurs for every 100 million miles driven. To prove with 95% confidence that a driverless car achieves, at least, this rate of reliability by driving them around to see, it would require they be driven 275 million miles without a fatality. With a fleet of 100 autonomous vehicles (larger than any known existing fleet) driving 24/7, it would take more than 12 years to drive these miles. But with 10,000 such vehicles, it would take just six weeks. Regulators will have to find other ways of estimating safety, but widespread deployment will be the true test. If safety standards are too strict, this might never happen
The assumption behind the above calculation is that every mile the every test vehicle travels has the identical independent probability of accidents. In chapter 4 of the report Kalra co-authored, many different cases were discussed, where the AVs outperform human in some, and human drivers outperform in some others, and some are challenging to both human and robotic drivers. 

Conditional Probability

There is one thing I learned from the excellent E. T. Jayens, every probability is a conditional probability. So, P(A|AV, H0) = Sigma(P(A|AV, Ci, H0)*P(Ci|AV, H0)
Human drivers' accident rate P(A|Human, H0) are available, and there are models to predict improvement without AVs.  

For regulation to setup the accepting criteria, 

  • Baseline of human reliability
  • a comprehensive list of test cases (Ci), and passing rate for each Ci
  • the modelling of likelihood of each Ci. Also the government can influence the likelihood, and make the AV and human drivers much safer.
  • ???


  • If we can train a pilot using simulator, can we train a robot with simulator? 
  • Can we test/validate a robot with simulator?
  • Open database for all autonomous vehicles developers?

Can the Robot Get a Driver's License

When I first went to the driving school, the instructor told me that everyone could get a driver's license, except the legally blind. The robotic driver can easily beat me in the road test, and get the driver's license. 


  • Ci's are not independent. (Need to pull out my math book...)
  • Scalibility: the outcome of Ci, and also the P(Ci|H0) can vary depending on how many AVs are on the road.
  •  X-ware: hardware (vehicle, sensor, etc.), software, driver/rider, environment, 
  • Interactions: AV2AV, AV2HD, AV2I, AV2Pedestrian
  • Software upgrade
  • AI learning, general learning, and adaptive learning toward the specific environment,  
  • ...

Nidhi Kalra, "With Driverless Cars, How Safe Is Safe Enough?"; Retrieved from
James M. Anderson, Nidhi Kalra, Karlyn D. Stanley, Paul Sorensen, Constantine Samaras, Oluwatobi A. Oluwatola, "Autonomous Vehicle Technology: A Guide for Policymakers"; Retrieved from