Cars turn to augmented reality

Car firms hope to turn windscreens into giant computer displays that overlay the real world with useful information, such as directions or even social media feeds.

The dangers of talking on a mobile phone or texting whilst driving are well known. So adding what, on first glance, look like added distractions may not seem like the best idea.

But researchers and car firms are increasingly looking to give drivers enhanced, or augmented reality (AR) that adds extra information ranging from directions to social media feeds to the view through the windscreen. And, perhaps the most unintuitive part of it is that it is designed to make the roads more secure for all of us.

“The biggest way augmented reality can be useful is in supporting safe driving,” says Professor Anind Dey, of the Human-Computer Interaction (HCI) Institute at Carnegie Mellon University. “The ability to enhance information around us that might otherwise be opaque to us is invaluable.”

AR is usually considered to be a live view of the real world, onto which extra data – usually pulled from the internet - is layered or superimposed. It is similar to the view of the world that Arnold Schwarzenegger had in the Terminator movies.

The technology is already well established elsewhere. For example, there are numerous smartphone apps that allow users to overlay the view from their camera with everything from directions and transport options to restaurant reviews and house prices. Point your phone at something of interest, power up your app and you are away.

Companies like Google are taking it one step further. Earlier this year the search giant unveiled its Google Glass project, a pair of minimal glasses that overlay a person’s vision with everything from weather data, location and diary appointments. It is an idea that has been seized upon by other firms, such as Microsoft.

And now AR is making its way into cars. It builds on so-called “heads up display” (HUD) technology that is commonly used in fighter planes and began to make its way into high-end cars in the late 1990s.

These HUDs are frequently used to project directions or speed onto the windscreen in the driver’s line of sight to minimize the amount of time the driver takes their eyes off the road. But augmented reality could allow these displays to go much further than that.

Looking glass

Car manufacturers such as BMW, Toyota, Mercedes and GM have shown off various prototypes in recent years. For example, a Mercedes concept unveiled earlier this year and known as Dice allowed drivers to conjure up information about passing places of interest by merely pointing at them while friends driving past the car show up as an icon on the windscreen, alongside their social network status.

Another concept from GM, known as the enhanced vision system, uses an array of sensors and cameras mounted inside and outside the vehicle to monitor the environment around the car and the driver’s eye and head movements. This information is then used to overlay the view from the windscreen with relevant information about driving conditions – such as the location of the building you are trying to find or dangers that you may not be aware of, such as an animal or child at the side of the road. Another mode marks the edges of the road when fog obscures the drivers view.

Other concepts take AR even further. Japanese car maker Toyota has shown off a prototype of “enhanced car windows” that would allow passengers to zoom in on places and objects of interest that they are passing.

However, the most extreme concept comes from designers at Keio University in Japan, who are working on technology that will make the back seat of a car appear transparent, so that when drivers are reversing they can see everything around them.

Their experimental setup uses cameras mounted outside the car to project an image inside the car in the driver’s field of view. Using similar technology to a heads-up display, the image is shown on a reflective screen mounted behind and between the two front seats. When a driver looks back over his shoulder, he sees the “real” image out of the rear view window continued, through augmented reality, right down the seats to the floor.

Innovations like this may be a long way from going into production. But other uses of AR could appear much sooner. For example, the most common use of AR is to overlay the world with directions that are overlayed on the road in front of you.

Almost all concepts show some variant of this and, earlier this year, electronics firm Pioneer claimed to introduce the world’s first GPS with AR, that can be installed in any vehicle.

Game view

This kind of innovation would be helpful to any driver. But Professor Dey and his team believe it could be especially useful for older people, whose sense of spatial awareness may be reduced. This is especially problematic when they must look down at a navigation system, for example.

“The amount of time being spent not looking out the windshield, and the number of times that occurs on a short drive, can get pretty high,” he explains. “Particularly when you are talking about elder drivers who may have diminished facilities for driving, you want to reduce the amount of time they are looking away.”

To test this theory, he devised a simulator that projects the navigation system across the full width of the screen. The driver sees the real world, but with a map superimposed on top. As turns or streets come up, the map updates like a panning computerised window that is locked to the real world around them.

Although the system is a prototype, it has given Prof Dey and his team clues about how a real system should work. For example, to ensure that one set of distractions is not replaced by another, he believes that drivers should be allowed to personalize a system so that they can receive notifications from the system in ways that feel intuitive to them, such as audio tones, or even by touch, using haptic feedback through the steering wheel and seats.

“We should be using technologies like augmented reality to personalise the driving experience much more so that we are currently.”

The trials have also flagged potential problems. For example, many of the subjects who have tried the system out in Professor Dey’s simulator have reported that it feels like playing a computer game – this detachment from reality is an issue that the engineers know needs to be addressed.

“You don’t want to lessen the impact of making a mistake,” Professor Dey explains.

In addition, and perhaps most importantly, more work needs to be done to work out what is the optimum amount of the windscreen that can be covered with AR graphics, he says.

After all, augmented reality is only useful if the driver is in touch with the reality behind it.

- BBC