Use Cases of Augmented Reality in Vehicles

Augmented Reality in the Automotive Industry

The global AR market was worth 15,2 billion US dollars in 2021 and analysts estimate the market to reach 90,8 billion US dollars by 2028. Augmented Reality is on the rise as the technology is maturing and different industry sectors see the potential in the technology. One of the markets that have understood the potential of AR is the automotive industry. According to Fortune Business Insights, the global AR in the automotive market size was 4,51 billion US dollars in 2021 but is expected to grow to 14,4 billion US dollars in 2028.  

Before digging deeper into the concept of AR in the automotive industry, it’s good to first understand what AR really is.  

More simply put, AR is used to provide additional information about the environment to the user. The enriched information offered by the AR is usually drawn on top of camera input or a see-through display.  

AR is today used in many different types of industries and products, such as healthcare, manufacturing, entertainment, sales, and marketing. An example of an industry segment that has taken advantage of AR are social media platforms, such as Snapchat, TikTok and Instagram, with social media AR filters that are used in photos to engage their users and improve user experience. As for the automotive industry, there are several ways to utilize AR technology: 

In-vehicle AR applications:  

• AR-enhanced navigation 
• ADAS visualization 
• Park assists 
• Point of interest highlights 

Off-vehicle AR applications:  

• Design, prototyping, and production;
  e.g.  AR-powered testing and prototyping processes 
• Maintenance and after-sales services;
  e.g. Training maintenance staff, vehicle maintenance instructions 
• Sales and marketing;
  e.g. AR-powered digital showrooms and guides 

As you can see, there are various in-vehicle and off-vehicle use cases for AR. Thus, in this article, we’ll focus on the in-vehicle AR use cases.  

Augmented Reality Displays

The two main display options, or ways to apply AR for in-vehicle use are AR HUDs (Heads Up Displays) and AR over Video feed.  

AR for Heads-up Displays

AR HUDs are the most natural way to visualize AR information as its directly in the user’s Field of View. BMW was back in the day one of the first OEMs who introduced the concept of AR HUDs for cars, where the HUD system was the first step toward utilizing AR in vehicles.  

The HUD (Heads-up-display), is a feature where vehicle data, such as current speed and fuel data are projected onto the windshield. HUD systems were first seen in fighter aircrafts during World War 2. The technology made it possible for the pilots to get flight information displayed on the same visual plane as the objects in the environment, eliminating the need to look back and forth between projections and the exterior environment.  

But what is then an AR HUD? AR HUDs function in the same way as a normal HUD, but in addition to just displaying basic vehicle data, AR HUDs also displays enriched data about the surroundings around you. Most of the HUDs today are 2D, but in the future there will be 3D HUDs that allow to visualize depth information on the windshield.  

Example of an AR HUD overlay by Basemark

Video AR for Heads-down Displays

Video AR is another way to use AR technology in cars. The difference between video AR and AR HUDs is that video AR is not something that is displayed on the windshield. Video AR is real-time camera footage from the front–facing camera that is used in heads-down displays (HDD), such as, the instrument cluster or on the central display system where enriched data is rendered on top of the camera input.

Example of an Video AR cluster by Basemark

Video AR for Standalone Devices (aftermarket AR application) 

A fairly new AR display concept that is getting more attention is automotive AR for standalone devices like smartphones. This concept is a way for OEMs to bring AR technology to older vehicle fleets that weren’t originally produced to support AR functionality. With this concept, consumers might in the future, have the option to purchase an add-on subscription to access automotive AR features, where a smartphone (or some other device) will be connected to the car and project the AR.  

Use cases of augmented reality in vehicles

The use cases of AR in vehicles are many, and they are expanding all the time. The use cases listed below can be applied to both AR HUDs and Video AR.  

AR Enhanced Navigation  

Using AR for navigation purposes increases the ease, accuracy, and safety of navigation. You have probably experienced the struggle of determining if the next turn is coming on the next crossing or the following since the navigation map doesn’t correspond to the surroundings. Your attention is desperately flicking between the road and the display until you realize you just missed the crossing where you should have turned.  

In the case of video AR-enhanced navigation, the front-view camera sends live footage to the central information display, where navigation features enrich the live footage showing the driver exactly where to turn next. Basemark has implemented a video AR navigation feature for the electric BMW iX model, read more about the collaboration here.  

AR enhanced navigation experiences are also applicable on HUDs, where the navigation will take place on the windshield – eliminating the need for the driver to move the head and sight away from the road.  

Example of our 1st generation AR Enhanced Navigation running on public roads since 2021

ADAS Visualization

Advanced Driver Assistance System, also called ADAS, are a set of technologies that assist drivers in driving and parking. As we reach higher levels of autonomous driving, the need to understand how well the vehicle can identify and make decisions based on different driving scenarios is crucial. Using AR to visualize the ADAS detections allows the driver to understand the reasoning behind actions while adding an additional level of safety measures.  

Listed below are common ADAS visualization functions that are typically enhanced with AR: 

Object detection

Object detection makes it easier for the driver to detect moving or stationary objects, such as cars in front of you, pedestrians, animals, traffic signs, and other possible roadblocks. With object detection in an AR HUD, you will be able to detect the object through markings on the windshield, and you will be able to see, for example, how far away the object is from your vehicle or the speed of the object. The use cases of object detection are many, and the most common features are;  

• Vehicle distance & Collision warning 
  Vehicle distance warning is activated when there is a sudden movement close to your vehicle, such as when a car in front of you stops or when a car from the side comes onto your lane. Collision Warning will detect the vehicle that possibly is in a collision course with your vehicle and display a sign on the windshield urging you to slow down or if the car is equipped with higher level ADAS, the car may even apply the brake automatically 

• Low-light pedestrian zone 
  Makes the driver aware of pedestrians by highlighting the zone where they are in, while giving additional information, such as how far away the pedestrians are from your vehicle and their walking speed, etc.  

 Lane detection

Lane detection will highlight the current lane you’re driving on to make it easier for the driver to stay in the middle of the lane. The AR can display a so-called “magic carpet” on the lane you are driving on. For example, the color of the carpet is green when you drive in the middle of the lane and turns red if you deviate from the lane. Optionally, instead of showing the magic carpet, the sides of the lane can also be highlighted. The lane detection can also visualize if you are not able to change to the lane next to you (e.g. emergency vehicle lane) The lane detection function is often combined with object detection features which makes it also possible to visualize unsafe lane changes based on the surrounding traffic.   

Example of our 2nd generation AR Navigation and AR Driver Assistance from Basemark

Although this feature might sound a bit basic at first, it’s actually a great feature that improves driver safety. Imagine driving in dim light, where there are several lanes alongside each other, or on a badly conditioned road. In situations like these, the lane enhancements can be a lifesaver.  

Parking Search Assist/Parking Assist

We all know the pain of searching for a parking spot, and in bigger cities, you might spend hours searching for a free spot. With Parking Search Assist, the AR can detect and map out on the windshield where there is a free parking spot. It would also display how big it is, which would help the driver determine if it is big enough to fit the car. In addition, Parking Assist can help you park your car by displaying the driving path for you. Thus, Parking Assist is a feature that is already common in cars today. Some premium cars even have already autonomous parking.   

Points of Interest Visualization

Point of interest visualization (POI) is an AR application that we’ll most likely see in cars in the future. Point of interest, also called location-based AR, combine GPS data with AR to generate relevant information or “points of interest” based on your current location. For example, while driving on a road, the POI application will detect and mark out where the nearest gas station or McDonald’s restaurant is located. In principle, it will work in a similar way as a navigation App on the phone, but the “points of interest” are displayed directly on e.g., the windshield. 

Benefits of Augmented Reality in Vehicles

Every time a new technology emerges, it has been developed to improve or make things work better than they do today. This is also true for augmented reality. 

Improving Driver Safety

One of the main reasons for utilizing AR in cars is to reduce car accidents and fatalities. AR technology helps the driver perceive the surroundings better and enhance reality, which reduces the cognitive load on the brain. This means that with the help of AR, you can improve driver reaction time. Imagine yourself driving on a country road at nightfall, and a deer runs onto the road. An AR HUD would then be able to warn you through red markings on the windshield that there’s an endangering object on the road. Without the AR HUD, some wouldn’t even notice the deer until it was too late.  

ADAS Combined with AR

As mentioned earlier, AR features in combination with Advanced Driver Assistance System applications are important keys to autonomous driving and advancements in vehicle safety.  

ADAS is a system that was originally developed to prevent car accidents caused by human errors. Most cars today include different kinds of ADAS applications, such as adaptive cruise control, pedestrian detection, lane departure warning, automatic braking etc. For example, when you get too close to the car in front of you the car automatically breaks, or when you’re driving outside the lane markings, the steering wheel automatically steers back onto the lane.   

Currently ADAS alerts are primarily indicated through audio warnings, flashing symbols, or vibrations in the steering wheel or the seat. AR alerts are visual that for example are displayed on the windshield. The combination of having alerts that evoke different human senses is likely to improve the safety of driving.  ADAS visualizations will also help driver to trust ADAS system more as driver constantly sees what computer detects while ADAS system is turned on and car operates autonomously. 

Improving Driver Experience

The automotive industry is currently going through a major shift, where CASE – connected, autonomous, sharable & electrified vehicles will soon be the new standard. In the context of CASE, augmented reality will play a major role in the user experience of the connected car. As more advanced AR features will take their place in vehicles, the more connected the car will be to the environment around the vehicle. It makes it possible to, for example, get enhanced information about vehicles around you or important information ahead about weather and road conditions along the route you’re going to drive.  

In addition, as an indirect result of advancing the use of AR technology to support the development of autonomous vehicles and driver safety allows for more innovative ways for improving driver experiences. It allows OEMs and Tier1-suppliers to think outside the box and develop features and experiences that no one would expect to be possible to experience in their own car. So long Bluetooth-connected smartphones – Hello virtual assistants and in-vehicle movie theatres!  

Challenges of Augmented Reality in Vehicles

While there are clear benefits of having AR in vehicles, there are some challenges that need to be addressed. 

User Experience Design

Although it’s cool to have enriched data reflecting on the windshield or displayed on the instrument cluster, it can also become overwhelming, if not confusing. Therefore, OEMs need to ensure that the information displayed is found relevant and doesn’t take the attention away from the driver and the actual driving situation.  

Lack of Development Tools

There are currently some vendors on the market today that offer hardware solutions for AR applications, such as screens and cameras. However, there are no dedicated toolchains for AR application development that have been developed specifically for creating automotive AR applications. Some use graphics toolchains that have been developed for the gaming industry, but there are some clear disadvantages of not using an automotive grade toolchain.  

AR Requires Wide Skillset of Expertise

Implementing AR requires specific skills within graphics and compute solutions, in addition to good understanding about navigation, ADAS, advanced computer vision etc. Also Augmented Reality solutions will require processes and knowledge about how to implement safety certifiable software (ISO 26262).  

Such expertise is not necessarily found in-house, which makes it challenging for OEMs to implement AR applications on their own.  

Computation Complexity

AR isn’t just graphics. AR is a combination of advanced graphics and compute solutions where the AR visualizations are based on ADAS data, as well as data from the infotainment system (e.g. navigation guidance). Therefore, to develop flawless AR graphics for vehicles, you need to have extensive knowledge about graphics and compute processing. For example, a common challenge to overcome is the latency issues between systems to ensure that relevant information is visualized in real-time without any delays. Solving this challenge requires advanced optimization of the connected graphics and compute processes.

Conclusion

Although some Augmented Reality applications are already available in cars today, the gap between how AR is utilized today versus the potential it withholds is immense. Within a few years, we will see a more extensive range of AR features in vehicles. It’s just a matter of speed of innovation and supporting technology that needs to catch up.