By Damian Nachman, Basemark Project Management and 3D Artist | December 11, 2020
The time has come for a dedicated automotive HMI performance evaluation tool. That’s why Basemark created BATS (Basemark Automotive Test Suite)
When it comes to automotive HMI development you need to be able to choose the correct optimization techniques for both the design and implementation of your target device. We all know just how time-consuming and expensive iterations can be, given the complexity of the real-time graphics world. One of the biggest challenges is nightmarish, late-term optimization that can damage or even destroy your delivery schedule. Unfortunately, most of the graphics benchmarks out there only simulate video game-specific use cases rather than automotive HMI use cases,. Those are not the right tools to deal with this problem.
But BATS is. No other existing graphics benchmark is able to:
- Realistically evaluate hardware performance using realistic Automotive HMI workloads
- Support a big variety of embedded platforms and graphics API’s
- Evaluate complex automotive setups compromised of multiple SoC’s and Displays
- Easily compare these platforms
- Tailor and fine-tune the workload configurations to match in-car implementation
- Account for computer vision (and in the future, also sensor fusion) implementations.
The Right Tools for the Right Job
BATS workloads use a mission-critical set of features optimized for automotive HMI development that will accelerate the time to market for a new project, an ongoing project, or a late project. Developers can get some information by running different graphics benchmarks like GFXBench – and carefully observing performance trends in relation to different scene statistics.
BUT this approach carries some serious risks. For instance, because video game workloads use completely different features compared to automotive HMI, gaming workloads focus on rendering features which are not relevant for Automotive / UI. (for example, large amount of light sources – or raytraced reflections)
On the other hand, these gaming-workloads ignore features which are important for HMI / UI, such as large amount of render layers, or large amount of dynamic text rendering.
BATS also includes computer vision tests, which are not considered in existing video game workload benchmarks.
For future iterations of BATS, we will also take Sensor Fusion into account, and plan to allow OEMs to test their own HMI implementation.
Most gaming benchmarks don’t provide sufficiently granular test results to be meaningful for automotive HMI development.
BATS Is an Automotive Relevant Benchmark
BATS allows benchmarking target embedded platforms, and receive valuable information on hardware bottlenecks.
These bottlenecks should be considered during both concept and implementation.
During implementation efforts, OEMs profile their workload. When performance is subpar – OEMs make iterations on design and implementation.
BATS “production ready” workloads allow for testing your platform, as early as the concept/requirements phase (before implementation), and thus eliminates unnecessary iteration time.
Automotive-specific benchmarks can save a lot of time and money by providing real automotive reference workloads that can close the gaps between design, implementation, and target performance.
If there is one thing worse than using the wrong tool for a job, it’s using no tool at all. Automotive HMI developers often don’t use any benchmarks before or during the development process.
This often leads to a situation where developers must then rely on trial-and-error: Implement in a certain way -> log the performance -> adapt the implementation -> repeat until required performance is achieved.
Realtime graphics implementations are complex, iterations take a long time and are very expensive.
This is a primitive, inefficient and time-consuming process that makes it hard to execute smooth, stable implementations within a reasonable amount of time. BATS eliminates this inefficient process by providing a performance measurement tool specifically designed for HMI workloads.
Optimizing both Design and Implementation
In Automotive HMI development projects, you need to choose the correct optimization techniques for both the design as well as its implementation for the target device.
Some devices, such as the NXP i.MX6 – while certainly capable of rendering real time graphics – are limited in terms of some performance characteristics, and therefore require careful design, implementation, and optimization in order to achieve acceptable performance.
Features can be implemented in a myriad of ways. Using BATS, technical artists can identify device capabilities by running reference workloads, manipulating parameters, and empirically analyzing the performance – and choose the correct implementation/optimization methods. This way you can achieve a model that is on-par with the original design, while running at an acceptable and consistent framerate.
Grasping the Trade-offs of Device Limitations
Before the correct implementation methods can be chosen, the device’s actual limitations need to be well understood. Optimization for real-time is a careful balancing act, and there is often a flipside to every method.
For example, If Texture-Memory bandwidth is identified as a serious bottleneck:
- The design implementation team would go around this limitation by using clever shaders that utilize texture channel packing in clever ways. This way, texture memory consumption is greatly reduced
- However, the flipside of such a method is performing more calculations in the shader program.
By analyzing the device’s capabilities, you can see if such a trade-off would result in a net-positive for performance. (Limited by memory bandwidth, but not by GPU load).
By creating a benchmark dedicated specifically to automotive HMI use cases, we can quickly and effectively pinpoint the device’s limitations and capabilities.
BATS introduces automotive industry relevant tests for machine vision, instrument clusters and IVI. BATS allows you to find bottlenecks quicky without unnecessary iterations. Using BATS before and during the development reduces both the time and cost of the development.
BATS Enables:
- Quickly understand the platform’s limitation and capabilities, in relation to automotive-relevant use cases
- Compare different SoCs / automotive platforms, and choose the correct setup
- Using benchmark results, understand platforms bottlenecks – and choose the correct implementation methods. (Asset design/production, Implementation and rendering techniques)
- Empower the design-implementation team and decrease iteration time.
The time has come to upgrade your toolkit. Let Basemark help you to go BATS!
