Picture perfect: driving next-generation image processing for automotive

The automobile is no longer a metal box with four wheels that gets you from A to B. As technology has evolved, vehicles have become a dynamic platform for innovation which is changing the driver experience and taking us closer to the reality of truly autonomous driving. Advanced Driver Assistance Systems (ADAS) capabilities are integral to this evolution. New ADAS applications such as mirror replacement, driver drowsiness detection and pedestrian protection systems drive the need for enhanced image processing within the vehicle. Enabling these technologies requires an increasing number of cameras in vehicles, with most high end mass market vehicles expected to contain up to twelve cameras within the next few years, according to Strategy Analytics.

However, this type of complex camera technology must quickly process and analyze images under the most extreme conditions and be specifically designed for stringent automotive safety standards. Simply integrating standard camera technologies used in smartphones, or even consumer video cameras, is not an option. To address these challenges, today we're launching the ARM Mali-C71 image signal processor (ISP), the first product in our Mali Camera family. The Mali-C71 was designed from the ground up for ADAS System on Chips (SoCs) with automotive needs such as adjusting for challenging lighting and weather conditions in mind.

High end vehicles could contain up to 10/12 cameras in the future

The changing role of the Image Signal Processor (ISP)

As the number of cameras per vehicle increases and sensor fusion technologies get smarter, the ISP function – processing raw pixels into high quality images for displays – encoding and further processing by computer vision algorithms is being centralised within the ADAS SoC. Therefore the SoC is capable of managing multiple cameras and delivering reliability and the highest levels of safety in the automotive market. This requires a new approach with built-in functional safety features and parallel virtual processing output for computer vision applications. To date, implementing this in a single SoC has been an elusive goal. Until now.

Mali-C71: Automotive grade clarity and reliability

Automotive cameras need to provide the highest possible levels of clarity and reliability. Where the camera output is generated for a display, image quality is paramount. Take for example mirror replacement, as shown in the image below. The Mali-C71 offers ultra-wide dynamic range (UWDR) up to 24 stops (in some cases, providing detail beyond what the human eye can see). To put this in perspective, the best DSLR camera can achieve around 15 stops of dynamic range. It removes noise and processes the multiple exposures from the camera, creates an ultra-WDR frame and sends it across to a display or a computer vision engine.

To be able to detect a pedestrian in shadow when conditions are bright, the camera needs the necessary dynamic range to detect all elements of the scene it is capturing.

Where the camera is being used as an input to ADAS – for example as part of a pedestrian protection or driver drowsiness detection system – it needs to meet the most stringent reliability and functional safety standards. The Mali-C71 brings low latency and advanced error detection with more than 300 dedicated fault detection circuits to enable system-level certification to the highest standards (ISO26262, ASIL D and IEC 61508, SIL3). We deliver full reference software to control the ISP, sensor, auto white balance and auto exposure, with plans to develop complete automotive software designed for ASIL compliance.

Next generation ISPs need to simultaneously generate data that can be rendered on a display while also processing the data for use by the computer vision engine. These tasks have contrasting requirements, and the Mali-C71 provides one piece of hardware IP capable of providing two differently processed outputs from a single image pipeline.

The Mali-C71 is the first imaging product we’re bringing to market since our acquisition of Apical in 2016. OmniVision Technologies, a leading developer of advanced digital imaging solutions, has welcomed the development:

“OmniVision has been closely collaborating with the ARM ISP team (formerly Apical) for several years now, and see the Mali-C71 as an important advancement in image signal processing for emerging automotive applications. ARM’s expertise in ISP technology together with OmniVision’s best-in-class HDR image sensor technologies offers an industry leading solution for our automotive customers.”

Will Foote, senior partnership manager, OmniVision Technologies

As the automotive industry continues to accelerate towards realising the vision of fully autonomous vehicles, the Mali-C71 will deliver the high quality output for display that is necessary for drivers to quickly make correct choices, and simultaneously provides reliable data for computer vision engines to make safe decisions when the car is in control.