Renesas develops in-vehicle SoC for autonomous driving with a video processing delay of only 70ms

Japan Renesas Electronics and Renesas Systems Design Co., Ltd. developed an in-vehicle computing SoC for future autonomous driving, and delivered a speech at the International Society for Semiconductor Integrated Circuit Technology "ISSCC 2016" on February 1, 2016 (Session 4.4). Renesas also demonstrated the use of the SoC to simultaneously process 12-channel Full HD (1980 x 1080 pixels, 30 frames per second) images at the same day's presentation. This is the third generation of Renesas In-Vehicle Information IC "R-CAR", which is scheduled to begin mass production in March 2008.

Automotive SoC developed by Renesas
(Image courtesy of ISSCC) (click to enlarge)

The demonstration will be on site. Nearly on the vehicle instrument, in the distance, the full HD image is displayed on 12 screens on the display. (Click to enlarge)

The board used for the demo. It is said that the SoC developed this time is installed below the middle heat sink and the like. (Click to enlarge)


According to reports, this SoC integrates multiple CPU cores, GPU cores and six (17) video processing processors on one chip. Renesas believes that by 2020, the various instruments, driver assistance systems (ADAS), audio and multiple displays of the car will be integrated, and this integrated SoC is developed for this purpose.

The SoC developed this time is manufactured using the 16nm FinFET process. The core voltage is 0.8V and the operating frequency is 400MHz. The external memory is envisioned to use the LPDDR-3200.

The intended use method is to use a video processing processor group to process images of an in-vehicle camera or the like for entertainment purposes, and to use these image data for object recognition and the like for ADAS and subsequent autopilot applications. The video that can be processed per second is 750 million pixels. If it is a full HD image of 30 frames per second, this is equivalent to an image of about 12 channels. It also supports 4K images, but the number of channels is smaller.

Reduce processing delay by 40% and reduce total delay time to 70ms through pipeline processing

If you simply add pixels, you can also increase the number of processor cores. According to Renesas, one of the key points of this development is to reduce the video processing delay. The reason is that even at low speeds such as parking, the image delay must be less than 100ms.

Video processing generally first uses a stream processor (SP) to process compressed video frames, and then uses a codec processor (CP) for decoding and the like. Conventional video processing ICs use different SPs for frame processing, and in order to eliminate such variations, buffering of several frames is required after SP processing. Only such a buffer will result in a delay of nearly 100ms.

In order to solve this problem, Renesas chose the video coding technology to make the processing time of the SP and so on constant, and waited for the SP processing of each frame to end, and handed over the data to the CP every few pixels. Thus, buffering is no longer needed after SP processing, thereby implementing pipeline processing of video data. The delay between SP and CP is greatly reduced to 1ms. According to reports, including the subsequent distortion compensation processing, the overall delay of video processing is reduced to 70ms.

However, this is not the ultimate goal. Fully automatic driving without a driver is expected to begin in the second half of the 2020s. If automatic driving is achieved, the processing delay must be further reduced. "The future goal is to reduce the processing delay of compressed video data to less than 10ms by reducing the frame buffer time before SP processing" (Chief Engineer, Third Division, Cockpit Business Promotion Division, Renesas Systems Design First Development Division, In the current ISSCC speech, Wang Yuecheng 2).

The storage controller sets the "back door" for the CPU core.

Another key point of the SoC is the ability to compress data as it is written to the memory by the SoC. For example, if 12 channels of full HD video data are input to the output memory in an uncompressed form, and the data encoding and decoding process is performed by the CP, a bandwidth of 20 GB/sec is required, which occupies 40% of the storage bandwidth. This may adversely affect the object recognition processing that the CPU core and the GPU core are responsible for. Therefore, Renesas saves storage bandwidth by compressing data while accessing the memory.

However, there is a problem with this. That is, the video data used by the CPU core and the like are basically uncompressed, so the compressed data stored in the memory cannot be directly used. In response to this, Renesas set the "Shadow space" of the compressed data in the storage controller. When the CPU core accesses the compressed data, it accesses this space and decompresses the data.

According to reports, in this way, the bandwidth when the CP accesses the memory is reduced to 30% when it is not compressed, and the bandwidth when other processors access the memory is reduced to 50% when it is not compressed. As a result, the power consumption of the bus connecting the processor or memory is reduced by approximately 50%, and the overall power consumption of the SoC is also reduced by 20% to 197mW.

There is still a key point in this development. That is, this SoC meets the standard "ISO26262" of the vehicle IC failure frequency. In this regard, Renesas introduced the ISSCC 2016 Session 4.5.