[libcamera-devel] [PATCH 0/1] Proposal of mapping between camera configurations and requested configurations

Thu Aug 13 14:51:49 CEST 2020

gentle ping


On Thu, Aug 6, 2020 at 3:17 PM Hirokazu Honda <hiroh at chromium.org> wrote:
>
> This is a proposal about how to map camera configurations and requested
> configurations in Android Camera HAL adaptation layer. Please also see the
> sample code in the following patch.
>
> # Software Stream Processing in libcamera
>
> _hiroh at chromium.org / Draft: 2020-08-06_
>
>
> # Objective
>
> Perform frame processing in libcamera to achieve requested stream configurations that are not supported natively by the camera hardware, but required by the Android Camera HAL interface.
>
>
> # Background
>
>
> ### Libcamera
>
> In addition to its native API, libcamera[^1] provides a number of camera APIs, for example, V4L2 Webcam API and Android Camera HAL3. The platform specific implementations are wrapped in libcamera core and a caller of libcamera doesn’t have to take care the platform.
>
>
> ### Android Camera HAL
>
> Chrome OS camera stack uses Android Camera HAL[^2] interface. Libcamera provides Android Camera HAL with an adaptation layer[^3] between libcamera core part and Android HAL, which is called Android HAL adaptation layer in this document.
>
> To present a uniform set of capabilities to the API users, Android Camera HAL API[^4] allows caller to request stream configurations that are beyond the device capabilities. For example, while a camera device is able to produce a single stream, a HAL caller requests three possibly different resolution streams (PRIV, YUV, JPEG). However, libcamera core implementation produces camera-capable streams. Therefore, we have to create three streams from the single stream produced by libcamera.
>
> Requests beyond the device capability is supported only in Android HAL at this moment. I describe the design in this document that the stream processing is performed in Android HAL adaptation layer.
>
>
> # Overview
>
>
> ## Current implementation
>
> The requested stream configuration is given by _camera3_device_t->ops->configure_streams()_ in Android Camera HAL. This delegates CameraDevice::configureStreams()[^5] in libcamera. The current implementation attempts all the given configurations and succeeds if and only if the camera device can produces them without any adjustments.
>
>
> ### libcamera::CameraConfiguration
>
>  It is CameraConfiguration[^6] that judges whether adjustments are required, or even requested configurations are infeasible.
>
> The procedure of configuration is that CameraDevice
>
>
>
> 1. Adds every configuration by CameraConfiguration::addConfiguration().
> 2. Assorts the added configurations by CameraConfiguration::validate().
>
> CameraConfiguration, especially for validate(), is implemented per pipeline. For instance, the CameraConfiguration implementation for IPU3 is IPU3CameraConfiguration[^7].
>
> validate() returns one of the below,
>
>
>
> *   Valid
>     *    A camera can produce streams with requested configurations.
> *   Adjusted
>     *   A camera cannot produce streams with requested configurations as-is, but can produce streams with different pixel formats or resolutions.
> *   Invalid
>     *   A camera cannot produce streams with either requested configurations or different pixel formats and resolutions. For instance, this is returned when the larger resolution is requested than the maximum supported one?
>
> What we need to resolve is, when Adjusted is returned, to map adjusted camera streams to requested camera streams and required processing.
>
>
> ## Stream processing
>
> The processing to be thought of are followings.
>
>
>
> *   Down-scaling
>     *   We don’t perform up-scaling because it affects stream qualities
>     *   Down-scaling is allowed for the same ratio to avoid producing distorted frames. For instance, scaling from 1280x720 (16:9) to 480x360 (4:3) is not allowed.
> *   Cropping
>     *   Cropping is executed only to change the frame ratio. Thus it must be done after down-scaling if required. For example, to convert 1280x720 to 480x360, first down-scale to 640x360 and then crop to 480x360.
> *   Format conversion
>     *   Pixel format conversion
>     *   JPEG encoding
>
>
> # Proposal
>
> Basically we only need to consider a mapping algorithm after validate(). However, to obtain less processing and better stream qualities, we should reorder given configurations within validate().
>
> I described how to map after validate() first, and next how to reorder within validate().
>
>
> ## How to map after validate()
>
> For each requested stream, we try to find a best-fit camera stream as follows.
>
>
>
> 1. Filter out smaller resolutions than the requested one.
> 2. Prioritize smaller resolutions to reduce number of processed pixels
> 3. If there is same ratio and same format as requested ones, select it
> 4. Otherwise, select one of the same ratio’s ones. If there is no same ratio one, select one of the same format’s ones.
> 5. If there is neither same ratio’s nor format’s one, select any convertible stream.
>
> The required processings are
>
>
> *   No-op [Same resolution and same format]
> *   Scale [Same ratio and same format, but different resolution]
> *   Pixel format conversion [Same resolution but different format]
> *   Scale and Pixel format conversion [Same ratio but different format and resolution]
> *   Scle and Crop [Same format, but different ratio]
> *   Scale, Crop and Pixel format conversion [Different ratio and format]
>
>
> ## How to sort within validate()
>
> Since up-scaling is not allowed in the proposal mapping, it is important to configure larger resolutions. Otherwise, configureStream() fails. Besides, producing as many as different ratios would be better so that we don’t have to get rid of captured content by cropping.
>
> This can be done in Android HAL adaptation layer, however I would do this within validate() because the above sorting strategy is general and each configuration class should have more information to prioritize the requested configurations.
>
> [^1]:
>      https://libcamera.org/index.html
>
> [^2]:
>      https://android.googlesource.com/platform/hardware/libhardware/+/master/include/hardware/camera3.h
>
> [^3]:
>      https://git.linuxtv.org/libcamera.git/tree/src/android
>
> [^4]:
>      https://developer.android.com/reference/android/hardware/camera2/CameraDevice#regular-capture
>
> [^5]:
>      https://git.linuxtv.org/libcamera.git/tree/src/android/camera_device.cpp#n934
>
> [^6]:
>      https://git.linuxtv.org/libcamera.git/tree/include/libcamera/camera.h#n28
>
> [^7]:
>      https://git.linuxtv.org/libcamera.git/tree/src/libcamera/pipeline/ipu3/ipu3.cpp#n62
>
> --
> 2.28.0.236.gb10cc79966-goog