[libcamera-devel] [RFC] The ISP and Device Conf Files

[Laurent Pinchart]

Hi Han-lin,

Thank you for the proposal, and sorry for taking long to get back to
you.

On Tue, Nov 30, 2021 at 07:29:29PM +0800, Hanlin Chen wrote:
> Hello everyone,
> 
> A topic I'd like to discuss is the possibility to use configuration
> files for the pipeline handler, and how they should be managed.

A contentious topic widely discussed in private circles, let's see if
public discussions can help finding a common position :-)

> We already have some implementation for static information. For
> example, the Android yaml conf, the json files used in raspberry pi
> and some hard coded information in camera_sensor_properties and in
> pipeline handler implementation.
> 
> For the information might be in configuration files, I would
> categorize them into 2 kinds:
> Firstly there is information static to a sensor or lens, some examples
> I can think of are
> 1. Sensor properties
>   1. The delay needed for setting exposure and gain.
>   2. Test pattern modes.
>   3. The color transformation properties for raw capture user (DNG),
> colorTransform, referenceIlluminant, etc.
>   3. Other static information, like physical size, black level, CFA types, etc.
> 2. Lens properties
>   1. static information, for example, aperture, focal length, hyperfocal, etc.
> 
> This kind of information is more like a database of a module for
> libcamera to query, or a supplement for information which cannot be
> retrieved from the driver. Currently we have
> camera_sensor_properties.cpp to hard code this information, including
> unitCellSize and supported test pattern mode.
> 
> For the static sensor/lens properties. An idea is to create a series
> of {model name}.yaml to collect the hard coded information in
> camera_properties and in the pipeline handler, and a class to read
> them by the model name. If a model cannot be found, a default one is
> provided, or an error can be thrown.

I don't think we can meaningfully operate with no sensor information at
all in most cases. At least the gain model (currently located in
src/ipa/libipa/camera_sensor_helper.cpp, but we're considering moving
that to camera_sensor_properties.cpp and handling the conversion between
standardized sensor controls and V4L2 controls in the CameraSensor
class) is required for any algorithm to function correctly. Other
information may be less crucial (color-related data or lens static
information for instance).

> The pipeline handler or any upper layer (Android) could access the
> data.

The data that the Android HAL requires should be exposed by the
libcamera native API. Is there any specific use case for accessing any
of the data listed above directly from the Android HAL instead of going
through the libcamera public API (which will of course need to be
extended to expose missing data) ?

> Another kind is the information specific to a certain device (If
> cannot be retrieved from the driver)
>  1. Camera Orientation
>  2. How sensor's grouped as dula/triple camera
>  3. Calibration Data and its location
>    1. Lens shading or distortion correction
>    2. How to get calibration data
>      1. EEPROM device path
>      2. Certain file or partition

We need an infrastructure to abstract this, to support different storage
backends for per-unit calibration data, transparently for upper layers.
Storage backends could be auto-detected to some extent (for instance,
the new ancillary links in the media controller API could help pointing
the camera sensor entity to the EEPROM associated with it when the
kernel is aware of the relationship between the two), but we'll also
likely need some ways to direct libcamera to manually selected storage
backends (such as files, or even partitions, I've seen both being used).
Note that it may also be possible to develop tools that would read those
storage backends and write the contents to a file, to be run once during
installation of libcamera for instance, and then use the file only at
runtime. I'm not sure if there are circumstances where would be
desirable compared to reading the backend directly at runtime (during
initialization).

>  4. Tuned data for image quality or ISP settings
> 
> I suppose other information is straightforward, but the tuned data may
> need more explanation. Following examples are mostly based on IPU3,
> but to some extent, they are general to IPU6, Rockchip and Qualcomm on
> ChromeOS.
> 
> In theory, the tuned data is provided to a certain device, but in most
> of the cases, for a platform, they are based on a sensor module for
> easier management. For ChromeOS, IPU3, IPU6 and Rockchip is the case.
> For RaspberryPi on libcamera, {sensor}.json is the case too. This
> might introduce risks that two different devices with the same sensor
> want different tuning, when they are from different OEMs. We've seen
> the problem for some ChromeOS Qualcomm devices.

Tuning definitely needs to be based on the camera module, not the camera
sensor only. The lens needs to be taken into account. This is something
that we've discussed with Raspberry Pi, and there's a general agreement
that a solution is needed, we just haven't designed it yet :-) We were
wondering if a mechanism where some data common to multiple modules
based on the same sensor could be shared in a single file. To be
researched.

> Who is going to read them? In most of the cases, only the IPA should
> read them, the examples are: {sensor}.json files for RaspberryPi, and
> aiqb files for closed IPU3 IPA. However, Intel IPU3 introduced a
> series of tuned xml files for configuring ISP, and in libcamera's
> current design, they could be read by the pipeline handler.
> 
> For the setting applied to ISP, Intel provides a tool to automatically
> generate the settings, according to the stream's configuration.
> https://github.com/intel/intel-ipu3-pipecfg. The file consists of a
> limited set of IMGU configurations, which includes parameters needed
> to set through the media controller and V4L2 API.
> 
> These configurations are grouped by the requested streams, resolutions
> and usecase (still capture or video). Examples can be found here:
> https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss/
> 
> Currently libcamera has a procedure to produce the settings in the
> pipeline handler, according to a tool provided by Intel. However, IPU3
> was not that well designed to guarantee a fully predictable behavior.
> We still see problems with the automatically generated settings. In
> practice, there were efforts to generate a xml file of a sensor, and
> then Intel or OEM validates and adjusts all parameters manually. The
> xml file then becomes a database for the camera stack to query what's
> the validated choices we have for the sensor? Following is how the xml
> file are used in Intel camera stack on ChromeOS:
> 1. The camera stack groups the streams into video and still
> 2. Search validated settings for video/still to satisfy requested streams
> 3. If found, configure both video and still pipes for processing
> 4. If not, reduce a video or still stream and re-search, the reduced
> stream will be SW scaled by the other streams
> 5. The validated single stream request for video or still is
> guaranteed, so at least one set of settings can be found
> 
> It's tuned based on device because its content may be modified
> manually due to tuning decisions. One interesting case is of Google
> Pixelbook Go (IPU3): There is a limitation on IPU3: the DPC(defect
> pixel correction) and BDS (Bayer down sampling) can not be enabled at
> the same time. One of the sensors used on Pixelbook Go is IMX208, and
> it seems to have lots of defect pixels at high gain level (>8x gain).
> In result, the image without DPC is not acceptable to the image
> quality requirement. However, when BDS cannot not be enabled, the
> video pipe can only do YUV scale on one stream. As a result, some
> configurations may have problems. For example
> 
>     CIO2 outputs: 1920x1080
>     CIO2 outputs ==> (Video IMGU YUV crop) ==> 1280x720 (stream A)
>     CIO2 outputs ==> (Video IMGU YUV scale) ==> 640x360 (stream B)
>     CIO2 outputs ==> (Still IMGU) ==> 1920x1080 (stream C)
> 
> Because we cannot do BDS on the CIO2 output, and only one steam can
> have yuv scale, the steam A can only be cropped to 720p, but the FOV
> would be sacrificed, which is not accepted. Another choice is that
> only one stream is enabled for the video pipe, and do software scale
> for stream B.
> 
>     CIO2 outputs ==> (Video IMGU YUV scale) ==> 1280x720 (stream A)
>     1280x720 (stream A) ==> (SW scale) ==> 640x360 (stream B)
> 
> This would trade performance for better image quality. After a series
> of discussions, the latter solution is chosen. In result, Intel
> removes all possible settings in the xml which would cause the problem
> manually, and the camera stack is forced to do software scale for the
> extra stream B (because no validated choice to satisfy A and B). Note
> that it's also possible that an OEM chooses a smaller FOV for
> performance. If so, the xml should be modified accordingly. All
> devices share the same camera stack implementation, but only xml files
> are different.
> 
> Some other tuning considerations which may affect the setting to ISP:
> 1. Do downscales on bayer stage (for performance/bandwidth) or on YUV
> stage (for quality/usage)
> 2. Limitation on certain blocks, For example, the IPU6 has a TNR block
> which can process at most 1080p. In this case, if image quality
> requirements think TNR is critical, we may choose to crop/scale before
> the block.

I'd like to remind a core design decision for libcamera. Configuration
of ISP processing steps, such as DPC or TNR (Temporal Noise Reduction),
is the responsibility of the IPA. Configuration of the pipeline,
including deciding what combination of streams can be produced, what
resolutions can be obtained from them, and how to configure the various
scalers (binning/skipping on the sensor, bayer-domain scaler, YUV-domain
scaler, ...) and crop rectangles in the pipeline, is the responsibility
of the pipeline handler.

The IPA module and the pipeline handler are meant to be designed
together, and can (and should) communicate information to each other as
needed.

To reuse the examples you've provided, whether a platform needs to
enable DPC or not is something that I would envision being supplied
through the IPA tuning file. DPC operation could be decided by the IPA
module and communicated to the pipeline handler, which could then take
it into account to restricts options for the configuration of the
pipeline and the supported streams and resolutions. The same could apply
to TNR.

This being said, I don't disagree that OEM-specific overrides for the
pipeline configuration could be useful, for instance to take power
budgets or other device-specific constraints into account. We should
however not take this as an excuse to limit the capabilities of the
libcamera native API. It can be useful for users to pass power
consumption hints to libcamera for instance. Of course, we need to find
a proper balance here to avoid feature bloat and deadly complexity.

> Consideration For General Distribution
> Although the ISP setting files are very convenient for devices with an
> image provider, like ChromeBook or Android phone, it's not very
> compatible with the distribution to the general linux, since there is
> unlikely a ISP setting file defined for a random sensor. Thus, the
> pipeline handler should always have a default procedure to generate
> one, as the IPU3 pipeline handler has done.

For the sake of completeness, it has been mentioned before the libcamera
and/or Linux distributions could ship such configuration files for all
"generic" devices, possibly using DMI for device identification. I don't
think that would really work it, it sounds to more more like a way to
shove the problem under the carpet and let someone else deal with it :-)
If someone still thinks this would be a workable solution, we could
discuss it.

> Here are some ideas for the above problems:
> 1. An optional device yaml to indicate device specific information, it
> can be installed by the device image provider like ChromeOS or Android
> vendor or by third parties. My imagination for the contents are:
> 
> cameras
>   0:
>     type: single
>     device_id: "\\_SB_.PCI0.I2C2.CAM0":
>     orientation: front
>     caliberation_type: nvmem
>     nvmem_path: "/sys/bus/i2c/devices/i2c-INT3499:00/eeprom"
>     isp_setting_file: data/setting/ipu3/ov5688.yaml # Installed by the
> image provider or with libcamera

Assuming this would be a YAML equivalent to the XML file you mentioned
above, I don't think we should consider it an ISP setting file, but a
pipeline configuration file. In this context, pipeline configuration
means the logical equivalent of setting the V4L2 selection rectangles
and formats through the pipeline (I'm saying logical equivalent because
those parameters may not be configured through V4L2 selection rectangles
and formats or all devices, some of them could be set through ISP
configuration buffers for instance).

>     ipa_tuning_file: data/tuning/ipu3/ov5688.aiqb
>   1:
>     type: single
>     device_id: "\\_SB_.PCI0.I2C2.CAM0"
>     orientation: front

Did you mean to set the orientation to back, and have a different device
ID than for camera 0 ?

>     caliberation_type: nvmem
>     nvmem_path: "/sys/bus/i2c/devices/i2c-INT3479:00/eeprom"
>     isp_setting_file: data/setting/ipu3/imx208.yaml # Installed by the
> image provider or libcamera
>     ipa_tuning_file: data/tuning/ipu3/imx208.aiqb
> 
> # --------- Imaginary dual camera case section ----------
>   2:
>     type: dual
>     device_id: "":
>     orientation: front
>     isp_conf_file: data/setting/ipu3/dual.yml # Installed by the image
> provider or libcamera

Should this be isp_setting_file like for cameras 0 and 1, or is it
something different ?

>     ipa_tuning_file: data/tuning/ipu3/dual.aiqb
>     physic_cameras: "0", "1"

Logical cameras is a very interesting topic we'll have to tackle too :-)

> If a device conf is presented, pipeline handler could use it directly,
> otherwise, it could generate default values according to the number of
> sensors probed, as libcamera already has done.
> 
> 4. For ISP setting and tuning files, the image provider could install
> them or, like RaspberryPi json files, installed with libcamera.

Raspberry Pi tuning files are stored in the libcamera source tree for
convenience, but as the number of supported platforms will grow, I would
expect tuning files (for all platforms) to be stored somewhere else. We
could have a separate git tree with a tuning file database for instance.
I would expect OEMs to install tuning files on the devices they ship,
but I'd like to come up with a useful enough tuning file database
service to convince them to also submit the tuning files for their
devices to the public database.

This assumes, of course, that tuning data wouldn't be unit-specific.
When performing per-unit tuning, we could possibly split the tuning data
in per-unit and per-model data and still provide the latter in a public
tuning database, but I'm not sure how practical such a scheme would be.

Let's also note that we currently support overridding tuning data files
using an environment variable (LIBCAMERA_RPI_TUNING_FILE), and I'd like
to generalize that mechanism to support both a more generic envvar (with
the ability to select a per-camera tuning file), and a C++ API to let
applications specify a tuning file (or even possibly tuning data).
There's lots of things that could be done here, we also have to find a
good balance between a powerful API and an overcomplicated system
(especially given that I would expect tuning file overrides to be mostly
used during development).

> I suppose the logic pipeline handler using the tuned files could be:
>   1. If a device conf is specifying a file installed by the image
> provider, use it.
>   2. Otherwise, if there is a tuned file for the sensor model
> installed by libcamera, use it.
>   3. Otherwise, try to generate default settings

Are we still talking about tuning data or, or about pipeline
configuration data ? Reading the rest of your text below I would say
it's the pipeline configuration data, but I don't think item 2 in the
list above would apply to that.

> There could be a setting manager class for above logics, and for the
> pipeline handler, it only queries whether the requested streams have a
> validated setting without knowing where the information comes from.
> Because the content structure for ISP settings/Tuning files are
> specific to a certain platform, I suppose its interface should be
> defined independently. For IPU3, the following is the structure I
> would image:
> 
> sensor_modes:
>   4224x3136:
>       name: "4224x3136"
>       binner: [1, 1]
>       pixel_array: [4224, 3136]
>       output: [4224, 3136]
>   2112x1568:
>       name: "2112x1568"
>       binner: [2, 2]
>       pixel_array: [4224, 3136]
>       output: [2112, 1568]
> 
> video_imgu_settings:
>   - sensor_mode: 4224x3136
>     input: [4224, 3136]
>     viewfinder: [1920, 1080]
>     main: [4096, 3072]
>     if: [4224, 3136]
>     bds: [4224, 3136]
>     filter: [4, 4]
>     env: [60, 28]
>     gdb: [4096, 3072]
> 
>   - sensor_mode: 4224x3136
>     input: [4224, 3136]
>     viewfinder: [1280, 960]
>     main: null
>     if: [4182, 3116]
>     bds: [1632, 1216]
>     filter: [4, 4]
>     env: [12, 4]
>     gdb: [1600, 1200]
> 
> still_imgu_settings:
>   - sensor_mode: 4224x3136
>     input: [4224, 3136]
>     viewfinder: [1920, 1080]
>     main: null
>     if: [4216, 2386]
>     bds: [4216, 2386]
>     filter: [4, 4]
>     env: [40, 28]
>     gdb: [4128, 2322]
> 
> As the Intel IPU3 implementation, it provides validated settings for
> both video and still IMGU, and the pipeline handler tries to query
> them. If failed, reduce the stream number and re-query. The single
> stream settings are always presented and validated. The pipeline
> handler reports the adjusted configuration back to the user, so the
> user can choose whether or how to do software scales for the reduced
> stream.

Note that, beside the validation and adjustment mechanism, the pipeline
handler must also expose static information about the supported streams
and their resolutions. I don't think the above precludes that.

> The context may be different for each platform, so I suppose it's hard
> to avoid discussing them case by case.
> 
> I'd love to hear your thoughts.

One point we haven't discussed is how a platform that doesn't have an
OEM configuration override for the pipeline configuration would be able
to compute the pipeline configuration automatically. For some platforms
that's fairly easy (for instance with Raspberry Pi, the constraints of
the ISP are fairly well understood, especially by Raspberry Pi, and
they're great contributors to libcamera :-)). For other platforms,
that's more difficult (I'm thinking about the IPU3 in particular here,
the hardware is old, and it's hard to get fresh and accurate
information, Intel has moved on). While I'm not opposed to an OEM
override mechanism for the pipeline configuration, I don't want this
mechanism to be used by vendors (OEMs or SoC vendors) to provide a
half-baked (or even completed unbaked) implementation of automatic
pipeline configuration in the pipeline handler. That would leave users
stuck with only what OEMs provide, for specific platforms, and would
defeat the point of libcamera which is to make cameras usable for
everybody.

-- 
Regards,

Laurent Pinchart