[libcamera-devel] [RFC] The ISP and Device Conf Files

[Hanlin Chen]

Hello everyone,

A topic I'd like to discuss is the possibility to use configuration
files for the pipeline handler, and how they should be managed.

We already have some implementation for static information. For
example, the Android yaml conf, the json files used in raspberry pi
and some hard coded information in camera_sensor_properties and in
pipeline handler implementation.

For the information might be in configuration files, I would
categorize them into 2 kinds:
Firstly there is information static to a sensor or lens, some examples
I can think of are
1. Sensor properties
  1. The delay needed for setting exposure and gain.
  2. Test pattern modes.
  3. The color transformation properties for raw capture user (DNG),
colorTransform, referenceIlluminant, etc.
  3. Other static information, like physical size, black level, CFA types, etc.
2. Lens properties
  1. static information, for example, aperture, focal length, hyperfocal, etc.

This kind of information is more like a database of a module for
libcamera to query, or a supplement for information which cannot be
retrieved from the driver. Currently we have
camera_sensor_properties.cpp to hard code this information, including
unitCellSize and supported test pattern mode.

For the static sensor/lens properties. An idea is to create a series
of {model name}.yaml to collect the hard coded information in
camera_properties and in the pipeline handler, and a class to read
them by the model name. If a model cannot be found, a default one is
provided, or an error can be thrown. The pipeline handler or any upper
layer (Android) could access the data.

Another kind is the information specific to a certain device (If
cannot be retrieved from the driver)
 1. Camera Orientation
 2. How sensor's grouped as dula/triple camera
 3. Calibration Data and its location
   1. Lens shading or distortion correction
   2. How to get calibration data
     1. EEPROM device path
     2. Certain file or partition
 4. Tuned data for image quality or ISP settings

I suppose other information is straightforward, but the tuned data may
need more explanation. Following examples are mostly based on IPU3,
but to some extent, they are general to IPU6, Rockchip and Qualcomm on
ChromeOS.

In theory, the tuned data is provided to a certain device, but in most
of the cases, for a platform, they are based on a sensor module for
easier management. For ChromeOS, IPU3, IPU6 and Rockchip is the case.
For RaspberryPi on libcamera, {sensor}.json is the case too. This
might introduce risks that two different devices with the same sensor
want different tuning, when they are from different OEMs. We've seen
the problem for some ChromeOS Qualcomm devices.

Who is going to read them? In most of the cases, only the IPA should
read them, the examples are: {sensor}.json files for RaspberryPi, and
aiqb files for closed IPU3 IPA. However, Intel IPU3 introduced a
series of tuned xml files for configuring ISP, and in libcamera's
current design, they could be read by the pipeline handler.

For the setting applied to ISP, Intel provides a tool to automatically
generate the settings, according to the stream's configuration.
https://github.com/intel/intel-ipu3-pipecfg. The file consists of a
limited set of IMGU configurations, which includes parameters needed
to set through the media controller and V4L2 API.

These configurations are grouped by the requested streams, resolutions
and usecase (still capture or video). Examples can be found here:
https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss/

Currently libcamera has a procedure to produce the settings in the
pipeline handler, according to a tool provided by Intel. However, IPU3
was not that well designed to guarantee a fully predictable behavior.
We still see problems with the automatically generated settings. In
practice, there were efforts to generate a xml file of a sensor, and
then Intel or OEM validates and adjusts all parameters manually. The
xml file then becomes a database for the camera stack to query what's
the validated choices we have for the sensor? Following is how the xml
file are used in Intel camera stack on ChromeOS:
1. The camera stack groups the streams into video and still
2. Search validated settings for video/still to satisfy requested streams
3. If found, configure both video and still pipes for processing
4. If not, reduce a video or still stream and re-search, the reduced
stream will be SW scaled by the other streams
5. The validated single stream request for video or still is
guaranteed, so at least one set of settings can be found

It's tuned based on device because its content may be modified
manually due to tuning decisions. One interesting case is of Google
Pixelbook Go (IPU3): There is a limitation on IPU3: the DPC(defect
pixel correction) and BDS (Bayer down sampling) can not be enabled at
the same time. One of the sensors used on Pixelbook Go is IMX208, and
it seems to have lots of defect pixels at high gain level (>8x gain).
In result, the image without DPC is not acceptable to the image
quality requirement. However, when BDS cannot not be enabled, the
video pipe can only do YUV scale on one stream. As a result, some
configurations may have problems. For example

    CIO2 outputs: 1920x1080
    CIO2 outputs ==> (Video IMGU YUV crop) ==> 1280x720 (stream A)
    CIO2 outputs ==> (Video IMGU YUV scale) ==> 640x360 (stream B)
    CIO2 outputs ==> (Still IMGU) ==> 1920x1080 (stream C)

Because we cannot do BDS on the CIO2 output, and only one steam can
have yuv scale, the steam A can only be cropped to 720p, but the FOV
would be sacrificed, which is not accepted. Another choice is that
only one stream is enabled for the video pipe, and do software scale
for stream B.

    CIO2 outputs ==> (Video IMGU YUV scale) ==> 1280x720 (stream A)
    1280x720 (stream A) ==> (SW scale) ==> 640x360 (stream B)

This would trade performance for better image quality. After a series
of discussions, the latter solution is chosen. In result, Intel
removes all possible settings in the xml which would cause the problem
manually, and the camera stack is forced to do software scale for the
extra stream B (because no validated choice to satisfy A and B). Note
that it's also possible that an OEM chooses a smaller FOV for
performance. If so, the xml should be modified accordingly. All
devices share the same camera stack implementation, but only xml files
are different.

Some other tuning considerations which may affect the setting to ISP:
1. Do downscales on bayer stage (for performance/bandwidth) or on YUV
stage (for quality/usage)
2. Limitation on certain blocks, For example, the IPU6 has a TNR block
which can process at most 1080p. In this case, if image quality
requirements think TNR is critical, we may choose to crop/scale before
the block.

Consideration For General Distribution
Although the ISP setting files are very convenient for devices with an
image provider, like ChromeBook or Android phone, it's not very
compatible with the distribution to the general linux, since there is
unlikely a ISP setting file defined for a random sensor. Thus, the
pipeline handler should always have a default procedure to generate
one, as the IPU3 pipeline handler has done.

Here are some ideas for the above problems:
1. An optional device yaml to indicate device specific information, it
can be installed by the device image provider like ChromeOS or Android
vendor or by third parties. My imagination for the contents are:

cameras
  0:
    type: single
    device_id: "\\_SB_.PCI0.I2C2.CAM0":
    orientation: front
    caliberation_type: nvmem
    nvmem_path: "/sys/bus/i2c/devices/i2c-INT3499:00/eeprom"
    isp_setting_file: data/setting/ipu3/ov5688.yaml # Installed by the
image provider or with libcamera
    ipa_tuning_file: data/tuning/ipu3/ov5688.aiqb
  1:
    type: single
    device_id: "\\_SB_.PCI0.I2C2.CAM0"
    orientation: front
    caliberation_type: nvmem
    nvmem_path: "/sys/bus/i2c/devices/i2c-INT3479:00/eeprom"
    isp_setting_file: data/setting/ipu3/imx208.yaml # Installed by the
image provider or libcamera
    ipa_tuning_file: data/tuning/ipu3/imx208.aiqb

# --------- Imaginary dual camera case section ----------
  2:
    type: dual
    device_id: "":
    orientation: front
    isp_conf_file: data/setting/ipu3/dual.yml # Installed by the image
provider or libcamera
    ipa_tuning_file: data/tuning/ipu3/dual.aiqb
    physic_cameras: "0", "1"

If a device conf is presented, pipeline handler could use it directly,
otherwise, it could generate default values according to the number of
sensors probed, as libcamera already has done.

4. For ISP setting and tuning files, the image provider could install
them or, like RaspberryPi json files, installed with libcamera.

I suppose the logic pipeline handler using the tuned files could be:
  1. If a device conf is specifying a file installed by the image
provider, use it.
  2. Otherwise, if there is a tuned file for the sensor model
installed by libcamera, use it.
  3. Otherwise, try to generate default settings

There could be a setting manager class for above logics, and for the
pipeline handler, it only queries whether the requested streams have a
validated setting without knowing where the information comes from.
Because the content structure for ISP settings/Tuning files are
specific to a certain platform, I suppose its interface should be
defined independently. For IPU3, the following is the structure I
would image:

sensor_modes:
  4224x3136:
      name: "4224x3136"
      binner: [1, 1]
      pixel_array: [4224, 3136]
      output: [4224, 3136]
  2112x1568:
      name: "2112x1568"
      binner: [2, 2]
      pixel_array: [4224, 3136]
      output: [2112, 1568]

video_imgu_settings:
  - sensor_mode: 4224x3136
    input: [4224, 3136]
    viewfinder: [1920, 1080]
    main: [4096, 3072]
    if: [4224, 3136]
    bds: [4224, 3136]
    filter: [4, 4]
    env: [60, 28]
    gdb: [4096, 3072]

  - sensor_mode: 4224x3136
    input: [4224, 3136]
    viewfinder: [1280, 960]
    main: null
    if: [4182, 3116]
    bds: [1632, 1216]
    filter: [4, 4]
    env: [12, 4]
    gdb: [1600, 1200]

still_imgu_settings:
  - sensor_mode: 4224x3136
    input: [4224, 3136]
    viewfinder: [1920, 1080]
    main: null
    if: [4216, 2386]
    bds: [4216, 2386]
    filter: [4, 4]
    env: [40, 28]
    gdb: [4128, 2322]

As the Intel IPU3 implementation, it provides validated settings for
both video and still IMGU, and the pipeline handler tries to query
them. If failed, reduce the stream number and re-query. The single
stream settings are always presented and validated. The pipeline
handler reports the adjusted configuration back to the user, so the
user can choose whether or how to do software scales for the reduced
stream.

The context may be different for each platform, so I suppose it's hard
to avoid discussing them case by case.

I'd love to hear your thoughts.