[libcamera-devel] [RFC] The ISP and Device Conf Files

[Hanlin Chen]

Hi Laurent, Thanks for the kind comments.

On Tue, Dec 14, 2021 at 1:56 PM Laurent Pinchart
<laurent.pinchart at ideasonboard.com> wrote:
>
> Hi Han-lin,
>
> Thank you for the proposal, and sorry for taking long to get back to
> you.
>
> On Tue, Nov 30, 2021 at 07:29:29PM +0800, Hanlin Chen wrote:
> > Hello everyone,
> >
> > A topic I'd like to discuss is the possibility to use configuration
> > files for the pipeline handler, and how they should be managed.
>
> A contentious topic widely discussed in private circles, let's see if
> public discussions can help finding a common position :-)
>
> > We already have some implementation for static information. For
> > example, the Android yaml conf, the json files used in raspberry pi
> > and some hard coded information in camera_sensor_properties and in
> > pipeline handler implementation.
> >
> > For the information might be in configuration files, I would
> > categorize them into 2 kinds:
> > Firstly there is information static to a sensor or lens, some examples
> > I can think of are
> > 1. Sensor properties
> >   1. The delay needed for setting exposure and gain.
> >   2. Test pattern modes.
> >   3. The color transformation properties for raw capture user (DNG),
> > colorTransform, referenceIlluminant, etc.
> >   3. Other static information, like physical size, black level, CFA types, etc.
> > 2. Lens properties
> >   1. static information, for example, aperture, focal length, hyperfocal, etc.
> >
> > This kind of information is more like a database of a module for
> > libcamera to query, or a supplement for information which cannot be
> > retrieved from the driver. Currently we have
> > camera_sensor_properties.cpp to hard code this information, including
> > unitCellSize and supported test pattern mode.
> >
> > For the static sensor/lens properties. An idea is to create a series
> > of {model name}.yaml to collect the hard coded information in
> > camera_properties and in the pipeline handler, and a class to read
> > them by the model name. If a model cannot be found, a default one is
> > provided, or an error can be thrown.
>
> I don't think we can meaningfully operate with no sensor information at
> all in most cases. At least the gain model (currently located in
> src/ipa/libipa/camera_sensor_helper.cpp, but we're considering moving
> that to camera_sensor_properties.cpp and handling the conversion between
> standardized sensor controls and V4L2 controls in the CameraSensor
> class) is required for any algorithm to function correctly. Other
> information may be less crucial (color-related data or lens static
> information for instance).
>
> > The pipeline handler or any upper layer (Android) could access the
> > data.
>
> The data that the Android HAL requires should be exposed by the
> libcamera native API. Is there any specific use case for accessing any
> of the data listed above directly from the Android HAL instead of going
> through the libcamera public API (which will of course need to be
> extended to expose missing data) ?
Hmmm, No specific case. The original thought is simply that it's like
a static (and open) database for sensor modules, and it's not limited
to any layer which wants to read it.
What in my mind was the DNG color transforms for raw capture, which is
hardly used in pipeline handlers, and we can reduce control numbers a
little.
Indeed, if we can centrally fill those controls in CameraSensor, it
might be a good choice too.
>
> > Another kind is the information specific to a certain device (If
> > cannot be retrieved from the driver)
> >  1. Camera Orientation
> >  2. How sensor's grouped as dula/triple camera
> >  3. Calibration Data and its location
> >    1. Lens shading or distortion correction
> >    2. How to get calibration data
> >      1. EEPROM device path
> >      2. Certain file or partition
>
> We need an infrastructure to abstract this, to support different storage
> backends for per-unit calibration data, transparently for upper layers.
> Storage backends could be auto-detected to some extent (for instance,
> the new ancillary links in the media controller API could help pointing
> the camera sensor entity to the EEPROM associated with it when the
> kernel is aware of the relationship between the two), but we'll also
> likely need some ways to direct libcamera to manually selected storage
> backends (such as files, or even partitions, I've seen both being used).
> Note that it may also be possible to develop tools that would read those
> storage backends and write the contents to a file, to be run once during
> installation of libcamera for instance, and then use the file only at
> runtime. I'm not sure if there are circumstances where would be
> desirable compared to reading the backend directly at runtime (during
> initialization).
There is a VPD partition in ChromeOS for calibration data, other
modules like gyro may decide to store it there.
Besides the storage location, I'm not sure if there are any rules for
the content format. I suppose the data filled by different vendors
could be different.
The one for Windows products may be different to the one for other OS?

I think a file to cache the content is good for practice.
ChromeOS doesn't have a clear "installation" phase for any package, so
we may still need to load it at least for once at runtime (and reload
it if it's corrupted for safety).
>
> >  4. Tuned data for image quality or ISP settings
> >
> > I suppose other information is straightforward, but the tuned data may
> > need more explanation. Following examples are mostly based on IPU3,
> > but to some extent, they are general to IPU6, Rockchip and Qualcomm on
> > ChromeOS.
> >
> > In theory, the tuned data is provided to a certain device, but in most
> > of the cases, for a platform, they are based on a sensor module for
> > easier management. For ChromeOS, IPU3, IPU6 and Rockchip is the case.
> > For RaspberryPi on libcamera, {sensor}.json is the case too. This
> > might introduce risks that two different devices with the same sensor
> > want different tuning, when they are from different OEMs. We've seen
> > the problem for some ChromeOS Qualcomm devices.
>
> Tuning definitely needs to be based on the camera module, not the camera
> sensor only. The lens needs to be taken into account. This is something
> that we've discussed with Raspberry Pi, and there's a general agreement
> that a solution is needed, we just haven't designed it yet :-) We were
> wondering if a mechanism where some data common to multiple modules
> based on the same sensor could be shared in a single file. To be
> researched.
I've seen the same sensor needs different tuning due to the fact that
the glasses on them are different.
I've also seen that the camera modules are the same, but the OEM (two
products with different OEMs) wants the different tuning.
The worst part is that they share the same ChromeOS image.
As a result, we end up loading different tuning files based on the
product name + module ID at run-time....
>
> > Who is going to read them? In most of the cases, only the IPA should
> > read them, the examples are: {sensor}.json files for RaspberryPi, and
> > aiqb files for closed IPU3 IPA. However, Intel IPU3 introduced a
> > series of tuned xml files for configuring ISP, and in libcamera's
> > current design, they could be read by the pipeline handler.
> >
> > For the setting applied to ISP, Intel provides a tool to automatically
> > generate the settings, according to the stream's configuration.
> > https://github.com/intel/intel-ipu3-pipecfg. The file consists of a
> > limited set of IMGU configurations, which includes parameters needed
> > to set through the media controller and V4L2 API.
> >
> > These configurations are grouped by the requested streams, resolutions
> > and usecase (still capture or video). Examples can be found here:
> > https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/master/baseboard-poppy/media-libs/cros-camera-hal-configs-poppy/files/gcss/
> >
> > Currently libcamera has a procedure to produce the settings in the
> > pipeline handler, according to a tool provided by Intel. However, IPU3
> > was not that well designed to guarantee a fully predictable behavior.
> > We still see problems with the automatically generated settings. In
> > practice, there were efforts to generate a xml file of a sensor, and
> > then Intel or OEM validates and adjusts all parameters manually. The
> > xml file then becomes a database for the camera stack to query what's
> > the validated choices we have for the sensor? Following is how the xml
> > file are used in Intel camera stack on ChromeOS:
> > 1. The camera stack groups the streams into video and still
> > 2. Search validated settings for video/still to satisfy requested streams
> > 3. If found, configure both video and still pipes for processing
> > 4. If not, reduce a video or still stream and re-search, the reduced
> > stream will be SW scaled by the other streams
> > 5. The validated single stream request for video or still is
> > guaranteed, so at least one set of settings can be found
> >
> > It's tuned based on device because its content may be modified
> > manually due to tuning decisions. One interesting case is of Google
> > Pixelbook Go (IPU3): There is a limitation on IPU3: the DPC(defect
> > pixel correction) and BDS (Bayer down sampling) can not be enabled at
> > the same time. One of the sensors used on Pixelbook Go is IMX208, and
> > it seems to have lots of defect pixels at high gain level (>8x gain).
> > In result, the image without DPC is not acceptable to the image
> > quality requirement. However, when BDS cannot not be enabled, the
> > video pipe can only do YUV scale on one stream. As a result, some
> > configurations may have problems. For example
> >
> >     CIO2 outputs: 1920x1080
> >     CIO2 outputs ==> (Video IMGU YUV crop) ==> 1280x720 (stream A)
> >     CIO2 outputs ==> (Video IMGU YUV scale) ==> 640x360 (stream B)
> >     CIO2 outputs ==> (Still IMGU) ==> 1920x1080 (stream C)
> >
> > Because we cannot do BDS on the CIO2 output, and only one steam can
> > have yuv scale, the steam A can only be cropped to 720p, but the FOV
> > would be sacrificed, which is not accepted. Another choice is that
> > only one stream is enabled for the video pipe, and do software scale
> > for stream B.
> >
> >     CIO2 outputs ==> (Video IMGU YUV scale) ==> 1280x720 (stream A)
> >     1280x720 (stream A) ==> (SW scale) ==> 640x360 (stream B)
> >
> > This would trade performance for better image quality. After a series
> > of discussions, the latter solution is chosen. In result, Intel
> > removes all possible settings in the xml which would cause the problem
> > manually, and the camera stack is forced to do software scale for the
> > extra stream B (because no validated choice to satisfy A and B). Note
> > that it's also possible that an OEM chooses a smaller FOV for
> > performance. If so, the xml should be modified accordingly. All
> > devices share the same camera stack implementation, but only xml files
> > are different.
> >
> > Some other tuning considerations which may affect the setting to ISP:
> > 1. Do downscales on bayer stage (for performance/bandwidth) or on YUV
> > stage (for quality/usage)
> > 2. Limitation on certain blocks, For example, the IPU6 has a TNR block
> > which can process at most 1080p. In this case, if image quality
> > requirements think TNR is critical, we may choose to crop/scale before
> > the block.
>
> I'd like to remind a core design decision for libcamera. Configuration
> of ISP processing steps, such as DPC or TNR (Temporal Noise Reduction),
> is the responsibility of the IPA. Configuration of the pipeline,
> including deciding what combination of streams can be produced, what
> resolutions can be obtained from them, and how to configure the various
> scalers (binning/skipping on the sensor, bayer-domain scaler, YUV-domain
> scaler, ...) and crop rectangles in the pipeline, is the responsibility
> of the pipeline handler.
>
> The IPA module and the pipeline handler are meant to be designed
> together, and can (and should) communicate information to each other as
> needed.
>
> To reuse the examples you've provided, whether a platform needs to
> enable DPC or not is something that I would envision being supplied
> through the IPA tuning file. DPC operation could be decided by the IPA
> module and communicated to the pipeline handler, which could then take
> it into account to restricts options for the configuration of the
> pipeline and the supported streams and resolutions. The same could apply
> to TNR.
I suppose in this case, the pipeline configuration files would need to
be tuned for all possible combinations as stream combinations * DPC *
TNR * other possible flags?
It seems that there is an implicit assumption I made here, but I
didn't state it well:
I took the pipeline configuration files as part of the tuning files.
They are tuned and present together for a specific product.
The difference is that one is read by the pipeline handler, and the
other by IPA.
Usually, they won't be tuned by the vendor for all possible cases, but
sadly only the used cases for a specific product.
>
> This being said, I don't disagree that OEM-specific overrides for the
> pipeline configuration could be useful, for instance to take power
> budgets or other device-specific constraints into account. We should
> however not take this as an excuse to limit the capabilities of the
> libcamera native API. It can be useful for users to pass power
> consumption hints to libcamera for instance. Of course, we need to find
> a proper balance here to avoid feature bloat and deadly complexity.
>
I think it might be similar to the Noise Reduction mode(, Edge and
HotPixel) in Android API.
Their "Fast" mode is basically a hint to HAL, and HAL should respect
it as much as possible, although there is no guarantee that the image
has significant difference from the "OFF" mode. On the other hand,
"HIGH_QUALITY" mode is a hard requirement, and there is an ITS test to
check if the SNR is really better.
Whether to support HIGH_QUALITY is up to the pipeline handler, if so,
the tuned details should contain related information.
However, how this kind of hint affects the content of its pipeline
configuration file is hard to predict before we jump into development
of a platform.

> > Consideration For General Distribution
> > Although the ISP setting files are very convenient for devices with an
> > image provider, like ChromeBook or Android phone, it's not very
> > compatible with the distribution to the general linux, since there is
> > unlikely a ISP setting file defined for a random sensor. Thus, the
> > pipeline handler should always have a default procedure to generate
> > one, as the IPU3 pipeline handler has done.
>
> For the sake of completeness, it has been mentioned before the libcamera
> and/or Linux distributions could ship such configuration files for all
> "generic" devices, possibly using DMI for device identification. I don't
> think that would really work it, it sounds to more more like a way to
> shove the problem under the carpet and let someone else deal with it :-)
> If someone still thinks this would be a workable solution, we could
> discuss it.
Agreed.
>
> > Here are some ideas for the above problems:
> > 1. An optional device yaml to indicate device specific information, it
> > can be installed by the device image provider like ChromeOS or Android
> > vendor or by third parties. My imagination for the contents are:
> >
> > cameras
> >   0:
> >     type: single
> >     device_id: "\\_SB_.PCI0.I2C2.CAM0":
> >     orientation: front
> >     caliberation_type: nvmem
> >     nvmem_path: "/sys/bus/i2c/devices/i2c-INT3499:00/eeprom"
> >     isp_setting_file: data/setting/ipu3/ov5688.yaml # Installed by the
> > image provider or with libcamera
>
> Assuming this would be a YAML equivalent to the XML file you mentioned
> above, I don't think we should consider it an ISP setting file, but a
> pipeline configuration file. In this context, pipeline configuration
> means the logical equivalent of setting the V4L2 selection rectangles
> and formats through the pipeline (I'm saying logical equivalent because
> those parameters may not be configured through V4L2 selection rectangles
> and formats or all devices, some of them could be set through ISP
> configuration buffers for instance).
>
> >     ipa_tuning_file: data/tuning/ipu3/ov5688.aiqb
> >   1:
> >     type: single
> >     device_id: "\\_SB_.PCI0.I2C2.CAM0"
> >     orientation: front
>
Yes, it's stupid copy paste error...
> Did you mean to set the orientation to back, and have a different device
> ID than for camera 0 ?
>
> >     caliberation_type: nvmem
> >     nvmem_path: "/sys/bus/i2c/devices/i2c-INT3479:00/eeprom"
> >     isp_setting_file: data/setting/ipu3/imx208.yaml # Installed by the
> > image provider or libcamera
> >     ipa_tuning_file: data/tuning/ipu3/imx208.aiqb
> >
> > # --------- Imaginary dual camera case section ----------
> >   2:
> >     type: dual
> >     device_id: "":
> >     orientation: front
> >     isp_conf_file: data/setting/ipu3/dual.yml # Installed by the image
> > provider or libcamera
>
> Should this be isp_setting_file like for cameras 0 and 1, or is it
> something different ?
I suppose it's something different. After all, it should handle two
sensors simultaneously.
>
> >     ipa_tuning_file: data/tuning/ipu3/dual.aiqb
> >     physic_cameras: "0", "1"
>
> Logical cameras is a very interesting topic we'll have to tackle too :-)
>
> > If a device conf is presented, pipeline handler could use it directly,
> > otherwise, it could generate default values according to the number of
> > sensors probed, as libcamera already has done.
> >
> > 4. For ISP setting and tuning files, the image provider could install
> > them or, like RaspberryPi json files, installed with libcamera.
>
> Raspberry Pi tuning files are stored in the libcamera source tree for
> convenience, but as the number of supported platforms will grow, I would
> expect tuning files (for all platforms) to be stored somewhere else. We
> could have a separate git tree with a tuning file database for instance.
> I would expect OEMs to install tuning files on the devices they ship,
> but I'd like to come up with a useful enough tuning file database
> service to convince them to also submit the tuning files for their
> devices to the public database.
>
> This assumes, of course, that tuning data wouldn't be unit-specific.
> When performing per-unit tuning, we could possibly split the tuning data
> in per-unit and per-model data and still provide the latter in a public
> tuning database, but I'm not sure how practical such a scheme would be.
>
> Let's also note that we currently support overridding tuning data files
> using an environment variable (LIBCAMERA_RPI_TUNING_FILE), and I'd like
> to generalize that mechanism to support both a more generic envvar (with
> the ability to select a per-camera tuning file), and a C++ API to let
> applications specify a tuning file (or even possibly tuning data).
> There's lots of things that could be done here, we also have to find a
> good balance between a powerful API and an overcomplicated system
> (especially given that I would expect tuning file overrides to be mostly
> used during development).
>
> > I suppose the logic pipeline handler using the tuned files could be:
> >   1. If a device conf is specifying a file installed by the image
> > provider, use it.
> >   2. Otherwise, if there is a tuned file for the sensor model
> > installed by libcamera, use it.
> >   3. Otherwise, try to generate default settings
>
> Are we still talking about tuning data or, or about pipeline
> configuration data ? Reading the rest of your text below I would say
> it's the pipeline configuration data, but I don't think item 2 in the
> list above would apply to that.
As above, because I took the pipeline configuration file as part of
the tuning file (it's indeed tuned for IPU3), it means that.
Item 2 is a backup solution for 3, if there is a problem with auto
generation, a configuration file provided by the community may be a
supplement, and installed by default.
Hmmm, but after a re-think, it seems to be not necessary.
>
> > There could be a setting manager class for above logics, and for the
> > pipeline handler, it only queries whether the requested streams have a
> > validated setting without knowing where the information comes from.
> > Because the content structure for ISP settings/Tuning files are
> > specific to a certain platform, I suppose its interface should be
> > defined independently. For IPU3, the following is the structure I
> > would image:
> >
> > sensor_modes:
> >   4224x3136:
> >       name: "4224x3136"
> >       binner: [1, 1]
> >       pixel_array: [4224, 3136]
> >       output: [4224, 3136]
> >   2112x1568:
> >       name: "2112x1568"
> >       binner: [2, 2]
> >       pixel_array: [4224, 3136]
> >       output: [2112, 1568]
> >
> > video_imgu_settings:
> >   - sensor_mode: 4224x3136
> >     input: [4224, 3136]
> >     viewfinder: [1920, 1080]
> >     main: [4096, 3072]
> >     if: [4224, 3136]
> >     bds: [4224, 3136]
> >     filter: [4, 4]
> >     env: [60, 28]
> >     gdb: [4096, 3072]
> >
> >   - sensor_mode: 4224x3136
> >     input: [4224, 3136]
> >     viewfinder: [1280, 960]
> >     main: null
> >     if: [4182, 3116]
> >     bds: [1632, 1216]
> >     filter: [4, 4]
> >     env: [12, 4]
> >     gdb: [1600, 1200]
> >
> > still_imgu_settings:
> >   - sensor_mode: 4224x3136
> >     input: [4224, 3136]
> >     viewfinder: [1920, 1080]
> >     main: null
> >     if: [4216, 2386]
> >     bds: [4216, 2386]
> >     filter: [4, 4]
> >     env: [40, 28]
> >     gdb: [4128, 2322]
> >
> > As the Intel IPU3 implementation, it provides validated settings for
> > both video and still IMGU, and the pipeline handler tries to query
> > them. If failed, reduce the stream number and re-query. The single
> > stream settings are always presented and validated. The pipeline
> > handler reports the adjusted configuration back to the user, so the
> > user can choose whether or how to do software scales for the reduced
> > stream.
>
> Note that, beside the validation and adjustment mechanism, the pipeline
> handler must also expose static information about the supported streams
> and their resolutions. I don't think the above precludes that.
>
> > The context may be different for each platform, so I suppose it's hard
> > to avoid discussing them case by case.
> >
> > I'd love to hear your thoughts.
>
> One point we haven't discussed is how a platform that doesn't have an
> OEM configuration override for the pipeline configuration would be able
> to compute the pipeline configuration automatically. For some platforms
> that's fairly easy (for instance with Raspberry Pi, the constraints of
> the ISP are fairly well understood, especially by Raspberry Pi, and
> they're great contributors to libcamera :-)). For other platforms,
> that's more difficult (I'm thinking about the IPU3 in particular here,
> the hardware is old, and it's hard to get fresh and accurate
> information, Intel has moved on). While I'm not opposed to an OEM
> override mechanism for the pipeline configuration, I don't want this
> mechanism to be used by vendors (OEMs or SoC vendors) to provide a
> half-baked (or even completed unbaked) implementation of automatic
> pipeline configuration in the pipeline handler. That would leave users
> stuck with only what OEMs provide, for specific platforms, and would
> defeat the point of libcamera which is to make cameras usable for
> everybody.
>
> --
> Regards,
>
> Laurent Pinchart