[libcamera-devel] [PATCH] ipa: raspberrypi: Add a "scientific" tuning for the IMX477

[Laurent Pinchart]

On Wed, Dec 21, 2022 at 01:00:17PM +0000, Kieran Bingham via libcamera-devel wrote:
> Quoting cpixip--- via libcamera-devel (2022-12-20 12:09:23)
> > Hello - just a short comment below ( I hope I am doing this right - not 
> > really familiar with this email/list-based system)
> 
> Hi - yes you're doing fine ;-)
> 
> > Am 19.12.2022 um 23:53 schrieb Laurent Pinchart:
> > > On Fri, Dec 16, 2022 at 03:09:55PM +0000, Naushir Patuck wrote:
> > >> On Fri, 16 Dec 2022 at 14:25, Kieran Bingham wrote:
> > >>> Quoting Naushir Patuck via libcamera-devel (2022-12-16 13:59:03)
> > >>>> Add a tuning file for the IMX477 more suited to scientific applications.
> > >>>> The key differences from the original tuning file are:
> > >>>>
> > >>>> - Disable ALSC block completely
> > >>>> - Pure rec709 gamma curve, and no contrast enhance
> > >>>> - New CT curve and CCMs based on the illumination spectrum of a black body
> > >>>> radiator up to about 3600 K and the CIE illuminant D for higher color
> > >>>> temperatures.
> > >>>>
> > >>>> Further details on the changes can be found at:
> > >>>> https://forums.raspberrypi.com/viewtopic.php?t=343449
> > >>>>
> > >>>> All credit for these changes go to Dr. Rolf Henkel.
> > >>> Some very interesting investigations in that thread.
> > >>>
> > >>> I'm curious if we'll end up with more configurations like this for each
> > >>> sensor!
> > >>>
> > >> I think that's entirely possible - and all part of the design with json config
> > >> files!
> > >>
> > >>> But I really like the graphs to present what the tuning files
> > >>> are doing comparitively. Much easier to grasp than a file of json
> > >>> data...
> > >>>
> > >>> No objections to this here but apart from admiring the graphs, I won't
> > >>> judge the data so:
> > >>>
> > >>> Acked-by: Kieran Bingham <kieran.bingham at ideasonboard.com>
> > >>>
> > >>>
> > >>> My only concern/comment would be how and where would we document the
> > >>> differences between these tuning files? Would users just be expected to
> > >>> try them to see ? Or would it be clear that a 'scientific' tuning file
> > >>> has a specific set of properties ?
> > >>
> > >> At the end of the day, I think this is a choice for informed users. We provide
> > >> a sensible default config for visually pleasing images, but if users have a
> > >> specific application that requires something custom, we leave it to them to
> > >> switch to the appropriate tuning file that they know works for them.
> > >
> > > Overall that makes sense to me. In this particular case, I think it
> > > would however make sense to give applications a way to control the tone
> > > mapping curve at runtime (including disabling contrast enhancement)
> > > instead of having different curves specified in tuning files.
> > 
> > Indeed, it would be great to be able to control certain parameters at 
> > runtime. There are a number of automatic algorithms working in libcamera 
> > which presently can only be deactivated in akward ways - mostly by 
> > editing the tuning file. That is, you need to stop and restart the 
> > camera for such changes to take place. It would be great (for people who 
> > know what they are doing) to be able to deactivate automatic processing 
> > while the camera is running, for example.
> 
> I would expect this is entirely possible by providing the controls to
> enable/disable automatic features, just like we do with manual
> gain/manual exposure.

It should be, but not all automatic features implemented in the
Raspberry Pi IPA have corresponding controls. Contrast enhancement is
one such feature that lacks a control.

Some features may be difficult to enable or disable at runtime, making
them a target for being controlled through the start() call only, but
even that would be an improvement compared to having to create custom
tuning files.

> > > This isn't something that needs to be implemented urgently, and I'm fine
> > > with this patch, but if we have tuning files that differed only by their
> > > tone mapping curve, I'd get more annoyed.
> > 
> > Note that the tone mapping curve is not the most important deviation of 
> > this new tuning file from the standard tuning file. It does help to get 
> > slightly less trendy colors.
> > 
> > However, more important is probably the ct-curve, combined with new CCM 
> > matrices. These new values vary much smoother than the original values 
> > over the range of CCTs.

Those would be more awkward to control through standard controls I
believe, as they are quite specific to the Raspberry Pi implementation
of the corresponding algorithms.

> > Another point: as soon as libcamera is starting to support sensors with 
> > interchangable lenses (and the Raspberry Pi HQ camera is one of these), 
> > the inclusion of the ALSC section in the tuning file makes it as well 
> > necessary to have a dedicated tuning file for every lens which you are 
> > planning to use. So maybe the addition of an info-section in the tuning 
> > file would be a thing to consider. Such an info-section could also 
> > include details on how the tuning file was created - something like that 
> > is currently missing. For example, the IMX477 tuning file for the HQ 
> > camera features ALSC-tables - but there is no information whatsoever 
> > which specific lens was used in the calibration.

That's a very good idea ! I've recorded it in
https://bugs.libcamera.org/show_bug.cgi?id=175.

> > LSC is in a sense rather independent from other sections in the 
> > processing pipeline. Maybe it even makes sense to split the monolithic 
> > json-file into independent parts?
> 
> Splitting could make sense so there's a default. But either way, we need
> to figure out how to make more 'module' specific tuning files.
> 
> Ultimately, I would expect a different lens to have impacts across
> the whole tuning data?

-- 
Regards,

Laurent Pinchart