[libcamera-devel] Colour spaces

[Laurent Pinchart]

Hi David,

Having given this some more thoughts, I think that your proposal to
model colour spaces in a way similar to V4L2 is probably the best
option. I can imagine that some applications may want to tweak the RGB
to YUV matrix (and possibly other parameters related to colour spaces),
but when we'll have to support that (if ever) I suppose we could just
define that any manually specified RGB to YUV matrix will override the
other colour space controls.

We still need to consider the issue of per-stream vs. global colour
spaces. I would also like to give some thoughts about how to document
this in an abstract pipeline model. For instance, do we want to mandate
a particular colour space (I suppose sRGB ?) after the colour correction
matrix ? It's the interactions between colour spaces and the tunable
parameters (gamma and tone mapping curves come to mind) that scare me.

Please see below for some additional comments, I'll try to provide more
after thinking about this topic further.

On Wed, Jan 13, 2021 at 09:59:31AM +0000, David Plowman wrote:
> Hi Laurent
> 
> Thanks for your reply. I'll try and answer some of your questions, and
> also try to give an idea of what I was expecting at the bottom.
> 
> On Tue, 12 Jan 2021 at 02:01, Laurent Pinchart wrote:
> > On Thu, Jan 07, 2021 at 01:46:04PM +0000, David Plowman wrote:
> > > Hi everyone
> > >
> > > I've just found myself wondering how I would signal to libcamera that
> > > I want a particular YUV colour space, such as JFIF for jpeg, BT601 for
> > > SD video, and so on. I haven't spotted a way to do this... have I
> > > perhaps missed something? (I notice a PixelFormatInfo class with a
> > > ColourEncoding enum, but it seems limited.)
> >
> > You haven't missed anything, it's not there.
> >
> > > If there really isn't a way to do this, then I suppose all the usual
> > > questions apply. Should it be a control? Or should it go in the stream
> > > configuration? And what values - start with the V4L2 ones?
> > >
> > > Anyone have any thoughts on this?
> >
> > Quite a few (sorry :-)). Please keep in mind that my knowledge is
> > limited regarding this topic though, so feel free to correct me when I'm
> > wrong.
> >
> > First of all, colour space is an umbrella term that is often abused to
> > mean different things, so I'd like to know which parts you're interested
> > in (it may well be all of them).
> >
> > I'm looking at BT.709, which nicely summarizes the colour-related
> > information in sections 1 and 3 (section 2 is not related to colours,
> > section 4 contains related information in the quantization levels, but
> > those are also present in section 3 as far as I can tell):
> >
> > 1. Opto-electronic conversion
> >
> > This specifies the opto-electronic transfer characteristics and the
> > chromaticity coordinates (in CIE xyY space) of R, G, B and the D65
> > reference white.
> >
> > 3. Signal format
> >
> > This specifies the transfer function (expressed as a gamma value), the
> > colour encoding and the quantization.
> 
> There's also the question of full/limited range YUV - I don't think
> that's covered just by conversion matrices and/or transfer functions.
> 
> > To obtain BT.709 from a given camera sensor, we need to take the
> > sensor's colour characteristics into account to calculate the correct
> > colour transformation matrix (and the tone mapping curve) to obtain the
> > BT.709 primaries. This could be done inside libcamera with an API to
> > specify the desired "colour space", but I don't think that's the best
> > option. Exposing the colour characteristics of the sensor is needed for
> > correct processing of RAW data, and with manual control of the colour
> > transformation matrix and tone mapping curve, we could then achieve any
> > output colour space.
> 
> I'm not sure about this. Speaking for the Pi, it doesn't have those
> kinds of interfaces, nor does v4l2 of course (unless they are custom
> ones). The Pi firmware will sort out the colour space for itself once
> you've said what you want, we wouldn't want to let outside code
> override that. I would expect this to be a fairly common approach
> (certainly v4l2 encourages it).
> 
> > We already expose the colour transformation matrix, so we're missing the
> 
> We expose the camera RGB -> sRGB matrix. We don't expose any RGB
> to/from YCbCr matrices.

Yes, not disagreement here.

> > tone mapping curve as a control, and the sensor colour characteristics
> 
> Our camera tuning has a gamma curve which is filled in by the tuning
> tool. It's worth noting that we never use the "official" transfer
> functions because they give you terrible (pale and washed out) images.
> So we certainly wouldn't want anyone else trying to set our gamma
> curves.

This we'll have to discuss in more details. I expect that some
applications would like to set custom tone mapping curves, and I need to
understand better how that would interact with tuning (and why the
standard curves don't produce good results).

> > as a property. The latter are exposed in Android as a combination of a
> > reference illuminant and a transform matrix. It's actually multiple
> > matrices, one to map CIE XYZ to the colour space of the reference sensor
> > (a.k.a. golden module) and one to map the reference sensor colour space
> > to the colour space of the device's sensor - I assume the latter will be
> > an identity matrix when devices don't undergo per-device calibration.
> > There's also a forward matrix that maps white balanced colours (using
> > the reference illuminant) from the reference sensor colour space to the
> > CIE XYZ D50 white point. I'm sure I'm missing a few subtle points.
> 
> This sounds like it's taken from the DNG spec. I think it's a fine API
> for a software raw converter; I'm less convinced about hardware. Just
> as an aside, our AWB calibration has some sensitivity parameters that
> account for module to module colour variation.
> 
> > This approach makes sense as it's very flexible, but it's also hard to
> > use, and we really need a helper class to deal with this and compute the
> > colour correction matrix for the sensor to produce standard preset
> > colourspaces.
> 
> I agree, it's very intimidating to use, especially for the application
> that "just wants HD video" (for example). Helpers mitigate it to some
> extent, but I can't escape the feeling that it should be more "under
> the hood".
> 
> > I suppose we can deal with quantization in either the colour
> > transformation matrix or the RGB to YUV matrix, and I'm not sure what's
> > best, or even what hardware typically support. Does the colour
> > transformation typically have an offset vector, or is it only a
> > multiplication matrix ?
> 
> That's all very interesting, especially about Android.  I guess I'm a
> bit surprised by that - why would they expect hardware in general to
> have all those kinds of fully general interfaces?
> 
> My own initial thoughts were along much simpler more v4l2-friendly
> lines, I'll list them below, but we can pick holes in them!
> 
> 1. I imagined a libcamera::ColorSpace enum. It's a completely plain
> enum. We'd have one in the stream configuration and application code
> could overwrite any defaults there. You might start by supporting
> v4l2's list of colour spaces.
> 
> 2. This enum would also go in the V4L2PixelFormat, thereby getting
> passed to the V4L2VideoDevice::setFormat method.

This part I don't like, I don't expect named colour spaces to be handled
by the ISP driver in general. You may have the flexibility to do so on
Rapsberry Pi, but that would be an exception, and even there, I'd prefer
exposing lower-level parameters and having the IPA handling the
calculations.

> 3. Down there we'd have a simple translation table which populates the
> colour space related fields in the v4l2_pix_format from the
> libcamera::ColorSpace.
> 
> Many questions remain, of course...  :)

-- 
Regards,

Laurent Pinchart