[libcamera-devel] Colour spaces

[David Plowman]

Hi Laurent

Thanks for your reply. I'll try and answer some of your questions, and
also try to give an idea of what I was expecting at the bottom.

On Tue, 12 Jan 2021 at 02:01, Laurent Pinchart
<laurent.pinchart at ideasonboard.com> wrote:
>
> Hi David,
>
> On Thu, Jan 07, 2021 at 01:46:04PM +0000, David Plowman wrote:
> > Hi everyone
> >
> > I've just found myself wondering how I would signal to libcamera that
> > I want a particular YUV colour space, such as JFIF for jpeg, BT601 for
> > SD video, and so on. I haven't spotted a way to do this... have I
> > perhaps missed something? (I notice a PixelFormatInfo class with a
> > ColourEncoding enum, but it seems limited.)
>
> You haven't missed anything, it's not there.
>
> > If there really isn't a way to do this, then I suppose all the usual
> > questions apply. Should it be a control? Or should it go in the stream
> > configuration? And what values - start with the V4L2 ones?
> >
> > Anyone have any thoughts on this?
>
> Quite a few (sorry :-)). Please keep in mind that my knowledge is
> limited regarding this topic though, so feel free to correct me when I'm
> wrong.
>
> First of all, colour space is an umbrella term that is often abused to
> mean different things, so I'd like to know which parts you're interested
> in (it may well be all of them).
>
> I'm looking at BT.709, which nicely summarizes the colour-related
> information in sections 1 and 3 (section 2 is not related to colours,
> section 4 contains related information in the quantization levels, but
> those are also present in section 3 as far as I can tell):
>
> 1. Opto-electronic conversion
>
> This specifies the opto-electronic transfer characteristics and the
> chromaticity coordinates (in CIE xyY space) of R, G, B and the D65
> reference white.
>
> 3. Signal format
>
> This specifies the transfer function (expressed as a gamma value), the
> colour encoding and the quantization.

There's also the question of full/limited range YUV - I don't think
that's covered just by conversion matrices and/or transfer functions.

>
>
> To obtain BT.709 from a given camera sensor, we need to take the
> sensor's colour characteristics into account to calculate the correct
> colour transformation matrix (and the tone mapping curve) to obtain the
> BT.709 primaries. This could be done inside libcamera with an API to
> specify the desired "colour space", but I don't think that's the best
> option. Exposing the colour characteristics of the sensor is needed for
> correct processing of RAW data, and with manual control of the colour
> transformation matrix and tone mapping curve, we could then achieve any
> output colour space.

I'm not sure about this. Speaking for the Pi, it doesn't have those
kinds of interfaces, nor does v4l2 of course (unless they are custom
ones). The Pi firmware will sort out the colour space for itself once
you've said what you want, we wouldn't want to let outside code
override that. I would expect this to be a fairly common approach
(certainly v4l2 encourages it).

>
> We already expose the colour transformation matrix, so we're missing the

We expose the camera RGB -> sRGB matrix. We don't expose any RGB
to/from YCbCr matrices.

> tone mapping curve as a control, and the sensor colour characteristics

Our camera tuning has a gamma curve which is filled in by the tuning
tool. It's worth noting that we never use the "official" transfer
functions because they give you terrible (pale and washed out) images.
So we certainly wouldn't want anyone else trying to set our gamma
curves.

> as a property. The latter are exposed in Android as a combination of a
> reference illuminant and a transform matrix. It's actually multiple
> matrices, one to map CIE XYZ to the colour space of the reference sensor
> (a.k.a. golden module) and one to map the reference sensor colour space
> to the colour space of the device's sensor - I assume the latter will be
> an identity matrix when devices don't undergo per-device calibration.
> There's also a forward matrix that maps white balanced colours (using
> the reference illuminant) from the reference sensor colour space to the
> CIE XYZ D50 white point. I'm sure I'm missing a few subtle points.

This sounds like it's taken from the DNG spec. I think it's a fine API
for a software raw converter; I'm less convinced about hardware. Just
as an aside, our AWB calibration has some sensitivity parameters that
account for module to module colour variation.

>
> This approach makes sense as it's very flexible, but it's also hard to
> use, and we really need a helper class to deal with this and compute the
> colour correction matrix for the sensor to produce standard preset
> colourspaces.

I agree, it's very intimidating to use, especially for the application
that "just wants HD video" (for example). Helpers mitigate it to some
extent, but I can't escape the feeling that it should be more "under
the hood".

>
> I suppose we can deal with quantization in either the colour
> transformation matrix or the RGB to YUV matrix, and I'm not sure what's
> best, or even what hardware typically support. Does the colour
> transformation typically have an offset vector, or is it only a
> multiplication matrix ?

That's all very interesting, especially about Android.  I guess I'm a
bit surprised by that - why would they expect hardware in general to
have all those kinds of fully general interfaces?

My own initial thoughts were along much simpler more v4l2-friendly
lines, I'll list them below, but we can pick holes in them!

1. I imagined a libcamera::ColorSpace enum. It's a completely plain
enum. We'd have one in the stream configuration and application code
could overwrite any defaults there. You might start by supporting
v4l2's list of colour spaces.

2. This enum would also go in the V4L2PixelFormat, thereby getting
passed to the V4L2VideoDevice::setFormat method.

3. Down there we'd have a simple translation table which populates the
colour space related fields in the v4l2_pix_format from the
libcamera::ColorSpace.

Many questions remain, of course...  :)

Best regards
David


>
> --
> Regards,
>
> Laurent Pinchart