<div dir="ltr"><div dir="ltr">Hi David,<div><br></div><div>Thank you for sketching this out. Colourspace is indeed something that folks using Raspberry Pi have been asking about.</div><div>I have a few questions, see inline:</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 28 May 2021 at 14:56, David Plowman <<a href="mailto:david.plowman@raspberrypi.com">david.plowman@raspberrypi.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi again<br>
<br>
In the interests of trying to move this discussion forward, I'd like<br>
to propose an idea, based on the assumption that my previous email was<br>
along the right lines (debatable, I know)!<br>
<br>
So what I'd like to do is create something like this:<br>
<br>
struct ColorSpace {<br>
enum class Standard {<br>
JFIF,<br>
SMPTE170M,<br>
REC709,<br>
REC2020,<br>
RAW,<br>
VIDEO<br>
};<br>
enum class Encoding {<br>
DEFAULT,<br>
REC601,<br>
REC709,<br>
REC2020<br>
};<br></blockquote><div><br></div><div>Are Standard and Encoding mutually exclusive? Perhaps I am missing the intention, but can we not combine them into one enum?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
enum class TransferFunction {<br>
DEFAULT,<br>
IDENTITY,<br>
SRGB,<br>
REC709<br>
};<br>
enum class Range {<br>
DEFAULT,<br>
FULL,<br>
LIMITED<br>
};<br></blockquote><div><br></div><div>Regarding the DEFAULT item for all the enums above, should we be explicit in what they give us? So possibly remove the DEFAULT item,</div><div>and the fist item is always the default, like TransferFunction::IDENTITY and Range::FULL for the above?</div><div><br></div><div>Thanks,</div><div>Naush</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
Standard standard;<br>
Encoding encoding;<br>
TransferFunction transferFunction;<br>
Range range;<br>
};<br>
<br>
This is fairly minimal, though it contains everything that the Pi<br>
(indeed most applications?) needs. Of course, adding more cases is<br>
trivial.<br>
<br>
It should feel familiar to the V4L2-cognoscenti, and as far as I can<br>
tell, the Android "dataspace" fits too. You choose your "standard" and<br>
can leave everything else at "default", or else you can override them<br>
as you wish.<br>
<br>
It's easy to use. We'd put it into the stream configuration, and it<br>
would get filled in according to the stream role - RAW for raw<br>
streams, JFIF for stills and VIDEO for video recording. The call to<br>
Configure would change VIDEO to one of SMPTE170M, REC709 or REC2020<br>
according to resolution.<br>
<br>
The pipeline handler of course gets the chance to change the colour<br>
spaces according to its own internal constraints, and ultimately it<br>
will get passed down to the V4L2 setFormat method. But note that<br>
libcamera application code wouldn't need to change at all, the<br>
(usually) "right" thing will simply happen by default.<br>
<br>
Any thoughts on the matter? Is this a way forward or have I misunderstood<br>
how it all works?<br>
<br>
Thanks!<br>
David<br>
<br>
On Sat, 22 May 2021 at 10:55, David Plowman<br>
<<a href="mailto:david.plowman@raspberrypi.com" target="_blank">david.plowman@raspberrypi.com</a>> wrote:<br>
><br>
> Hi everyone<br>
><br>
> I wanted to pick up this discussion on colour spaces again, as it<br>
> remains a bit of a gap in the API currently. I was wondering, has<br>
> anyone else had any time to consider this question?<br>
><br>
> I think a lot of this comes down to what Android requires, so I've<br>
> been trying to look that up. Unfortunately I haven't really found any<br>
> resources that show me, in practice, what camera applications do.<br>
> (Can anyone enlighten me, or point me at helpful references?)<br>
><br>
> Anyway, in the absence of clear examples, here are some of the<br>
> (possibly mistaken) impressions I've formed:<br>
><br>
> 1. There's something called an "android dataspace" which seems to<br>
> cover the notion of colourspaces. It even seems to feature in each<br>
> stream of the "camera stream configuration".<br>
><br>
> 2. The "dataspace" appears to be a portmanteau of various smaller<br>
> bitfields, representing standards, transfer functions, range<br>
> quantisation and so on, indeed not so very unlike V4L2.<br>
><br>
> 3. Previously there was some discussion of colour transforms, forward<br>
> matrices, reference illuminants and so on. I'm still of the view that<br>
> these are things you query but don't set. In fact, I suspect they're<br>
> provided mainly for writing DNG files - not unlike what we have in<br>
> dng_writer.cpp. (Is anyone actually familiar with this?)<br>
><br>
> Any thoughts on this subject? If folks think this is broadly correct then<br>
> I'm happy trying to proceed along those lines - and otherwise, I of<br>
> course look forward to being corrected!<br>
><br>
> Thanks<br>
> David<br>
><br>
><br>
> On Tue, 12 Jan 2021 at 02:01, Laurent Pinchart<br>
> <<a href="mailto:laurent.pinchart@ideasonboard.com" target="_blank">laurent.pinchart@ideasonboard.com</a>> wrote:<br>
> ><br>
> > Hi David,<br>
> ><br>
> > On Thu, Jan 07, 2021 at 01:46:04PM +0000, David Plowman wrote:<br>
> > > Hi everyone<br>
> > ><br>
> > > I've just found myself wondering how I would signal to libcamera that<br>
> > > I want a particular YUV colour space, such as JFIF for jpeg, BT601 for<br>
> > > SD video, and so on. I haven't spotted a way to do this... have I<br>
> > > perhaps missed something? (I notice a PixelFormatInfo class with a<br>
> > > ColourEncoding enum, but it seems limited.)<br>
> ><br>
> > You haven't missed anything, it's not there.<br>
> ><br>
> > > If there really isn't a way to do this, then I suppose all the usual<br>
> > > questions apply. Should it be a control? Or should it go in the stream<br>
> > > configuration? And what values - start with the V4L2 ones?<br>
> > ><br>
> > > Anyone have any thoughts on this?<br>
> ><br>
> > Quite a few (sorry :-)). Please keep in mind that my knowledge is<br>
> > limited regarding this topic though, so feel free to correct me when I'm<br>
> > wrong.<br>
> ><br>
> > First of all, colour space is an umbrella term that is often abused to<br>
> > mean different things, so I'd like to know which parts you're interested<br>
> > in (it may well be all of them).<br>
> ><br>
> > I'm looking at BT.709, which nicely summarizes the colour-related<br>
> > information in sections 1 and 3 (section 2 is not related to colours,<br>
> > section 4 contains related information in the quantization levels, but<br>
> > those are also present in section 3 as far as I can tell):<br>
> ><br>
> > 1. Opto-electronic conversion<br>
> ><br>
> > This specifies the opto-electronic transfer characteristics and the<br>
> > chromaticity coordinates (in CIE xyY space) of R, G, B and the D65<br>
> > reference white.<br>
> ><br>
> > 3. Signal format<br>
> ><br>
> > This specifies the transfer function (expressed as a gamma value), the<br>
> > colour encoding and the quantization.<br>
> ><br>
> ><br>
> > To obtain BT.709 from a given camera sensor, we need to take the<br>
> > sensor's colour characteristics into account to calculate the correct<br>
> > colour transformation matrix (and the tone mapping curve) to obtain the<br>
> > BT.709 primaries. This could be done inside libcamera with an API to<br>
> > specify the desired "colour space", but I don't think that's the best<br>
> > option. Exposing the colour characteristics of the sensor is needed for<br>
> > correct processing of RAW data, and with manual control of the colour<br>
> > transformation matrix and tone mapping curve, we could then achieve any<br>
> > output colour space.<br>
> ><br>
> > We already expose the colour transformation matrix, so we're missing the<br>
> > tone mapping curve as a control, and the sensor colour characteristics<br>
> > as a property. The latter are exposed in Android as a combination of a<br>
> > reference illuminant and a transform matrix. It's actually multiple<br>
> > matrices, one to map CIE XYZ to the colour space of the reference sensor<br>
> > (a.k.a. golden module) and one to map the reference sensor colour space<br>
> > to the colour space of the device's sensor - I assume the latter will be<br>
> > an identity matrix when devices don't undergo per-device calibration.<br>
> > There's also a forward matrix that maps white balanced colours (using<br>
> > the reference illuminant) from the reference sensor colour space to the<br>
> > CIE XYZ D50 white point. I'm sure I'm missing a few subtle points.<br>
> ><br>
> > This approach makes sense as it's very flexible, but it's also hard to<br>
> > use, and we really need a helper class to deal with this and compute the<br>
> > colour correction matrix for the sensor to produce standard preset<br>
> > colourspaces.<br>
> ><br>
> > I suppose we can deal with quantization in either the colour<br>
> > transformation matrix or the RGB to YUV matrix, and I'm not sure what's<br>
> > best, or even what hardware typically support. Does the colour<br>
> > transformation typically have an offset vector, or is it only a<br>
> > multiplication matrix ?<br>
> ><br>
> > --<br>
> > Regards,<br>
> ><br>
> > Laurent Pinchart<br>
</blockquote></div></div>