<div dir="ltr"><div dir="ltr">Hi Laurent,<div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 4 Feb 2021 at 21:57, Laurent Pinchart <<a href="mailto:laurent.pinchart@ideasonboard.com">laurent.pinchart@ideasonboard.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Naush,<br>
<br>
On Wed, Jan 27, 2021 at 08:14:29AM +0000, Naushir Patuck wrote:<br>
> On Tue, 26 Jan 2021 at 22:53, Laurent Pinchart wrote:<br>
> > On Tue, Jan 26, 2021 at 04:24:08PM +0000, Naushir Patuck wrote:<br>
> > > In preparation for fast colour denoise, set the low resolution ISP<br>
> > > output stream (Output1) to a 1/4 resolution of the application requested<br>
> > > stream (Output0). This only happens if the application has not requested<br>
> > > an additional YUV or RGB stream.<br>
> > ><br>
> > > We also constrain this 1/4 resolution to at most 1200 pixels in the<br>
> > > largest dimension to avoid being too taxing on memory usage and system<br>
> > > bandwidth.<br>
> > ><br>
> > > Also switch the default StreamRole::VideoRecording to YUV420 to allow<br>
> > > fast colour denoise to run.<br>
> ><br>
> > Could you explain why all this is needed ? What happens if an<br>
> > application requests the second output with a higher resolution that<br>
> > this, will fast colour denoise still work correctly ?<br>
> <br>
> Fast colour denoise essentially works by doing some analysis at a 1:4 or<br>
> 1:1 resolution image when compared with the output image resolution. It<br>
> then applies correction on the output image directly based on this<br>
> analysis. If the application were to require a second output stream that<br>
> is not 1:4 or 1:1 in resolution ratio, the colour denoise will silently be<br>
> bypassed.<br>
<br>
That's useful information. Maybe this could be captured in a comment in<br>
the code ?<br></blockquote><div><br></div><div>Yes, that's a good idea.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
I also wonder if we shouldn't consider this in the metadata we return.<br>
When the CDN is silently bypassed, wouldn't it be better if the metadata<br>
reflected that ?<br></blockquote><div><br></div><div>Yes, we could consider that. The options I would consider are:</div><div>1) Only return the metadata when/if the control is set by the application indicating what has been applied (i.e. either set or not based on the requested format).</div><div>2) Return the metadata on each frame indicating the CDN status.</div><div><br></div><div>My preference would be for 1, it matches how the metadata returns when we set other ISP related params. but I am not entirely against doing 2. What are your thoughts?</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> There is another constraint that this analysis image must be in<br>
> YUV420 format. Unfortunately, this is a limitation with our<br>
> implementation that we currently have.<br>
<br>
Only the analysis image, or also the main output image ? This would also<br>
be useful to capture in a comment, so that it doesn't get ignored later<br>
when the code is reworked.<br></blockquote><div><br></div><div>Both images have to be YUV420 format.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> > > Signed-off-by: Naushir Patuck <<a href="mailto:naush@raspberrypi.com" target="_blank">naush@raspberrypi.com</a>><br>
> > > Reviewed-by: David Plowman <<a href="mailto:david.plowman@raspberrypi.com" target="_blank">david.plowman@raspberrypi.com</a>><br>
> > > Reviewed-by: Kieran Bingham <<a href="mailto:kieran.bingham@ideasonboard.com" target="_blank">kieran.bingham@ideasonboard.com</a>><br>
> > > ---<br>
> > > .../pipeline/raspberrypi/raspberrypi.cpp | 35 ++++++++++++++++++-<br>
> > > 1 file changed, 34 insertions(+), 1 deletion(-)<br>
> > ><br>
> > > diff --git a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp<br>
> > > index d9895c779725..fe4c75f09925 100644<br>
> > > --- a/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp<br>
> > > +++ b/src/libcamera/pipeline/raspberrypi/raspberrypi.cpp<br>
> > > @@ -496,7 +496,7 @@ CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera *camera,<br>
> > ><br>
> > > case StreamRole::VideoRecording:<br>
> > > fmts = data->isp_[Isp::Output0].dev()->formats();<br>
> > > - pixelFormat = formats::NV12;<br>
<br>
Here for instance, you could write<br>
<br>
/*<br>
* The colour denoise algorithm require the analysis<br>
* image, produced by the second ISP output, to be in<br>
* YUV420 format. Select this format as the default, to<br>
* maximize chances that it will be picked by<br>
* applications and enable usage of the colour denoise<br>
* algorithm.<br>
*/<br>
<br>
Or something more accurate, as I'm not sure to have picked all the<br>
important information correctly :-)<br></blockquote><div><br></div><div>Sounds correct to me :) I will add a comment with that wording.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> > > + pixelFormat = formats::YUV420;<br>
> > > size = { 1920, 1080 };<br>
> > > bufferCount = 4;<br>
> > > outCount++;<br>
> > > @@ -608,6 +608,7 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)<br>
> > > * StreamConfiguration appropriately.<br>
> > > */<br>
> > > V4L2DeviceFormat format;<br>
> > > + bool output1Set = false;<br>
> > > for (unsigned i = 0; i < config->size(); i++) {<br>
> > > StreamConfiguration &cfg = config->at(i);<br>
> > ><br>
> > > @@ -632,6 +633,9 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)<br>
> > > format.size = cfg.size;<br>
> > > format.fourcc = fourcc;<br>
> > ><br>
> > > + LOG(RPI, Info) << "Setting " << stream->name() << " to a resolution of "<br>
> > > + << format.toString();<br>
> > > +<br>
> > > ret = stream->dev()->setFormat(&format);<br>
> > > if (ret)<br>
> > > return -EINVAL;<br>
> > > @@ -645,6 +649,35 @@ int PipelineHandlerRPi::configure(Camera *camera, CameraConfiguration *config)<br>
> > ><br>
> > > cfg.setStream(stream);<br>
> > > stream->setExternal(true);<br>
> > > +<br>
> > > + if (i != maxIndex)<br>
> > > + output1Set = true;<br>
> > > + }<br>
> > > +<br>
> > > + /*<br>
> > > + * If ISP::Output1 stream has not been requested by the application, we<br>
> > > + * set it up for internal use now. This second stream will be used for<br>
> > > + * fast colour denoise, and must be a quarter resolution of the ISP::Output0<br>
> > > + * stream. However, also limit the maximum size to 1200 pixels in the<br>
> > > + * larger dimension, just to avoid being wasteful with buffer allocations<br>
> > > + * and memory bandwidth.<br>
> > > + */<br>
> > > + if (!output1Set) {<br>
> > > + V4L2DeviceFormat output1Format = format;<br>
<br>
If only the analysis image has to be in YUV420 format, should YUV420 be<br>
selected here in case output 0 uses a different format ?<br></blockquote><div><br></div><div>While this is true, I don't think it makes any practical difference. With the existing code, if the output 0 format is YUV420, output 1 will be as well, and we get colour denoise working. If output 1 format is not YUV420, colour denoise will not work, so the format of output 1 is somewhat irrelevant. However, as per the comment in a previous email, what I really want to happen here is to disable the Output 1 stream if Output 0 is not in YUV420 format. I've added a \todo for that in the above comment.</div><div><br></div><div>Regards,</div><div>Naush</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> > > + constexpr unsigned int maxDimensions = 1200;<br>
> > > + const Size limit = Size(maxDimensions, maxDimensions).boundedToAspectRatio(format.size);<br>
> > > +<br>
> > > + output1Format.size = (format.size / 2).boundedTo(limit).alignedDownTo(2, 2);<br>
> > > +<br>
> > > + LOG(RPI, Info) << "Setting ISP Output1 (internal) to a resolution of "<br>
> > > + << output1Format.toString();<br>
> > > +<br>
> > > + ret = data->isp_[Isp::Output1].dev()->setFormat(&output1Format);<br>
> > > + if (ret) {<br>
> > > + LOG(RPI, Error) << "Failed to set format on ISP Output1: "<br>
> > > + << ret;<br>
> > > + return -EINVAL;<br>
> > > + }<br>
> > > }<br>
> > ><br>
> > > /* ISP statistics output format. */<br>
<br>
-- <br>
Regards,<br>
<br>
Laurent Pinchart<br>
</blockquote></div></div>