[RFQ] Multiple ISP outputs and scaler crop

[Naushir Patuck]

On Thu, 2 May 2024 at 20:28, Kieran Bingham
<kieran.bingham at ideasonboard.com> wrote:
>
> Quoting Naushir Patuck (2024-05-01 10:26:14)
> > Hi all,
> >
> > We have come across some new use cases where users will need different
> > scaler crops applied to each of the 2 ISP outputs configured by the
> > application.  This is not possible on the VC4 hardware, but we can
> > apply different crop configurations on Pi5/PISP.
> >
> > Right now, libcamera has a single ScalerCrop control that we apply to
> > both outputs.  This means that the input FoV is identical for both
> > outputs, and can distort the pixel aspect ratio on the "low res"
> > output if the output aspect ratio on both ISP outputs are not the
> > same.
> >
> > I would like to propose a couple of solutions to this:
> >
> > 1) Change the ScalerCrop control to be a span, so that users can
> > specify crop for multiple output branches through the single control.
> > This will incur an API breakage though, so may not be desirable.
> >
> > 2) Add a new control called ScalerCrops (with an s), that behaves
> > identically to ScalerCrop, but works with a span of rectangles.  The
> > value is handled in exactly the same way as  ScalerCrop, but allows
> > the user to set muitple crops, one per output.  Of course the wording
> > of this control needs to be explicit about the behavior, e.g. index 0
> > corresponds to configured stream 0, and what if we only want to change
> > one stream's scaler crop, etc.
> >
> > Thoughts?
>
> Shouldn't this be a control on a Stream?
> I think the difficulty is we don't yet have 'stream' controls. What
> other controls are there that can apply distinctly on different output
> streams?

Effectively yes, this is a control on a stream and having a span
control allows you to effectively set this per-stream.  If we were to
be more drastic, then perhaps the Request class could be updated to
take an array or map of controls for each stream.  But then what about
controls that are not steam specific?  Right now, I can only see a
handful of possible ISP specific controls that may be changed
per-stream (hardware allowing):

ScalerCrop
ColourCorrectionMatrix
Sharpness
Saturation
NoiseReductionMode

Not all of these can be changed per-stream on VC4 / pisp hardware.

Naush

>
> --
> Kieran
>
>
> >
> > Regards,
> > Naush