[RFQ] Multiple ISP outputs and scaler crop

[David Plowman]

Hi everyone

On Fri, 3 May 2024 at 08:11, Naushir Patuck <naush at raspberrypi.com> wrote:
>
> On Thu, 2 May 2024 at 20:28, Kieran Bingham
> <kieran.bingham at ideasonboard.com> wrote:
> >
> > Quoting Naushir Patuck (2024-05-01 10:26:14)
> > > Hi all,
> > >
> > > We have come across some new use cases where users will need different
> > > scaler crops applied to each of the 2 ISP outputs configured by the
> > > application.  This is not possible on the VC4 hardware, but we can
> > > apply different crop configurations on Pi5/PISP.
> > >
> > > Right now, libcamera has a single ScalerCrop control that we apply to
> > > both outputs.  This means that the input FoV is identical for both
> > > outputs, and can distort the pixel aspect ratio on the "low res"
> > > output if the output aspect ratio on both ISP outputs are not the
> > > same.
> > >
> > > I would like to propose a couple of solutions to this:
> > >
> > > 1) Change the ScalerCrop control to be a span, so that users can
> > > specify crop for multiple output branches through the single control.
> > > This will incur an API breakage though, so may not be desirable.
> > >
> > > 2) Add a new control called ScalerCrops (with an s), that behaves
> > > identically to ScalerCrop, but works with a span of rectangles.  The
> > > value is handled in exactly the same way as  ScalerCrop, but allows
> > > the user to set muitple crops, one per output.  Of course the wording
> > > of this control needs to be explicit about the behavior, e.g. index 0
> > > corresponds to configured stream 0, and what if we only want to change
> > > one stream's scaler crop, etc.
> > >
> > > Thoughts?
> >
> > Shouldn't this be a control on a Stream?
> > I think the difficulty is we don't yet have 'stream' controls. What
> > other controls are there that can apply distinctly on different output
> > streams?
>
> Effectively yes, this is a control on a stream and having a span
> control allows you to effectively set this per-stream.  If we were to
> be more drastic, then perhaps the Request class could be updated to
> take an array or map of controls for each stream.  But then what about
> controls that are not steam specific?  Right now, I can only see a
> handful of possible ISP specific controls that may be changed
> per-stream (hardware allowing):
>
> ScalerCrop
> ColourCorrectionMatrix
> Sharpness
> Saturation
> NoiseReductionMode

Yes, also brightness and contrast. Obviously it all depends on where
you have blocks in your pipeline. More radically, if you had a PH that
would run the ISP multiple times transparently to create all the
requested output streams (assuming you can't make them all in one
pass), then pretty much anything could be per-stream. Not that we have
any intentions of going there. Yet :)

There are also colour spaces and transforms, but those are probably
per-stream configuration, not controls (i.e. you wouldn't change them
after setting them up).

It all reminds me more than a little of the Kamaros discussion about
controls, and how you know what they apply to...!

David

>
> Not all of these can be changed per-stream on VC4 / pisp hardware.
>
> Naush
>
> >
> > --
> > Kieran
> >
> >
> > >
> > > Regards,
> > > Naush