[libcamera-devel] Camera mode selection

[David Plowman]

Hi everyone

Here's another little use case to add to the list.

The other day I got interested in a "quad Bayer" sensor, this is one
where each pixel in the Bayer pattern is replaced by a little 2x2
block all of that same colour channel. So you actually get (for
example) 2 rows of RRGGRRGG.... and then 2 rows of GGBBGGBB...

This sensor has a 2x2 binned mode - which actually gives you true
Bayer, and without the resampling headaches of binning a "regular"
Bayer sensor, but it also has an in-built HDR mode. This mode gives
you the same resolution and similar-ish frame rates to the binned
mode, but the pixels in each little block can be driven at different
exposures, and it combines all four of them into a single "HDR pixel".

So is this another case where the application simply has to know which
of the sensor modes it wants? Is there an argument for associating
flags with camera modes indicating special features like this (and how
would V4L2 signal it...)?

Best regards
David

On Thu, 28 Jan 2021 at 09:59, David Plowman
<david.plowman at raspberrypi.com> wrote:
>
> Hi Naush
>
> Thanks for all the replies!
>
> On Mon, 25 Jan 2021 at 14:48, Naushir Patuck <naush at raspberrypi.com> wrote:
> >
> > Hi David,
> >
> > On Mon, 25 Jan 2021 at 14:30, David Plowman <david.plowman at raspberrypi.com> wrote:
> >>
> >> Hi Kieran
> >>
> >> Thanks for the reply. Let me try and clarify one or two of those points!
> >>
> >> On Mon, 25 Jan 2021 at 12:39, Kieran Bingham
> >> <kieran.bingham at ideasonboard.com> wrote:
> >> >
> >> > Hi David,
> >> >
> >> > On 22/01/2021 14:29, David Plowman wrote:
> >> > > Hi everyone
> >> > >
> >> > > I was wondering if I might return to the topic of camera mode
> >> > > selection. (Apologies if this might turn into another long
> >> > > meta-discussion!)
> >> > >
> >> > > We already know that there are questions as to how and when you might
> >> > > trade off things like better resolution versus higher framerates, and
> >> > > it's not clear how an application might signal what it wants.
> >> > >
> >> > > Another case I've found myself faced with recently is how to select
> >> > > camera modes jointly, such as when doing preview and then capture.
> >> > > Typically the preview will ask for a smaller resolution, and capture a
> >> > > larger one. You might adjust the preview to have the same aspect ratio
> >> > > as the capture. But so far as I understand it, it's difficult to
> >> > > guarantee that the preview mode will have the same field of view as
> >> > > the capture.
> >> >
> >> > Are you considering this use case as configuring one stream as preview,
> >> > running, and then reconfiguring for capture? or as two stream
> >> > configurations?
> >>
> >> Yes, I'm treating the two separately. So firstly I'd configure the
> >> stream for preview, probably with the same aspect ratio as the
> >> eventual capture, but with significantly lower resolution. The idea is
> >> of course to get reasonable framerates for preview (and make the
> >> system work less hard).
> >>
> >> For capture, the camera has to be stopped and reconfigured for the big
> >> capture resolution.
> >>
> >> >
> >> >
> >> > > Do others see this as a problem, and if so how might one fix it? Have
> >> > > more in the way of flags that "hint" at what the application wants?
> >> > > Perhaps pass in a second resolution with the rule that the mode
> >> > > selected for the first resolution must have the save FoV as the
> >> > > second?
> >> >
> >> > Is this where the 'Raw' stream would be configured, but without
> >> > providing buffers to control what the FoV defined at the sensor actually
> >> > is ?
> >>
> >> Well, maybe. I understand with the preview that I could request a raw
> >> stream, leave it at the full resolution, and not pass any buffers.
> >> But, correct me if I'm wrong, I'd end up in full resolution mode, with
> >> slow framerates and so forth - is that right?
> >
> >
> > I think this might be vendor specific to an extent.  However, for Raspberry Pi, yes, if you request a RAW stream at full resolution, then the sensor will run at full resolution (low framerate) and the ISP will scale down to the requested viewfinder resolution.
> >
> >>
> >>
> >> I suppose I could try halving the raw dimensions on the assumption
> >> that there's a 2x2 binned mode. But that's not always guaranteed...
> >> what would happen then?
> >
> >
> > This is a tricky one, as you said, there is no guarantee that we will always have a 2x2 binned with full FoV.
> >
> >>
> >>
> >> >
> >> > > I often find myself coming back to the idea of selecting modes
> >> > > explicitly - applications frequently know what sensor they're dealing
> >> > > with, and probably know exactly which modes they want too.
> >> >
> >
> >
> > There is certainly scope for this, but would the application really know, or want to know about the attached sensor on a platform?  I do see a use where the application might want to list all sensor modes available in order for it to drive things exactly the way it wants. However, for full control, we will also want:
> >
> > - A method (in Camera?) to return all sensor modes as a vector or any appropriate container.
>
> I think this is a good idea. We need to maintain a convenient mode of
> operation for applications that don't much care and "just want an
> image of this size" (as we have now). But there will always be
> applications that want to control this more closely and no amount of
> rewriting the "try and guess the best camera mode" will suit everyone.
>
> > - The sensor modes want to describe the FoV.  We can somewhat derive this today with the crops and scales advertised.  Perhaps a libcamera helper to translate between scale/crop/binning into fov?  Perhaps even a notion of "linked" modes, for example, all modes with the same FoV?
>
> I think we should be advertising this in our camera formats, or in the
> stream configuration, or somewhere! If we have a function that returns
> a list/vector of all the allowed formats, this information should be
> in there. Making things like binning/scaling explicit, and including
> information on framerates, would also be desirable.
>
> > - A simple method of requesting the specific mode from the sensor through configure.  I am ok with the existing method of requesting a RAW stream without providing buffers, but it does seem like a workaround, and perhaps a formal method to do this explicitly would be better.
>
> Agree. I'd love to point to an item in the list of supported formats
> and say "that one, please!"
>
> Thanks
> David
>
> >
> > Regards,
> > Naush
> >
> >
> >>
> >> > To clarify, for a 'mode' do you mean a fixed setting of (Sensor)
> >> > width/height/framerate? or something more?
> >>
> >> I suppose I mean fixed width, height, field of view (i.e. binning and
> >> cropping within the sensor), and (maximum) framerate.
> >>
> >>
> >> Thanks!
> >>
> >> David
> >>
> >> > --
> >> > Kieran
> >> >
> >> >
> >> >
> >> > >
> >> > > Anyway, I'd be interested to hear people's thoughts on this one!
> >> > >
> >> > > Thanks and best regards
> >> > > David
> >> > > _______________________________________________
> >> > > libcamera-devel mailing list
> >> > > libcamera-devel at lists.libcamera.org
> >> > > https://lists.libcamera.org/listinfo/libcamera-devel
> >> > >
> >> >
> >> > --
> >> > Regards
> >> > --
> >> > Kieran
> >> _______________________________________________
> >> libcamera-devel mailing list
> >> libcamera-devel at lists.libcamera.org
> >> https://lists.libcamera.org/listinfo/libcamera-devel