[libcamera-devel] Camera mode selection

[Naushir Patuck]

Hi David,

On Mon, 25 Jan 2021 at 14:30, David Plowman <david.plowman at raspberrypi.com>
wrote:

> Hi Kieran
>
> Thanks for the reply. Let me try and clarify one or two of those points!
>
> On Mon, 25 Jan 2021 at 12:39, Kieran Bingham
> <kieran.bingham at ideasonboard.com> wrote:
> >
> > Hi David,
> >
> > On 22/01/2021 14:29, David Plowman wrote:
> > > Hi everyone
> > >
> > > I was wondering if I might return to the topic of camera mode
> > > selection. (Apologies if this might turn into another long
> > > meta-discussion!)
> > >
> > > We already know that there are questions as to how and when you might
> > > trade off things like better resolution versus higher framerates, and
> > > it's not clear how an application might signal what it wants.
> > >
> > > Another case I've found myself faced with recently is how to select
> > > camera modes jointly, such as when doing preview and then capture.
> > > Typically the preview will ask for a smaller resolution, and capture a
> > > larger one. You might adjust the preview to have the same aspect ratio
> > > as the capture. But so far as I understand it, it's difficult to
> > > guarantee that the preview mode will have the same field of view as
> > > the capture.
> >
> > Are you considering this use case as configuring one stream as preview,
> > running, and then reconfiguring for capture? or as two stream
> > configurations?
>
> Yes, I'm treating the two separately. So firstly I'd configure the
> stream for preview, probably with the same aspect ratio as the
> eventual capture, but with significantly lower resolution. The idea is
> of course to get reasonable framerates for preview (and make the
> system work less hard).
>
> For capture, the camera has to be stopped and reconfigured for the big
> capture resolution.
>
> >
> >
> > > Do others see this as a problem, and if so how might one fix it? Have
> > > more in the way of flags that "hint" at what the application wants?
> > > Perhaps pass in a second resolution with the rule that the mode
> > > selected for the first resolution must have the save FoV as the
> > > second?
> >
> > Is this where the 'Raw' stream would be configured, but without
> > providing buffers to control what the FoV defined at the sensor actually
> > is ?
>
> Well, maybe. I understand with the preview that I could request a raw
> stream, leave it at the full resolution, and not pass any buffers.
> But, correct me if I'm wrong, I'd end up in full resolution mode, with
> slow framerates and so forth - is that right?
>

I think this might be vendor specific to an extent.  However, for
Raspberry Pi, yes, if you request a RAW stream at full resolution, then the
sensor will run at full resolution (low framerate) and the ISP will scale
down to the requested viewfinder resolution.


>
> I suppose I could try halving the raw dimensions on the assumption
> that there's a 2x2 binned mode. But that's not always guaranteed...
> what would happen then?
>

This is a tricky one, as you said, there is no guarantee that we will
always have a 2x2 binned with full FoV.


>
> >
> > > I often find myself coming back to the idea of selecting modes
> > > explicitly - applications frequently know what sensor they're dealing
> > > with, and probably know exactly which modes they want too.
> >
>

There is certainly scope for this, but would the application really know,
or want to know about the attached sensor on a platform?  I do see a use
where the application might want to list all sensor modes available in
order for it to drive things exactly the way it wants. However, for full
control, we will also want:

- A method (in Camera?) to return all sensor modes as a vector or any
appropriate container.
- The sensor modes want to describe the FoV.  We can somewhat derive this
today with the crops and scales advertised.  Perhaps a libcamera helper to
translate between scale/crop/binning into fov?  Perhaps even a notion of
"linked" modes, for example, all modes with the same FoV?
- A simple method of requesting the specific mode from the sensor through
configure.  I am ok with the existing method of requesting a RAW stream
without providing buffers, but it does seem like a workaround, and perhaps
a formal method to do this explicitly would be better.

Regards,
Naush



> > To clarify, for a 'mode' do you mean a fixed setting of (Sensor)
> > width/height/framerate? or something more?
>
> I suppose I mean fixed width, height, field of view (i.e. binning and
> cropping within the sensor), and (maximum) framerate.
>

> Thanks!
>
> David
>
> > --
> > Kieran
> >
> >
> >
> > >
> > > Anyway, I'd be interested to hear people's thoughts on this one!
> > >
> > > Thanks and best regards
> > > David
> > > _______________________________________________
> > > libcamera-devel mailing list
> > > libcamera-devel at lists.libcamera.org
> > > https://lists.libcamera.org/listinfo/libcamera-devel
> > >
> >
> > --
> > Regards
> > --
> > Kieran
> _______________________________________________
> libcamera-devel mailing list
> libcamera-devel at lists.libcamera.org
> https://lists.libcamera.org/listinfo/libcamera-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20210125/be65057e/attachment.htm>