[libcamera-devel] Camera mode selection

[David Plowman]

Hi Naush

Thanks for all the replies!

On Mon, 25 Jan 2021 at 14:48, Naushir Patuck <naush at raspberrypi.com> wrote:
>
> Hi David,
>
> On Mon, 25 Jan 2021 at 14:30, David Plowman <david.plowman at raspberrypi.com> wrote:
>>
>> Hi Kieran
>>
>> Thanks for the reply. Let me try and clarify one or two of those points!
>>
>> On Mon, 25 Jan 2021 at 12:39, Kieran Bingham
>> <kieran.bingham at ideasonboard.com> wrote:
>> >
>> > Hi David,
>> >
>> > On 22/01/2021 14:29, David Plowman wrote:
>> > > Hi everyone
>> > >
>> > > I was wondering if I might return to the topic of camera mode
>> > > selection. (Apologies if this might turn into another long
>> > > meta-discussion!)
>> > >
>> > > We already know that there are questions as to how and when you might
>> > > trade off things like better resolution versus higher framerates, and
>> > > it's not clear how an application might signal what it wants.
>> > >
>> > > Another case I've found myself faced with recently is how to select
>> > > camera modes jointly, such as when doing preview and then capture.
>> > > Typically the preview will ask for a smaller resolution, and capture a
>> > > larger one. You might adjust the preview to have the same aspect ratio
>> > > as the capture. But so far as I understand it, it's difficult to
>> > > guarantee that the preview mode will have the same field of view as
>> > > the capture.
>> >
>> > Are you considering this use case as configuring one stream as preview,
>> > running, and then reconfiguring for capture? or as two stream
>> > configurations?
>>
>> Yes, I'm treating the two separately. So firstly I'd configure the
>> stream for preview, probably with the same aspect ratio as the
>> eventual capture, but with significantly lower resolution. The idea is
>> of course to get reasonable framerates for preview (and make the
>> system work less hard).
>>
>> For capture, the camera has to be stopped and reconfigured for the big
>> capture resolution.
>>
>> >
>> >
>> > > Do others see this as a problem, and if so how might one fix it? Have
>> > > more in the way of flags that "hint" at what the application wants?
>> > > Perhaps pass in a second resolution with the rule that the mode
>> > > selected for the first resolution must have the save FoV as the
>> > > second?
>> >
>> > Is this where the 'Raw' stream would be configured, but without
>> > providing buffers to control what the FoV defined at the sensor actually
>> > is ?
>>
>> Well, maybe. I understand with the preview that I could request a raw
>> stream, leave it at the full resolution, and not pass any buffers.
>> But, correct me if I'm wrong, I'd end up in full resolution mode, with
>> slow framerates and so forth - is that right?
>
>
> I think this might be vendor specific to an extent.  However, for Raspberry Pi, yes, if you request a RAW stream at full resolution, then the sensor will run at full resolution (low framerate) and the ISP will scale down to the requested viewfinder resolution.
>
>>
>>
>> I suppose I could try halving the raw dimensions on the assumption
>> that there's a 2x2 binned mode. But that's not always guaranteed...
>> what would happen then?
>
>
> This is a tricky one, as you said, there is no guarantee that we will always have a 2x2 binned with full FoV.
>
>>
>>
>> >
>> > > I often find myself coming back to the idea of selecting modes
>> > > explicitly - applications frequently know what sensor they're dealing
>> > > with, and probably know exactly which modes they want too.
>> >
>
>
> There is certainly scope for this, but would the application really know, or want to know about the attached sensor on a platform?  I do see a use where the application might want to list all sensor modes available in order for it to drive things exactly the way it wants. However, for full control, we will also want:
>
> - A method (in Camera?) to return all sensor modes as a vector or any appropriate container.

I think this is a good idea. We need to maintain a convenient mode of
operation for applications that don't much care and "just want an
image of this size" (as we have now). But there will always be
applications that want to control this more closely and no amount of
rewriting the "try and guess the best camera mode" will suit everyone.

> - The sensor modes want to describe the FoV.  We can somewhat derive this today with the crops and scales advertised.  Perhaps a libcamera helper to translate between scale/crop/binning into fov?  Perhaps even a notion of "linked" modes, for example, all modes with the same FoV?

I think we should be advertising this in our camera formats, or in the
stream configuration, or somewhere! If we have a function that returns
a list/vector of all the allowed formats, this information should be
in there. Making things like binning/scaling explicit, and including
information on framerates, would also be desirable.

> - A simple method of requesting the specific mode from the sensor through configure.  I am ok with the existing method of requesting a RAW stream without providing buffers, but it does seem like a workaround, and perhaps a formal method to do this explicitly would be better.

Agree. I'd love to point to an item in the list of supported formats
and say "that one, please!"

Thanks
David

>
> Regards,
> Naush
>
>
>>
>> > To clarify, for a 'mode' do you mean a fixed setting of (Sensor)
>> > width/height/framerate? or something more?
>>
>> I suppose I mean fixed width, height, field of view (i.e. binning and
>> cropping within the sensor), and (maximum) framerate.
>>
>>
>> Thanks!
>>
>> David
>>
>> > --
>> > Kieran
>> >
>> >
>> >
>> > >
>> > > Anyway, I'd be interested to hear people's thoughts on this one!
>> > >
>> > > Thanks and best regards
>> > > David
>> > > _______________________________________________
>> > > libcamera-devel mailing list
>> > > libcamera-devel at lists.libcamera.org
>> > > https://lists.libcamera.org/listinfo/libcamera-devel
>> > >
>> >
>> > --
>> > Regards
>> > --
>> > Kieran
>> _______________________________________________
>> libcamera-devel mailing list
>> libcamera-devel at lists.libcamera.org
>> https://lists.libcamera.org/listinfo/libcamera-devel