[libcamera-devel] [PATCH v2 1/3] libcamera: controls: Add frame duration control

[Dave Stevenson]

Hi Jacopo

On Wed, 3 Jun 2020 at 11:11, Jacopo Mondi <jacopo at jmondi.org> wrote:
>
> Hi Naush,
>
> On Wed, Jun 03, 2020 at 10:44:18AM +0100, Naushir Patuck wrote:
> > Hi Jacopo,
> >
> > On Wed, 3 Jun 2020 at 09:42, Jacopo Mondi <jacopo at jmondi.org> wrote:
> > >
> > > Hi Naush,
> > >
> > >     I won't go into details here as I would like Laurent to chime in
> > > and express his view on this topic, but I would have a question in the
> > > meantime.
> > >
> > > On Tue, Jun 02, 2020 at 10:38:41AM +0100, Naushir Patuck wrote:
> > > > Hi Jacopo,
> > > >
> > > >
> > > > On Mon, 1 Jun 2020 at 16:40, Jacopo Mondi <jacopo at jmondi.org> wrote:
> > > > >
> > > > > Hi Naush,
> > > > >
> > > > > On Thu, May 28, 2020 at 09:52:51AM +0100, Naushir Patuck wrote:
> > >
> > > [snip]
> > >
> > > > > > > > On "FrameDuration" itself: would it make sense to only considered it
> > > > > > > > when running with AE off ? In that case it would be applications
> > > > > > > > responsibility to opportunely calculate the exposure time and the frame
> > > > > > > > duration, but I think it's expected if you run in manual mode. And for
> > > > > > > > reference, that's what Android does as well :)
> > > > > > > >
> > > > > > > > It would also rule out the need to specify how the frameDurationLimits
> > > > > > > > would be used when running in manual mode and wanting to configure a
> > > > > > > > precise value there, as I guess you would have to give to min and max
> > > > > > > > the same value to express that.
> > > > > >
> > > > > > FrameDurationLimits should only really apply with AE enabled.  AE
> > > > > > varies the shutter speed all over the place, that could cause
> > > > > > framerate changes that we need to clip.  If AE is disabled and we have
> > > > > > a manual shutter speed set, that dictates the frame duration, and any
> > > > > > limits will not apply.  Perhaps I should add this in the description?
> > > > > >
> > > >
> > > > I have to correct myself here.  FrameDurationLimits will also apply
> > > > with manual shutter speeds when AE is disabled.  We could, say, have a
> > > > manual shutter speed request of 25 ms, but still want to run at 30fps.
> > > > There is a question on what happens if they clash, e.g. request a
> > > > manual shutter speed of 66ms, and a FrameDuration of 30fps?  What gets
> > > > prioritised?  Currently, in our implementation, FrameDuration will
> > > > always be prioritised.
> > >
> > > Is there a rationle behind this decision ? I'm asking as I see Android
> > > going in the other direction, with the exposure time always overriding
> > > the requested frame duration.
> >
> > This was an entirely arbitrary choice on my part - based on what was
> > easiest to do at the time :)
> >
> > > It is specified in their documentation that the mximum frame duration
> > > a camera reports (lower frame rate) shall be at least the maximum
> > > allowed exposure time, so that all possible exposure values are known
> > > to be in the requested FPS range. I guess this should be enforced even
> > > if we give frame duration priority, or should we cap the maximum frame
> > > duration to the maximum available exposure time ?
> >
> > I have not looked at the Android docs in ages, but do they have two
> > sets of frame duration limits, one that the sensor will report (sounds
> > like this is what you are working on?) and one that the user requests
>
> yes, there's a static property that reports the minFrameDurations for
> each supported stream configuration (including the ISP introduced
> dealys)
> https://jmondi.org/android_metadata_tags/docs.html#static_android.scaler.availableMinFrameDurations
>
> And two controls for the sensor frame durations, one static property
> https://jmondi.org/android_metadata_tags/docs.html#static_android.sensor.info.maxFrameDuration
>
> and a control
> https://jmondi.org/android_metadata_tags/docs.html#controls_android.sensor.frameDuration
>
> > (of course, bounded by the former)?  This was where my thinking was
> > going, and with frame duration priority, exposure time is capped by
> > the user request (if given), or the sensor limits.  But as always, if
> > the consensus is to behave differently, I am happy to change things
> > around.
>
> I don't have preferences, nor reasons that make me think one is better
> than the other. I wonder if we should enforce this at all or let pipeline
> handler/IPA decide that.

V4L2 has it that VBLANK (ie frame rate) clips the max exposure time,
therefore if libcamera doesn't follow that then you have to be very
conscious of doing things in the correct order - reset VBLANK / frame
rate first to reset exposure ranges, and then set your desired
exposure time.
I'm trying to think whether I've encountered sensors that won't allow
you to adjust the blanking periods whilst streaming, as those would
limit the exposure time without stopping and restarting. None are
jumping to mind, but others may have encountered such.

For AE modes, there is the override control
V4L2_CID_EXPOSURE_AUTO_PRIORITY [1], but otherwise frame rate all
other references appear to say frame rate dictates max exposure.

For performance tests it could be considered useful if the specified
frame rate was guaranteed to be delivered. Then again you could claim
it was a flawed test if it set the exposure time to a value that
contradicted the frame rate it was trying to measure.

  Dave

[1] https://www.kernel.org/doc/html/latest/media/uapi/v4l/ext-ctrls-camera.html

> Thanks
>   j
>
> >
> > Regards,
> > Naush
> >
> >
> > >
> > > Just trying to get more data points to make an informed decision, as
> > > your question on what has to be prioritized is quite sensible...
> > >
> > > Thanks
> > >   j
> _______________________________________________
> libcamera-devel mailing list
> libcamera-devel at lists.libcamera.org
> https://lists.libcamera.org/listinfo/libcamera-devel