[libcamera-devel] [PATCH 1/1] libcamera: controls: Add StartupFrame control

[Naushir Patuck]

On Tue, 11 Jul 2023 at 11:31, Kieran Bingham
<kieran.bingham at ideasonboard.com> wrote:
>
> Quoting Naushir Patuck via libcamera-devel (2023-07-11 11:23:32)
> > Hi all,
> >
> > On a semi-related topic, we talked offline about improving the drop frame
> > support by queuing a request buffer multiple times to avoid the need for
> > allocating internal buffers.  I've tried this out and here are my findings.
> >
> > Firstly, to handle queuing a single buffer multiple times, I need to increase
> > the number of cache slots in V4L2BufferCache().  Perhaps
> > V4L2VideoDevice::importBuffers()
> > should be updated to not take in a count parameter and we just allocate slots
> > for the maximum buffer count possible in V4L2 (32 I think)?  There has been a
> > long-standing \todo in the RPi code to choose an appropriate value, and the
> > maximum number is really the only appropriate value I can think of.
>
> I still think allocating the maximum here in the v4l2 components is
> appropriate as they are 'cheap' ...

Agree, I think 32 is the limit according to V4L2.

>
>
>
> > Once I got this working, unfortunately I realised this method would never
> > actually work correctly in the common scenario where the application configures
> > and uses a RAW stream.  In such cases we would queue the RAW buffer into Unicam
> > N times for N dropped frames.  However this buffer is also imported into the ISP
> > for processing and stats generation, all while it is also being filled by Unicam
> > for the next sensor frame.  This makes the stats entirely unusable.
>
>
> Aha that's a shame. I thought the restrictions were going to be in the
> kernel side, so at least it's interesting to know that we /can/ queue
> the same buffer multiple times (with a distinct v4l2_buffer id) and get
> somewhat of the expected behaviour....
>
> >
> > So in the end we either have to allocate additional buffers for drop frames
> > (like we do right now), or we implement something like this series where the
> > application is responsible for dropping/ignoring these frames.
>
> Of course if the expected behaviour doesn't suit the use case ... then
> ...
>
> This may all be different for pipeline's with an inline ISP though ...
> so the research is still useful.

So the question is... should we continue with this series as a possible
improvement if the pipeline handler wants to support this control?

>
> --
> Kieran
>
> >
> > Regards,
> > Naush
> >
> >
> >
> >
> >
> > On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel
> > <libcamera-devel at lists.libcamera.org> wrote:
> > >
> > > This control is passed back in a frame as metadata to indicate whether
> > > the camera system is still in a startup phase, and the application is
> > > advised to avoid using the frame.
> > >
> > > Signed-off-by: David Plowman <david.plowman at raspberrypi.com>
> > > ---
> > >  src/libcamera/control_ids.yaml | 15 +++++++++++++++
> > >  1 file changed, 15 insertions(+)
> > >
> > > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > > index adea5f90..4742d907 100644
> > > --- a/src/libcamera/control_ids.yaml
> > > +++ b/src/libcamera/control_ids.yaml
> > > @@ -694,6 +694,21 @@ controls:
> > >              Continuous AF is paused. No further state changes or lens movements
> > >              will occur until the AfPauseResume control is sent.
> > >
> > > +  - StartupFrame:
> > > +      type: bool
> > > +      description: |
> > > +        The value true indicates that the camera system is still in a startup
> > > +        phase where the output images may not be reliable, or that certain of
> > > +        the control algorithms (such as AEC/AGC, AWB, and possibly others) may
> > > +        still be changing quite rapidly.
> > > +
> > > +        Applications are advised to avoid using these frames. Mostly, they will
> > > +        occur when the camera system starts for the first time, although,
> > > +        depending on the sensor and the implementation, they could occur at
> > > +        other times.
> > > +
> > > +        The value false indicates that this is a normal frame.
> > > +
> > >    # ----------------------------------------------------------------------------
> > >    # Draft controls section
> > >
> > > --
> > > 2.30.2
> > >