[libcamera-devel] [PATCH 1/1] libcamera: controls: Add StartupFrame control

[Kieran Bingham]

Quoting Naushir Patuck via libcamera-devel (2023-07-11 11:23:32)
> Hi all,
> 
> On a semi-related topic, we talked offline about improving the drop frame
> support by queuing a request buffer multiple times to avoid the need for
> allocating internal buffers.  I've tried this out and here are my findings.
> 
> Firstly, to handle queuing a single buffer multiple times, I need to increase
> the number of cache slots in V4L2BufferCache().  Perhaps
> V4L2VideoDevice::importBuffers()
> should be updated to not take in a count parameter and we just allocate slots
> for the maximum buffer count possible in V4L2 (32 I think)?  There has been a
> long-standing \todo in the RPi code to choose an appropriate value, and the
> maximum number is really the only appropriate value I can think of.

I still think allocating the maximum here in the v4l2 components is
appropriate as they are 'cheap' ...



> Once I got this working, unfortunately I realised this method would never
> actually work correctly in the common scenario where the application configures
> and uses a RAW stream.  In such cases we would queue the RAW buffer into Unicam
> N times for N dropped frames.  However this buffer is also imported into the ISP
> for processing and stats generation, all while it is also being filled by Unicam
> for the next sensor frame.  This makes the stats entirely unusable.


Aha that's a shame. I thought the restrictions were going to be in the
kernel side, so at least it's interesting to know that we /can/ queue
the same buffer multiple times (with a distinct v4l2_buffer id) and get
somewhat of the expected behaviour....

> 
> So in the end we either have to allocate additional buffers for drop frames
> (like we do right now), or we implement something like this series where the
> application is responsible for dropping/ignoring these frames.

Of course if the expected behaviour doesn't suit the use case ... then
...

This may all be different for pipeline's with an inline ISP though ...
so the research is still useful.

--
Kieran

> 
> Regards,
> Naush
> 
> 
> 
> 
> 
> On Wed, 31 May 2023 at 13:50, David Plowman via libcamera-devel
> <libcamera-devel at lists.libcamera.org> wrote:
> >
> > This control is passed back in a frame as metadata to indicate whether
> > the camera system is still in a startup phase, and the application is
> > advised to avoid using the frame.
> >
> > Signed-off-by: David Plowman <david.plowman at raspberrypi.com>
> > ---
> >  src/libcamera/control_ids.yaml | 15 +++++++++++++++
> >  1 file changed, 15 insertions(+)
> >
> > diff --git a/src/libcamera/control_ids.yaml b/src/libcamera/control_ids.yaml
> > index adea5f90..4742d907 100644
> > --- a/src/libcamera/control_ids.yaml
> > +++ b/src/libcamera/control_ids.yaml
> > @@ -694,6 +694,21 @@ controls:
> >              Continuous AF is paused. No further state changes or lens movements
> >              will occur until the AfPauseResume control is sent.
> >
> > +  - StartupFrame:
> > +      type: bool
> > +      description: |
> > +        The value true indicates that the camera system is still in a startup
> > +        phase where the output images may not be reliable, or that certain of
> > +        the control algorithms (such as AEC/AGC, AWB, and possibly others) may
> > +        still be changing quite rapidly.
> > +
> > +        Applications are advised to avoid using these frames. Mostly, they will
> > +        occur when the camera system starts for the first time, although,
> > +        depending on the sensor and the implementation, they could occur at
> > +        other times.
> > +
> > +        The value false indicates that this is a normal frame.
> > +
> >    # ----------------------------------------------------------------------------
> >    # Draft controls section
> >
> > --
> > 2.30.2
> >