[libcamera-devel] [RFC PATCH 0/8] Request metadata: SensorSequence

[Naushir Patuck]

Hi Kieran and Laurent,


On Tue, 7 Dec 2021 at 00:07, Laurent Pinchart <
laurent.pinchart at ideasonboard.com> wrote:

> Hi Kieran,
>
> Thank you for the patches.
>
> On Mon, Dec 06, 2021 at 11:39:40PM +0000, Kieran Bingham wrote:
> > When completing a request, the individual stream buffers contain a
> > sequence number. This number is generated by the device that ultimately
> > fills that stream, but it might not be the sensor. Processing through an
> > ISP could cause the sequence numbers and timestamp data associated with
> > the completed buffer to be values representative of the ISP processing
> > rather than the (intended) capture device.
> >
> > Provide a new metadata control, still to be fully sketched out which
> > gives us a defined value to report the Camera sequence number. This
> > allows pipeline handlers to correctly set the value according to the
> > device that represents the capture from the sensor.
> >
> > This plumbing then allows applications to detect frame drops, which were
> > otherwise going unnoticed, and as such some basic additions have been
> > made to cam, qcam, and gstreamer to support this new data.
> >
> > Still possible:
> >  - Adding a validation to lc-compliance to make sure pipelines set the
> >    SensorSequence on every frame.
> >
> >  - Probably expecting some better gstreamer event integration perhaps?
> >
> >  - qcam should report more statistics on the processing overall
> >
> >  - libcamera Tracepoints could be added as an event to track when
> >    frames are detected as dropped by the core framework.
> >
> > Anything else?
>
> A basic design question: when a frame is dropped, shouldn't we report
> the corresponding request as failed ? That's how the Android camera HAL
> API operates, and while that by itself isn't a good enough reason to do
> that same, I think it offers a better way to ensure that controls get
> synchronized with the buffers that frames are captured to.
>

For the Raspberry Pi platforms, it is possible for the Unicam driver
to drop frames without the knowledge of the pipeline handler, if for example
we do not recycle bayer buffers quick enough.  Allowing the application
to look at sensor timestamps would allow it to detect these drops. Unless
I am mistaken, this would not be possible by failing the request, as the
sensor buffer recycling loop is asynchronous to the application request
loop. Having said that, they could be synchronised....

Regards,
Naush


>
> > Kieran Bingham (8):
> >   libcamera: controls: Add SensorSequence metadata control
> >   libcamera: pipeline: Set the Sensor sequence number for all pipelines
> >   cam: Use SensorTimestamp rather than buffer metadata
> >   cam: Use Sensor sequence numbers and detect frame drop
> >   qcam: main_window: Fix include ordering
> >   qcam: Use Sensor sequence numbers and detect frame drop
> >   gstreamer: gstlibcamerasrc: Fix include ordering
> >   gstreamer: Use Sensor sequence numbers and detect frame drop
> >
> >  src/cam/camera_session.cpp                    | 24 ++++++++--
> >  src/cam/camera_session.h                      |  1 +
> >  src/gstreamer/gstlibcamerasrc.cpp             | 46 +++++++++++++++----
> >  src/libcamera/control_ids.yaml                |  8 ++++
> >  src/libcamera/pipeline/ipu3/ipu3.cpp          |  4 +-
> >  .../pipeline/raspberrypi/raspberrypi.cpp      |  3 ++
> >  src/libcamera/pipeline/rkisp1/rkisp1.cpp      |  4 +-
> >  src/libcamera/pipeline/simple/simple.cpp      | 12 +++--
> >  src/libcamera/pipeline/uvcvideo/uvcvideo.cpp  |  2 +
> >  src/qcam/main_window.cpp                      | 28 +++++++++--
> >  src/qcam/main_window.h                        |  1 +
> >  11 files changed, 111 insertions(+), 22 deletions(-)
>
> --
> Regards,
>
> Laurent Pinchart
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20211207/c8a052ab/attachment-0001.htm>