[libcamera-devel] [RFC PATCH 0/8] Request metadata: SensorSequence

[Laurent Pinchart]

Hi Kieran,

Thank you for the patches.

On Mon, Dec 06, 2021 at 11:39:40PM +0000, Kieran Bingham wrote:
> When completing a request, the individual stream buffers contain a
> sequence number. This number is generated by the device that ultimately
> fills that stream, but it might not be the sensor. Processing through an
> ISP could cause the sequence numbers and timestamp data associated with
> the completed buffer to be values representative of the ISP processing
> rather than the (intended) capture device.
> 
> Provide a new metadata control, still to be fully sketched out which
> gives us a defined value to report the Camera sequence number. This
> allows pipeline handlers to correctly set the value according to the
> device that represents the capture from the sensor.
> 
> This plumbing then allows applications to detect frame drops, which were
> otherwise going unnoticed, and as such some basic additions have been
> made to cam, qcam, and gstreamer to support this new data.
> 
> Still possible:
>  - Adding a validation to lc-compliance to make sure pipelines set the
>    SensorSequence on every frame.
> 
>  - Probably expecting some better gstreamer event integration perhaps? 
> 
>  - qcam should report more statistics on the processing overall
> 
>  - libcamera Tracepoints could be added as an event to track when
>    frames are detected as dropped by the core framework.
> 
> Anything else?

A basic design question: when a frame is dropped, shouldn't we report
the corresponding request as failed ? That's how the Android camera HAL
API operates, and while that by itself isn't a good enough reason to do
that same, I think it offers a better way to ensure that controls get
synchronized with the buffers that frames are captured to.

> Kieran Bingham (8):
>   libcamera: controls: Add SensorSequence metadata control
>   libcamera: pipeline: Set the Sensor sequence number for all pipelines
>   cam: Use SensorTimestamp rather than buffer metadata
>   cam: Use Sensor sequence numbers and detect frame drop
>   qcam: main_window: Fix include ordering
>   qcam: Use Sensor sequence numbers and detect frame drop
>   gstreamer: gstlibcamerasrc: Fix include ordering
>   gstreamer: Use Sensor sequence numbers and detect frame drop
> 
>  src/cam/camera_session.cpp                    | 24 ++++++++--
>  src/cam/camera_session.h                      |  1 +
>  src/gstreamer/gstlibcamerasrc.cpp             | 46 +++++++++++++++----
>  src/libcamera/control_ids.yaml                |  8 ++++
>  src/libcamera/pipeline/ipu3/ipu3.cpp          |  4 +-
>  .../pipeline/raspberrypi/raspberrypi.cpp      |  3 ++
>  src/libcamera/pipeline/rkisp1/rkisp1.cpp      |  4 +-
>  src/libcamera/pipeline/simple/simple.cpp      | 12 +++--
>  src/libcamera/pipeline/uvcvideo/uvcvideo.cpp  |  2 +
>  src/qcam/main_window.cpp                      | 28 +++++++++--
>  src/qcam/main_window.h                        |  1 +
>  11 files changed, 111 insertions(+), 22 deletions(-)

-- 
Regards,

Laurent Pinchart