[libcamera-devel] [RFC PATCH 0/8] Request metadata: SensorSequence

[Kieran Bingham]

When completing a request, the individual stream buffers contain a
sequence number. This number is generated by the device that ultimately
fills that stream, but it might not be the sensor. Processing through an
ISP could cause the sequence numbers and timestamp data associated with
the completed buffer to be values representative of the ISP processing
rather than the (intended) capture device.

Provide a new metadata control, still to be fully sketched out which
gives us a defined value to report the Camera sequence number. This
allows pipeline handlers to correctly set the value according to the
device that represents the capture from the sensor.

This plumbing then allows applications to detect frame drops, which were
otherwise going unnoticed, and as such some basic additions have been
made to cam, qcam, and gstreamer to support this new data.

Still possible:
 - Adding a validation to lc-compliance to make sure pipelines set the
   SensorSequence on every frame.

 - Probably expecting some better gstreamer event integration perhaps? 

 - qcam should report more statistics on the processing overall

 - libcamera Tracepoints could be added as an event to track when
   frames are detected as dropped by the core framework.

Anything else?


Kieran Bingham (8):
  libcamera: controls: Add SensorSequence metadata control
  libcamera: pipeline: Set the Sensor sequence number for all pipelines
  cam: Use SensorTimestamp rather than buffer metadata
  cam: Use Sensor sequence numbers and detect frame drop
  qcam: main_window: Fix include ordering
  qcam: Use Sensor sequence numbers and detect frame drop
  gstreamer: gstlibcamerasrc: Fix include ordering
  gstreamer: Use Sensor sequence numbers and detect frame drop

 src/cam/camera_session.cpp                    | 24 ++++++++--
 src/cam/camera_session.h                      |  1 +
 src/gstreamer/gstlibcamerasrc.cpp             | 46 +++++++++++++++----
 src/libcamera/control_ids.yaml                |  8 ++++
 src/libcamera/pipeline/ipu3/ipu3.cpp          |  4 +-
 .../pipeline/raspberrypi/raspberrypi.cpp      |  3 ++
 src/libcamera/pipeline/rkisp1/rkisp1.cpp      |  4 +-
 src/libcamera/pipeline/simple/simple.cpp      | 12 +++--
 src/libcamera/pipeline/uvcvideo/uvcvideo.cpp  |  2 +
 src/qcam/main_window.cpp                      | 28 +++++++++--
 src/qcam/main_window.h                        |  1 +
 11 files changed, 111 insertions(+), 22 deletions(-)

-- 
2.30.2