<div dir="ltr"><div dir="ltr">Hi Kieran and Laurent,<div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, 7 Dec 2021 at 00:07, Laurent Pinchart <<a href="mailto:laurent.pinchart@ideasonboard.com">laurent.pinchart@ideasonboard.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Kieran,<br>
<br>
Thank you for the patches.<br>
<br>
On Mon, Dec 06, 2021 at 11:39:40PM +0000, Kieran Bingham wrote:<br>
> When completing a request, the individual stream buffers contain a<br>
> sequence number. This number is generated by the device that ultimately<br>
> fills that stream, but it might not be the sensor. Processing through an<br>
> ISP could cause the sequence numbers and timestamp data associated with<br>
> the completed buffer to be values representative of the ISP processing<br>
> rather than the (intended) capture device.<br>
> <br>
> Provide a new metadata control, still to be fully sketched out which<br>
> gives us a defined value to report the Camera sequence number. This<br>
> allows pipeline handlers to correctly set the value according to the<br>
> device that represents the capture from the sensor.<br>
> <br>
> This plumbing then allows applications to detect frame drops, which were<br>
> otherwise going unnoticed, and as such some basic additions have been<br>
> made to cam, qcam, and gstreamer to support this new data.<br>
> <br>
> Still possible:<br>
>Â - Adding a validation to lc-compliance to make sure pipelines set the<br>
>Â Â SensorSequence on every frame.<br>
> <br>
>Â - Probably expecting some better gstreamer event integration perhaps? <br>
> <br>
>Â - qcam should report more statistics on the processing overall<br>
> <br>
>Â - libcamera Tracepoints could be added as an event to track when<br>
>Â Â frames are detected as dropped by the core framework.<br>
> <br>
> Anything else?<br>
<br>
A basic design question: when a frame is dropped, shouldn't we report<br>
the corresponding request as failed ? That's how the Android camera HAL<br>
API operates, and while that by itself isn't a good enough reason to do<br>
that same, I think it offers a better way to ensure that controls get<br>
synchronized with the buffers that frames are captured to.<br></blockquote><div><br></div><div>For the Raspberry Pi platforms, it is possible for the Unicam driver</div><div>to drop frames without the knowledge of the pipeline handler, if for example</div><div>we do not recycle bayer buffers quick enough. Allowing the application</div><div>to look at sensor timestamps would allow it to detect these drops. Unless</div><div>I am mistaken, this would not be possible by failing the request, as the</div><div>sensor buffer recycling loop is asynchronous to the application request</div><div>loop. Having said that, they could be synchronised....</div><div><br></div><div>Regards,</div><div>Naush</div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
> Kieran Bingham (8):<br>
>Â Â libcamera: controls: Add SensorSequence metadata control<br>
>Â Â libcamera: pipeline: Set the Sensor sequence number for all pipelines<br>
>Â Â cam: Use SensorTimestamp rather than buffer metadata<br>
>Â Â cam: Use Sensor sequence numbers and detect frame drop<br>
>Â Â qcam: main_window: Fix include ordering<br>
>Â Â qcam: Use Sensor sequence numbers and detect frame drop<br>
>Â Â gstreamer: gstlibcamerasrc: Fix include ordering<br>
>Â Â gstreamer: Use Sensor sequence numbers and detect frame drop<br>
> <br>
> src/cam/camera_session.cpp          | 24 ++++++++--<br>
> src/cam/camera_session.h           | 1 +<br>
> src/gstreamer/gstlibcamerasrc.cpp       | 46 +++++++++++++++----<br>
> src/libcamera/control_ids.yaml        | 8 ++++<br>
> src/libcamera/pipeline/ipu3/ipu3.cpp     | 4 +-<br>
> .../pipeline/raspberrypi/raspberrypi.cpp   | 3 ++<br>
> src/libcamera/pipeline/rkisp1/rkisp1.cpp   | 4 +-<br>
> src/libcamera/pipeline/simple/simple.cpp   | 12 +++--<br>
> src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 2 +<br>
> src/qcam/main_window.cpp           | 28 +++++++++--<br>
> src/qcam/main_window.h            | 1 +<br>
>Â 11 files changed, 111 insertions(+), 22 deletions(-)<br>
<br>
-- <br>
Regards,<br>
<br>
Laurent Pinchart<br>
</blockquote></div></div>