[libcamera-devel] [PATCH 6/8] qcam: Use Sensor sequence numbers and detect frame drop

[Umang Jain]

Hi Kieran

On 12/7/21 5:09 AM, Kieran Bingham wrote:
> The stream buffer sequence numbers might produce sequential
> monotonic sequence numbers from an ISP producing a frame for every
> input.
>
> This however, doesn't capture pipeline stalls that cause us to miss or
> drop frames from the sensor.
>
> Use the SensorSequence metadata to report sequence information, and
> report with qInfo if a frame drop is detected.
>
> Signed-off-by: Kieran Bingham <kieran.bingham at ideasonboard.com>


Reviewed-by: Umang Jain <umang.jain at ideasonboard.com>

> ---
>   src/qcam/main_window.cpp | 22 ++++++++++++++++++++++
>   src/qcam/main_window.h   |  1 +
>   2 files changed, 23 insertions(+)
>
> diff --git a/src/qcam/main_window.cpp b/src/qcam/main_window.cpp
> index dd0e51f55b70..8d1d5b275aed 100644
> --- a/src/qcam/main_window.cpp
> +++ b/src/qcam/main_window.cpp
> @@ -12,6 +12,7 @@
>   #include <string>
>   
>   #include <libcamera/camera_manager.h>
> +#include <libcamera/control_ids.h>
>   #include <libcamera/version.h>
>   
>   #include <QComboBox>
> @@ -510,6 +511,7 @@ int MainWindow::startCapture()
>   	previousFrames_ = 0;
>   	framesCaptured_ = 0;
>   	lastBufferTime_ = 0;
> +	sequence_ = 0;
>   
>   	ret = camera_->start();
>   	if (ret) {
> @@ -719,6 +721,26 @@ void MainWindow::processCapture()
>   		request = doneQueue_.dequeue();
>   	}
>   
> +	/* Parse the request metadata for useful information */
> +	for (const auto &ctrl : request->metadata()) {
> +		const int id = ctrl.first;
> +		const ControlValue &value = ctrl.second;
> +
> +		if (id == controls::SensorSequence) {
> +			/* Handle basic frame drop detection and reporting. */
> +			int64_t sequence = value.get<int64_t>();
> +			if (sequence_ == 0)
> +				sequence_ = sequence - 1;
> +			unsigned int drops = sequence - sequence_ - 1;
> +			if (drops)
> +				qInfo() << "sequence: [" << sequence << "] *"
> +					<< drops << " frame drops detected *";
> +			sequence_ = sequence;
> +		}
> +
> +		/* \todo Handle all/other metadata types here. */
> +	}
> +
>   	/* Process buffers. */
>   	if (request->buffers().count(vfStream_))
>   		processViewfinder(request->buffers().at(vfStream_));
> diff --git a/src/qcam/main_window.h b/src/qcam/main_window.h
> index 3fbe872c0b5b..97568d6b619d 100644
> --- a/src/qcam/main_window.h
> +++ b/src/qcam/main_window.h
> @@ -120,6 +120,7 @@ private:
>   	QMutex mutex_; /* Protects freeBuffers_, doneQueue_, and freeQueue_ */
>   
>   	uint64_t lastBufferTime_;
> +	uint64_t sequence_;
>   	QElapsedTimer frameRateInterval_;
>   	uint32_t previousFrames_;
>   	uint32_t framesCaptured_;