[Q] early metadata reporting and the raspberry pi pipelines

[Barnabás Pőcze]

Hi David, Naush

TLDR: Early metadata reporting (before request completion) requires constraints
on how pipeline handlers access the metadata list. The raspberry pi pipelines handlers
clear the metadata list, which would no longer be allowed. When can that happen?
Can raspberry pi pipelines be potentially extended to supply metadata earlier?

There are plans to provides a mechanism to pipeline handlers by which they can
return metadata items to the application before the request completes (similarly
to `bufferCompleted`).[0][1] This will most likely lead to pipeline handlers being
restricted in how they can access the requests' metadata lists.

For example, probably most critically, setting a given metadata item would only
be possible once until the request is reused, to ensure that apps receive a metadata
item once and with its final value. Allowing already "published" metadata items to
be modified/erased would require an appreciably more complicated design, and the
likelihood of applications handling it correctly seems slim. So any given metadata
item should only be reported if its final value is available.

As a consequence, a pipeline handlers will probably be constrained to access the metadata
list as an "append-only log". Currently, however, the raspberry pi pipelines call `clear()`
on the request metadata in `{Vc4,PiSP}CameraData::tryRunPipeline()`[2]:

         /*
	 * Clear the request metadata and fill it with some initial non-IPA
	 * related controls. We clear it first because the request metadata
	 * may have been populated if we have dropped the previous frame.
	 */
         request->metadata().clear();
         fillRequestMetadata(bayerFrame.controls, request);

As far as I understand, this is mainly to handle the initial N dropped frames,
which are handled in `CameraData::checkRequestCompleted()` (i.e. request gets
metadata but in the end the request is not completed, so clearing is needed),
but I want to get confirmation if that is correct and if there are other cases
that need consideration.

Of course a pipeline handler could still queue metadata in a private `ControlList`
and then set it when the request is completed, in case it does not want to or
cannot provide metadata earlier. This is what is currently implemented in the
raspberry pi pipeline handlers at the moment for testing purposes.

So in summary, I would just like to get your thoughts regarding when the raspberry pi
pipelines might need to clear the request metadata, and whether you think the rpi
pipeline handlers could me modified (how easily?) to support early metadata reporting.


[0]: https://patchwork.libcamera.org/cover/22111/
[1]: https://patchwork.libcamera.org/cover/22226/
[2]: https://patchwork.libcamera.org/patch/12251/


Thanks,
Barnabás Pőcze