[Q] early metadata reporting and the raspberry pi pipelines

[Naushir Patuck]

Hi Barnabás,

Thanks for the detailed explanation.  From my initial reading, I don't
think this is going to be a big issue for our pipeline handlers at
all.

On Thu, 1 May 2025 at 18:00, Barnabás Pőcze
<barnabas.pocze at ideasonboard.com> wrote:
>
> Hi David, Naush
>
> TLDR: Early metadata reporting (before request completion) requires constraints
> on how pipeline handlers access the metadata list. The raspberry pi pipelines handlers
> clear the metadata list, which would no longer be allowed. When can that happen?
> Can raspberry pi pipelines be potentially extended to supply metadata earlier?
>
> There are plans to provides a mechanism to pipeline handlers by which they can
> return metadata items to the application before the request completes (similarly
> to `bufferCompleted`).[0][1] This will most likely lead to pipeline handlers being
> restricted in how they can access the requests' metadata lists.
>
> For example, probably most critically, setting a given metadata item would only
> be possible once until the request is reused, to ensure that apps receive a metadata
> item once and with its final value. Allowing already "published" metadata items to
> be modified/erased would require an appreciably more complicated design, and the
> likelihood of applications handling it correctly seems slim. So any given metadata
> item should only be reported if its final value is available.

So will the API require multiple calls to "metadataCompleted()" from
pipeline handlers when items of metadata (Controls) have been added to
the request?  Or would pipeline handlers call "metadataComplete()"
only once when all metadata has been populated?  Either way is fine
for us I think.

>
> As a consequence, a pipeline handlers will probably be constrained to access the metadata
> list as an "append-only log". Currently, however, the raspberry pi pipelines call `clear()`
> on the request metadata in `{Vc4,PiSP}CameraData::tryRunPipeline()`[2]:
>
>          /*
>          * Clear the request metadata and fill it with some initial non-IPA
>          * related controls. We clear it first because the request metadata
>          * may have been populated if we have dropped the previous frame.
>          */
>          request->metadata().clear();
>          fillRequestMetadata(bayerFrame.controls, request);
>
> As far as I understand, this is mainly to handle the initial N dropped frames,
> which are handled in `CameraData::checkRequestCompleted()` (i.e. request gets
> metadata but in the end the request is not completed, so clearing is needed),
> but I want to get confirmation if that is correct and if there are other cases
> that need consideration.

This is correct, and I think the call to fillRequestMetdata() can be
moved to somewhere in checkRequestCompleted() to avoid having to clear
it because of dropped frames.  This change could be done immediately
to avoid any problems in the future.

However, I would really like to go one step further and remove the
whole notion of dropping frames in the pipeline handlers.  This would
allow quite a few simplifications to our code :)  I think we have this
on our agenda to discuss in Nice.

>
> Of course a pipeline handler could still queue metadata in a private `ControlList`
> and then set it when the request is completed, in case it does not want to or
> cannot provide metadata earlier. This is what is currently implemented in the
> raspberry pi pipeline handlers at the moment for testing purposes.

This would work as well, but I think can be avoided.

Regards,

Naush

>
> So in summary, I would just like to get your thoughts regarding when the raspberry pi
> pipelines might need to clear the request metadata, and whether you think the rpi
> pipeline handlers could me modified (how easily?) to support early metadata reporting.
>
>
> [0]: https://patchwork.libcamera.org/cover/22111/
> [1]: https://patchwork.libcamera.org/cover/22226/
> [2]: https://patchwork.libcamera.org/patch/12251/
>
>
> Thanks,
> Barnabás Pőcze