[libcamera-devel] Debayering

[Jacopo Mondi]

Hi Dorota,

On Wed, Nov 10, 2021 at 02:01:23PM +0100, Dorota Czaplejewicz wrote:
> Hi all,
>
> I'm making sure that Librem 5 cameras can be easily used via libcamera. One of the steps that are needed for the "easily" part is having a builtin debayering mechanism.
>
> Before I jump head-in, I wanted to ask for some guidance about where to place this support, and how to expose it to the consumer.
>
> For a simplest implementation, supported cameras could add the RGB format to the list of supported formats, and transparently convert the bayer data to RGB if configured for RGB. That opens up a question about the opt-in mechanism, but is otherwise simple enough.
>
> But it comes with a downside: the raw bayer frames are lost, and the setting cannot be changed at runtime, even though it's a pure software operation. To overcome that, some other API than configuration would have to be used.
>
> I intend to start by adding a CPU-based debayering implementation for the sake of building it up quickly. Would a simple implementation like I outlined above suitable for upstreaming?

For the record, this seems potentially similar to other 'plugins' like the
software JPEG encoder we have in the HAL. It would be lovely to find a
proper place where to place them :)

I used to think at software post-processors like these more like
external plugins to libcamera, something applications could easily pipe
their streams through instead of receiving processed data from an
additional software-generated stream from the Camera, but I would be
more than happy to have my mind changed.

If I got your proposal right, the sw-debayer would be an opt-in
mechanism for pipeline handlers, which in the case they cannot perform
debayering through an ISP would create an additional 'virtual' stream
that application can select transparently from the Camera. The ideal
target would be the simple pipeline handler, as it is meant to support
platforms without an ISP.

The first concern I have for such a design is about the processing
time software debayering could take as, compared to JPEG which is
triggered just for still capture requests, the debayering would need to
be applied to every frame. Is this correct ?

I cannot quantify how long it could take (and I guess it also depends
on the platform capabilities and the system load) but it doesn't seems
impossible that it could create stalls in the pipeline, requiring the
pipeline handler to drop frames and reduce the frame rate. Do you have
any idea about the possible numbers here ?

I'm also a bit skeptical that implementing opt-in at pipeline handler
level is the right decision. As I understood it, such an opt-in
mechanism should be driven by applications, not be platform specific
even if I understand that for the simple pipeline handler this looks
like something desirable for all platforms.

There are other possible developments which might be worth a
comparison with.

There is an implementation of a SW ISP on the list from Siyuan. My
understanding is that it should target format conversion too and not
just statistics handling. I've cc-ed Paul and Siyuan in case they want
to chime in.

Laurent is also working on an API to perform stream reprocessing. The
usual use case is to have one RAW frame as captured from the
camera 'reprocessed' through the capture pipeline. In example on
Android one of the most common reprocessing is through a JPEG encoder
to implement ZSL so I'm not sure if it applies 100%, but it might be
worth exploring if instead of an additional virtual 'output' stream
from the Camera we could be represent a debayer as an 'input' stream
to the Camera.

Can I ask how you have envisioned your capture pipeline ? I
assume you have a raw sensor connected to a pipeline handler that has
no ISP. What component would be driving the adjustment of the sensor
parameters in such an architecture ?

Thanks
   j

>
> Cheers,
> Dorota


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 833 bytes
Desc: not available
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20211110/b326649b/attachment.sig>