[libcamera-devel] Debayering
Laurent Pinchart
laurent.pinchart at ideasonboard.com
Fri Nov 12 01:09:43 CET 2021
Hello,
On Thu, Nov 11, 2021 at 11:56:07AM +0000, Kieran Bingham wrote:
> Hi Jacopo, Dorota,
>
> + Added Siyuan and Paul to Cc directly
>
> Quoting Jacopo Mondi (2021-11-10 20:32:29)
> > Hi Dorota,
> >
> > On Wed, Nov 10, 2021 at 02:01:23PM +0100, Dorota Czaplejewicz wrote:
> > > Hi all,
> > >
> > > I'm making sure that Librem 5 cameras can be easily used via
> > > libcamera. One of the steps that are needed for the "easily" part is
> > > having a builtin debayering mechanism.
> > >
> > > Before I jump head-in, I wanted to ask for some guidance about where
> > > to place this support, and how to expose it to the consumer.
> > >
> > > For a simplest implementation, supported cameras could add the RGB
> > > format to the list of supported formats, and transparently convert
> > > the bayer data to RGB if configured for RGB. That opens up a
> > > question about the opt-in mechanism, but is otherwise simple enough.
>
> I think this should be 'opt in' only by the platform (not the 'camera'
> or 'sensor'). To me that means it's a pipeline handler decision as to
> whether it exposes a software/GPU ISP. (a CPU ISP should only be
> considered a shortcut to GPU accelleration later I believe).
>
> The Camera would only expose the outputs supported by the capabilities
> of the (Soft)ISP, just as on other platforms the Camera only exposes the
> features supported by the HW ISP.
>
> This can be handled by either the SimplePipelineInfo if we just use the
> simple pipeline handler, or if the platform is more complex because of
> necessitating a SoftISP then a dedicated pipeline handler (where the
> SoftISP is just an internal re-usable component).
Kieran is reading my mind :-)
> > > But it comes with a downside: the raw bayer frames are lost, and the
> > > setting cannot be changed at runtime, even though it's a pure
> > > software operation. To overcome that, some other API than
> > > configuration would have to be used.
> > >
> > > I intend to start by adding a CPU-based debayering implementation
> > > for the sake of building it up quickly. Would a simple
> > > implementation like I outlined above suitable for upstreaming?
>
> That would depend on how it's architectured. I don't think we'd like
> 'quick hacks' in place internally to be merged.
>
> > For the record, this seems potentially similar to other 'plugins' like
> > the software JPEG encoder we have in the HAL. It would be lovely to
> > find a proper place where to place them :)
>
> It is certainly similar, but I think there's a key difference that it is
> more of a somewhat required functionality for a (complex) camera
> pipeline.
>
> Otherwise we may as well tell applications to just read the raw output
> from the CSI2 receiver themselves ...
>
> > I used to think at software post-processors like these more like
> > external plugins to libcamera, something applications could easily pipe
> > their streams through instead of receiving processed data from an
> > additional software-generated stream from the Camera, but I would be
> > more than happy to have my mind changed.
> >
> > If I got your proposal right, the sw-debayer would be an opt-in
> > mechanism for pipeline handlers, which in the case they cannot perform
> > debayering through an ISP would create an additional 'virtual' stream
> > that application can select transparently from the Camera. The ideal
> > target would be the simple pipeline handler, as it is meant to support
> > platforms without an ISP.
>
> Agreed, the Simple Pipeline Handler could probably wrap most of the
> work generically with a SoftISP/SoftIPA.
>
> You call it a 'virtual stream' though. Are the outputs of the IPU3 IMGU
> 'virtual'? Are these different?
>
> > The first concern I have for such a design is about the processing
> > time software debayering could take as, compared to JPEG which is
> > triggered just for still capture requests, the debayering would need to
> > be applied to every frame. Is this correct ?
>
> Yes, I can't imagine this not being for every frame.
>
> Note that even JPEG could be for every frame to support MJPEG streams
> which are found on webcams ...
> (but I'm not saying MJPEG should be part of this).
>
> > I cannot quantify how long it could take (and I guess it also depends
> > on the platform capabilities and the system load) but it doesn't seems
> > impossible that it could create stalls in the pipeline, requiring the
> > pipeline handler to drop frames and reduce the frame rate. Do you have
> > any idea about the possible numbers here ?
>
> It would certainly be lower performance if the implementation uses the
> CPU. This could later be abstracted to use the GPU when available too,
> internally in the SoftISP component.
>
> Any low-performance CPU option IMO would just be a stepping stone to
> getting to a hardware accelerated (GPU) version.
>
> > I'm also a bit skeptical that implementing opt-in at pipeline handler
> > level is the right decision. As I understood it, such an opt-in
> > mechanism should be driven by applications, not be platform specific
> > even if I understand that for the simple pipeline handler this looks
> > like something desirable for all platforms.
>
> I disagree here. I don't think this should be driven by applications.
>
> Here's what we currently have...
>
> ┌─────────────────────────────────────────────┐ ┌────────────────────┐
> │ libcamera │ │ │
> │ ┌─────────────────────────────────────────┐ │ │ Camera Application │
> │ │ Simple Pipeline Handler │ │ │ │
> │ │ ┌────────┐ ┌───────────┐ │ │ │ │
> │ │ │ │ │ CSI2 │ │ │ │ │
> │ │ │ Sensor ├───►│ Receiver ├───────────┼─┼──┼─► │
> │ │ │ │ │ │ │ │ │ │
> │ │ └────────┘ └───────────┘ │ │ │ │
> │ │ │ │ │ │
> │ └─────────────────────────────────────────┘ │ │ │
> │ │ │ │
> └─────────────────────────────────────────────┘ └────────────────────┘
>
> Although it has an 'optional' SimpleConvertor so it's also this:
>
> ┌─────────────────────────────────────────────┐ ┌────────────────────┐
> │ libcamera │ │ │
> │ ┌─────────────────────────────────────────┐ │ │ Camera Application │
> │ │ Simple Pipeline Handler │ │ │ │
> │ │ ┌────────┐ ┌───────────┐ ┌────────┐ │ │ │ │
> │ │ │ │ │ CSI2 │ │Simple │ │ │ │ │
> │ │ │ Sensor ├─►│ Reciever ├─►│ ├─┼─┼──┼──► │
> │ │ │ │ │ │ │Convert │ │ │ │ │
> │ │ └────────┘ └───────────┘ └────────┘ │ │ │ │
> │ │ │ │ │ │
> │ └─────────────────────────────────────────┘ │ │ │
> │ │ │ │
> └─────────────────────────────────────────────┘ └────────────────────┘
>
> And for platforms that have a raw sensor if this isn't handled by the
> pipeline handler we would end up with this:
>
> ┌─────────────────────────────────────────────┐ ┌────────────────────┐
> │ libcamera │ │ │
> │ ┌─────────────────────────────────────────┐ │ │ Camera Application │
> │ │ │ │ │ │
> │ │ Simple Pipeline Handler │ │ │ │
> │ │ ┌────────┐ ┌───────────┐ │ │ │ ┌──────────┐ │
> │ │ │ │ │ CSI2 │ │ │ │ │ 3a │ │
> │ │ │ Sensor ├───►│ Receiver ├───────────┼─┼──┼─► │ debayer │ │
> │ │ │ │ │ │ │ │ │ │ │ │
> │ │ └───▲────┘ └───────────┘ │ │ │ └─────┬────┘ │
> │ │ │ │ │ │ │ │
> │ │ └─────────────────────────────────┼─┼──┼─────────┘ │
> │ │ │ │ │ │
> │ └─────────────────────────────────────────┘ │ │ │
> │ │ │ │
> └─────────────────────────────────────────────┘ └────────────────────┘
>
> (even if maybe the 3a/debayer is some other library?)
>
> Does that match what you have suggested? (I'm not 100% sure)
>
> If you consider an application like cheese, (which granted uses our
> gstreamer element) It doesn't know how to debayer, and it doesn't know how
> to feedback to the sensor that "... oh ... it's a bit too bright, can
> you lower the gain..."
>
> I don't believe any layer to add that functionality should live between
> libcamera and the application. That is functionality that an application
> should /expect/ libcamera to be providing.
>
> For any platform with no ISP - but a CSI2 receiver and a raw/bayer
> sensor connected ... we want to do some processing. That means stats to
> get the autoexposure/gain, and debayering. It doesn't all have to be at
> once ... gain/exposure could be manual to start with for instance...
>
> But I really believe it needs to be this with the SoftISP block being
> similar in nature to the SimpleConvertor. A block that can be optionally
> used if the platform 'requires' it:
>
>
> ┌─────────────────────────────────────────────┐ ┌────────────────────┐
> │ libcamera │ │ │
> │ │ │ Camera Application │
> │ ┌─────────────────────────────────────────┐ │ │ │
> │ │ Simple-ish Pipeline Handler │ │ │ │
> │ │ ┌────────┐ ┌───────────┐ ┌───────┐ │ │ │ │
> │ │ │ │ │ CSI2 │ │ CPU │ │ │ │ │
> │ │ │ Sensor ├───►│ Receiver ├─►┤ ISP ├──┼─┼──┼─► │
> │ │ │ │ │ │ │ GPU │ │ │ │ │
> │ │ └────▲───┘ └───────────┘ └────┬──┘ │ │ │ │
> │ │ │ │ │ │ │ │
> │ └──────┼────────┬───────────┬───────┼─────┘ │ │ │
> │ │ ├───────────┤ │ │ │ │
> │ └────────┤ SimpleIPA │◄──────┘ │ │ │
> │ └───────────┘ │ │ │
> │ │ │ │
> └─────────────────────────────────────────────┘ └────────────────────┘
>
> Of course this can still pass through the raw stream directly to the
> application if that's what it really wants ... (just as we handle raw
> streams in RPi or .. IPU3?* )
That's how I was envisioning this. It should be up to the pipeline
handler how to build a pipeline from the building blocks available on
the platform. So far we only support dedicated hardware devices (ISPs
for IPU3, RkISP1 and RPi, and a V4L2 memory-to-memory format converter
and scaler for the simple pipeline handler). Some platforms that lack an
ISP but have a powerful enough GPU should be able to use a GPU-based
software ISP implementation, and a CPU-based reference implementation
could also be useful for development. Further in the future, I could
also imagine a platform with a hardware ISP using the GPU for additional
processing, before and/or after ISP processing.
Let's also note that some platforms (I'm thinking about some i.MX SoCs)
lack an ISP but can compute bayer statistics in hardware. The design of
the soft ISP should take this into account, to make software statistics
calculation optional when hardware statistics are available.
> *side note - do raw streams work on IPU3 currently?
>
> I'm weary about how/who/where the additional buffers required get
> allocated though.
>
> > There are other possible developments which might be worth a
> > comparison with.
> >
> > There is an implementation of a SW ISP on the list from Siyuan. My
> > understanding is that it should target format conversion too and not
> > just statistics handling. I've cc-ed Paul and Siyuan in case they want
> > to chime in.
>
> Added to cc.
>
> > Laurent is also working on an API to perform stream reprocessing. The
> > usual use case is to have one RAW frame as captured from the
> > camera 'reprocessed' through the capture pipeline. In example on
> > Android one of the most common reprocessing is through a JPEG encoder
> > to implement ZSL so I'm not sure if it applies 100%, but it might be
> > worth exploring if instead of an additional virtual 'output' stream
> > from the Camera we could be represent a debayer as an 'input' stream
> > to the Camera.
>
> I'm sure Laurent can comment here, and I'm happy to be proven wrong but
> I'm not sure reprocessing can fit here as it's not - reprocessing - it's
> just ... processing...
A reprocessing API could fit here in the sense that an application could
capture raw frames and send them back to the pipeline handler for
processing by the soft ISP, the same would it could do it on a platform
with a hardware ISP, but the base use case is capture of processed
frames without using a reprocessing API. That's what I think we need to
implement first.
> > Can I ask how you have envisioned your capture pipeline ? I
> > assume you have a raw sensor connected to a pipeline handler that has
> > no ISP. What component would be driving the adjustment of the sensor
> > parameters in such an architecture ?
>
> I believe that's the core design and architecture of libcamera, and
> therefore why something like the last diagram above is what should be
> considered as a starting point when designing this.
--
Regards,
Laurent Pinchart
More information about the libcamera-devel
mailing list