[libcamera-devel] [GSoC 2021]OpenGL/OpenCL software ISP Project

[paul.elder at ideasonboard.com]

Hello Siyuan,

On Wed, Mar 24, 2021 at 09:28:42PM +0800, 范思远 wrote:
> Hello paul,
>     Sorry to bother you. I have some ideas/questions about the project and want to talk to you.
>     First, I choose the rpi 3B+ and have finished to configure the
>     libcamera and OpenCL framework in rpi. I try to run the qcam and
>     obtained the RAW image. It's so cool. I also finished the code
>     with C standard library to read the RAW image caputured by ov5647.

Ooh, nice!

I was going to add that OpenGL compute shaders are also an option, but
if you managed to get it to work with OpenCL already then that's great!

>     Maybe it is convenient to transplant. In my view, I need to use
>     the cam/qcam interface to complete the Isp pipeline, right? I have
>     a question about it.

cam and qcam are applications, which use libcamera. libcamera exposes
Camera objects for applications to interact with, so that the
applications don't have to deal with devices directly (that's the whole
point of libcamera's existence). So I don't think the correct route is
to make cam/qcam interface with the software ISP.

I think it's better to implement the software ISP as a device that a
pipeline handler could use. That being said, designing an interface for
/any/ pipeline handler to determine if they should use the software ISP or
the hardware ISP is probably out of scope for this project. I think it
would be enough to make the raspberrypi pipeline handler use the
software ISP, or add it into the simple pipeline handler, or even make a
new pipeline handler just for the software ISP. Though the first option
might be too complex and the last option might be a lot for the project.
Explore how pipeline handlers in libcamera work and maybe you'll get an
idea of which direction you'd want to go. Here's some documentation [1]
on how to write a pipeline handler; maybe it'll provide some
inspiration (other than reading existing pipeline handlers).

>     Second, I want to implement the software that allows easy access
>     to each stage of the camera imaging pipeline. Users can modify the
>     individual components to evaluating the final camera output.I
>     think it is very beneficial for both researchers and developers.

Ooh yes, that's a good idea. It's quite a bit more than what I had in
mind for the project though, so just make sure you plan it out well,
both the method and the timeline.

On second thought, maybe a new pipeline handler just for your software
ISP might be best, especially if you want to make it modular. (I'm just
thinking out loud; perhaps it'll become food for thought :) )

>     Finally, after a simple test, I found it will be better combine
>     OpenCL and Neon in rpi. So I will apply both methods. 

Great!

>     Could you give me some advice? 

I think you're heading in a good direction.


Thanks,

Paul