[libcamera-devel] [PATCH v4 0/3] qcam: NV family YUV format conversion by using OpenGL shader

[Laurent Pinchart]

Hi Show,

On Tue, Aug 25, 2020 at 05:21:38PM +0800, Show Liu wrote:
> Hi Laurent,
> 
> Thanks for your review.
> 
> On Mon, Aug 24, 2020 at 8:11 AM Laurent Pinchart wrote:
> > On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote:
> > > Hi,
> > >
> > > This is the version 4 patchset for qcam to accelerate the YUV format conversion
> > > by OpenGL shader. Most of the modifications are according to the comments
> > > and suggestions from V3.
> > > And the big change is using the QOpenGLFramebufferObject with QOffscreenSurface
> > > instead of the QOpenGLWidget, the main reason that use dynamic_cast
> > > to get the viewfinder_ is really weird.
> > > In this version, OpenGL rendering enabled by --render=gles,
> > > and if any error occurs during OpenGL configuration, it will return to use QT rendering.
> > > But compared with the previous version, this version performance dropped a lot
> > > due to I using the toImage() function to get the image when rendering complete,
> > > it's really an inefficient operation. the framerate down to 16.x ~ 18.x fps(1280 x 800).
> > > I am still trying to improve this performance issue. Any suggestions are welcome.
> >
> > Isn't it expected though if you render to an off-screen surface and then
> > convert it to a QImage ? What was the issue with rendering directly on
> > the screen with QOpenGLWidget ? Just the dynamic cast ?
> 
> Some other reasons are like:
> 1. The viewfinder and viewfinderGL are very similar, too many parts are
> duplicate.
> 2. My idea is that if viewfinder is able to handle both Qt and OpenGL
> rendering would be perfect.
>     The original processing flow when viewfinder got a new frame from
> camera is like below
>           Frame -> rendering -> call update() -> paintEvent( painting)
>     But in QOpenGLWidget way, the processing flow become
>           Frame -> call update() -> rendering and painting in paintGL()
>     That's why I try to render into off-screen to fit the original
> viewfinder process flow.
> 
> BUT honestly it seems not a good idea to render to an external off screen surface
> then convert to QImage due to the performance issue. I am trying to find some way
> to paint the QOpenGLFramebufferObject directly.
> 
> OR just letting the viewfinder inherit from QOpenGLWidgetand QOpenGLFunctions
> and make it able to handle Qt and OpenGL rendering both.
> I am still trying to figure out a good solution for that.

How about starting with a ViewfinderGL implementation that duplicates
code from the existing Viewfinder class, to get the feature merged, and
then reworking both classes on top to reduce code duplication ? I can
help with the latter.

> > > Show Liu (3):
> > >   qcam: add OpenGL shader code as QT resource
> > >   qcam: add OpenGL renderer
> > >   qcam: use the OpenGL renderer as NV family YUV format converter
> > >
> > >  src/qcam/assets/shader/NV_2_planes_UV_f.glsl |  32 ++
> > >  src/qcam/assets/shader/NV_2_planes_VU_f.glsl |  32 ++
> > >  src/qcam/assets/shader/NV_3_planes_UV_f.glsl |  33 ++
> > >  src/qcam/assets/shader/NV_3_planes_VU_f.glsl |  33 ++
> > >  src/qcam/assets/shader/NV_vertex_shader.glsl |  16 +
> > >  src/qcam/assets/shader/shaders.qrc           |  10 +
> > >  src/qcam/main.cpp                            |   3 +
> > >  src/qcam/main_window.cpp                     |   2 +
> > >  src/qcam/main_window.h                       |   1 +
> > >  src/qcam/meson.build                         |   3 +
> > >  src/qcam/renderer.cpp                        | 346 +++++++++++++++++++
> > >  src/qcam/renderer.h                          |  81 +++++
> > >  src/qcam/viewfinder.cpp                      |  46 ++-
> > >  src/qcam/viewfinder.h                        |  10 +
> > >  14 files changed, 640 insertions(+), 8 deletions(-)
> > >  create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> > >  create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> > >  create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> > >  create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> > >  create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl
> > >  create mode 100644 src/qcam/assets/shader/shaders.qrc
> > >  create mode 100644 src/qcam/renderer.cpp
> > >  create mode 100644 src/qcam/renderer.h

-- 
Regards,

Laurent Pinchart