[libcamera-devel] [PATCH v4 0/3] qcam: NV family YUV format conversion by using OpenGL shader

[Show Liu]

Hi Laurent,

Thanks for your review.



On Mon, Aug 24, 2020 at 8:11 AM Laurent Pinchart <
laurent.pinchart at ideasonboard.com> wrote:

> Hi Show,
>
> On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote:
> > Hi,
> >
> > This is the version 4 patchset for qcam to accelerate the YUV format
> conversion
> > by OpenGL shader. Most of the modifications are according to the comments
> > and suggestions from V3.
> > And the big change is using the QOpenGLFramebufferObject with
> QOffscreenSurface
> > instead of the QOpenGLWidget, the main reason that use dynamic_cast
> > to get the viewfinder_ is really weird.
> > In this version, OpenGL rendering enabled by --render=gles,
> > and if any error occurs during OpenGL configuration, it will return to
> use QT rendering.
> > But compared with the previous version, this version performance dropped
> a lot
> > due to I using the toImage() function to get the image when rendering
> complete,
> > it's really an inefficient operation. the framerate down to 16.x ~ 18.x
> fps(1280 x 800).
> > I am still trying to improve this performance issue. Any suggestions are
> welcome.
>
> Isn't it expected though if you render to an off-screen surface and then
> convert it to a QImage ? What was the issue with rendering directly on
> the screen with QOpenGLWidget ? Just the dynamic cast ?
>

Some other reasons are like:
1. The viewfinder and viewfinderGL are very similar, too many parts are
duplicate.
2. My idea is that if viewfinder is able to handle both Qt and OpenGL
rendering would be perfect.
    The original processing flow when viewfinder got a new frame from
camera is like below
          Frame -> rendering -> call update() -> paintEvent( painting)
    But in QOpenGLWidget way, the processing flow become
          Frame -> call update() -> rendering and painting in paintGL()
    That's why I try to render into off-screen to fit the original
viewfinder process flow.

BUT honestly it seems not a good idea to render to an external off screen
surface
then convert to QImage due to the performance issue. I am trying to find
some way
to paint the QOpenGLFramebufferObject directly.

OR just letting the viewfinder inherit from QOpenGLWidget
and QOpenGLFunctions
and make it able to handle Qt and OpenGL rendering both.
I am still trying to figure out a good solution for that.



Best Regards,
Show Liu

> > Show Liu (3):
> >   qcam: add OpenGL shader code as QT resource
> >   qcam: add OpenGL renderer
> >   qcam: use the OpenGL renderer as NV family YUV format converter
> >
> >  src/qcam/assets/shader/NV_2_planes_UV_f.glsl |  32 ++
> >  src/qcam/assets/shader/NV_2_planes_VU_f.glsl |  32 ++
> >  src/qcam/assets/shader/NV_3_planes_UV_f.glsl |  33 ++
> >  src/qcam/assets/shader/NV_3_planes_VU_f.glsl |  33 ++
> >  src/qcam/assets/shader/NV_vertex_shader.glsl |  16 +
> >  src/qcam/assets/shader/shaders.qrc           |  10 +
> >  src/qcam/main.cpp                            |   3 +
> >  src/qcam/main_window.cpp                     |   2 +
> >  src/qcam/main_window.h                       |   1 +
> >  src/qcam/meson.build                         |   3 +
> >  src/qcam/renderer.cpp                        | 346 +++++++++++++++++++
> >  src/qcam/renderer.h                          |  81 +++++
> >  src/qcam/viewfinder.cpp                      |  46 ++-
> >  src/qcam/viewfinder.h                        |  10 +
> >  14 files changed, 640 insertions(+), 8 deletions(-)
> >  create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> >  create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> >  create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> >  create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> >  create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl
> >  create mode 100644 src/qcam/assets/shader/shaders.qrc
> >  create mode 100644 src/qcam/renderer.cpp
> >  create mode 100644 src/qcam/renderer.h
>
> --
> Regards,
>
> Laurent Pinchart
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20200825/bc556292/attachment-0001.htm>