[libcamera-devel] [PATCH v4 0/3] qcam: NV family YUV format conversion by using OpenGL shader

[Show Liu]

Hi Laurent,


On Wed, Aug 26, 2020 at 3:06 AM Laurent Pinchart <
laurent.pinchart at ideasonboard.com> wrote:

> Hi Show,
>
> On Tue, Aug 25, 2020 at 05:21:38PM +0800, Show Liu wrote:
> > Hi Laurent,
> >
> > Thanks for your review.
> >
> > On Mon, Aug 24, 2020 at 8:11 AM Laurent Pinchart wrote:
> > > On Sat, Aug 22, 2020 at 12:15:59AM +0800, Show Liu wrote:
> > > > Hi,
> > > >
> > > > This is the version 4 patchset for qcam to accelerate the YUV format
> conversion
> > > > by OpenGL shader. Most of the modifications are according to the
> comments
> > > > and suggestions from V3.
> > > > And the big change is using the QOpenGLFramebufferObject with
> QOffscreenSurface
> > > > instead of the QOpenGLWidget, the main reason that use dynamic_cast
> > > > to get the viewfinder_ is really weird.
> > > > In this version, OpenGL rendering enabled by --render=gles,
> > > > and if any error occurs during OpenGL configuration, it will return
> to use QT rendering.
> > > > But compared with the previous version, this version performance
> dropped a lot
> > > > due to I using the toImage() function to get the image when
> rendering complete,
> > > > it's really an inefficient operation. the framerate down to 16.x ~
> 18.x fps(1280 x 800).
> > > > I am still trying to improve this performance issue. Any suggestions
> are welcome.
> > >
> > > Isn't it expected though if you render to an off-screen surface and
> then
> > > convert it to a QImage ? What was the issue with rendering directly on
> > > the screen with QOpenGLWidget ? Just the dynamic cast ?
> >
> > Some other reasons are like:
> > 1. The viewfinder and viewfinderGL are very similar, too many parts are
> > duplicate.
> > 2. My idea is that if viewfinder is able to handle both Qt and OpenGL
> > rendering would be perfect.
> >     The original processing flow when viewfinder got a new frame from
> > camera is like below
> >           Frame -> rendering -> call update() -> paintEvent( painting)
> >     But in QOpenGLWidget way, the processing flow become
> >           Frame -> call update() -> rendering and painting in paintGL()
> >     That's why I try to render into off-screen to fit the original
> > viewfinder process flow.
> >
> > BUT honestly it seems not a good idea to render to an external off
> screen surface
> > then convert to QImage due to the performance issue. I am trying to find
> some way
> > to paint the QOpenGLFramebufferObject directly.
> >
> > OR just letting the viewfinder inherit from QOpenGLWidgetand
> QOpenGLFunctions
> > and make it able to handle Qt and OpenGL rendering both.
> > I am still trying to figure out a good solution for that.
>
> How about starting with a ViewfinderGL implementation that duplicates
> code from the existing Viewfinder class, to get the feature merged, and
> then reworking both classes on top to reduce code duplication ? I can
> help with the latter.
>

Sounds Great.
I will have the next version based on viewfinderGL ASAP.

Best Regards,
Show Liu

>
> > > > Show Liu (3):
> > > >   qcam: add OpenGL shader code as QT resource
> > > >   qcam: add OpenGL renderer
> > > >   qcam: use the OpenGL renderer as NV family YUV format converter
> > > >
> > > >  src/qcam/assets/shader/NV_2_planes_UV_f.glsl |  32 ++
> > > >  src/qcam/assets/shader/NV_2_planes_VU_f.glsl |  32 ++
> > > >  src/qcam/assets/shader/NV_3_planes_UV_f.glsl |  33 ++
> > > >  src/qcam/assets/shader/NV_3_planes_VU_f.glsl |  33 ++
> > > >  src/qcam/assets/shader/NV_vertex_shader.glsl |  16 +
> > > >  src/qcam/assets/shader/shaders.qrc           |  10 +
> > > >  src/qcam/main.cpp                            |   3 +
> > > >  src/qcam/main_window.cpp                     |   2 +
> > > >  src/qcam/main_window.h                       |   1 +
> > > >  src/qcam/meson.build                         |   3 +
> > > >  src/qcam/renderer.cpp                        | 346
> +++++++++++++++++++
> > > >  src/qcam/renderer.h                          |  81 +++++
> > > >  src/qcam/viewfinder.cpp                      |  46 ++-
> > > >  src/qcam/viewfinder.h                        |  10 +
> > > >  14 files changed, 640 insertions(+), 8 deletions(-)
> > > >  create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> > > >  create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> > > >  create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> > > >  create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> > > >  create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl
> > > >  create mode 100644 src/qcam/assets/shader/shaders.qrc
> > > >  create mode 100644 src/qcam/renderer.cpp
> > > >  create mode 100644 src/qcam/renderer.h
>
> --
> Regards,
>
> Laurent Pinchart
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20200826/8434f1e2/attachment-0001.htm>