[libcamera-devel] [PATCH v4 1/3] qcam: add OpenGL shader code as QT resource
Show Liu
show.liu at linaro.org
Wed Aug 26 08:34:45 CEST 2020
Hi Laurent,
On Wed, Aug 26, 2020 at 3:37 AM Laurent Pinchart <
laurent.pinchart at ideasonboard.com> wrote:
> Hi Show,
>
> On Tue, Aug 25, 2020 at 05:38:55PM +0800, Show Liu wrote:
> > On Mon, Aug 24, 2020 at 8:07 AM Laurent Pinchart wrote:
> > > On Mon, Aug 24, 2020 at 03:04:56AM +0300, Laurent Pinchart wrote:
> > > > On Sat, Aug 22, 2020 at 12:16:00AM +0800, Show Liu wrote:
> > > > > qcam: add OpenGL shader code as QT resource
> > > >
> > > > s/QT/Qt/
> > > >
> > > > > Signed-off-by: Show Liu <show.liu at linaro.org>
> > > > > ---
> > > > > src/qcam/assets/shader/NV_2_planes_UV_f.glsl | 32
> +++++++++++++++++++
> > > > > src/qcam/assets/shader/NV_2_planes_VU_f.glsl | 32
> +++++++++++++++++++
> > > > > src/qcam/assets/shader/NV_3_planes_UV_f.glsl | 33
> ++++++++++++++++++++
> > > > > src/qcam/assets/shader/NV_3_planes_VU_f.glsl | 33
> ++++++++++++++++++++
> > > > > src/qcam/assets/shader/NV_vertex_shader.glsl | 16 ++++++++++
> > > > > src/qcam/assets/shader/shaders.qrc | 10 ++++++
> > > > > src/qcam/meson.build | 1 +
> > > > > 7 files changed, 157 insertions(+)
> > > > > create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> > > > > create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> > > > > create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> > > > > create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> > > > > create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl
> > > > > create mode 100644 src/qcam/assets/shader/shaders.qrc
> > > > >
> > > > > diff --git a/src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> b/src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> > > > > new file mode 100644
> > > > > index 0000000..32c6e90
> > > > > --- /dev/null
> > > > > +++ b/src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> > > > > @@ -0,0 +1,32 @@
> > > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */
> > > > > +/*
> > > > > + * Copyright (C) 2020, Linaro
> > > > > + *
> > > > > + * NV_2_planes_UV_f.glsl - Fragment shader code for NV12, NV16
> and NV24 formats
> > > > > + */
> > > > > +
> > > > > +#ifdef GL_ES
> > > > > +precision highp float;
> > > > > +#endif
> > > > > +
> > > > > +varying vec2 textureOut;
> > > > > +uniform sampler2D tex_y;
> > > > > +uniform sampler2D tex_u;
> > > > > +
> > > > > +void main(void)
> > > > > +{
> > > > > + vec3 yuv;
> > > > > + vec3 rgb;
> > > > > + mat3 convert_mat = mat3(
> > > >
> > > > Maybe yuv2rgb or yuv2rgb_mat instead of convert_mat ? Or
> yuv2rgb_bt601
> > > > or yuv2rgb_bt601_mat ?
> >
> > sure, I will fix it.
> >
> > > > > +
> vec3(1.1640625, 1.1640625, 1.1640625),
> > > > > + vec3(0.0,
> -0.390625, 2.015625),
> > > > > +
> vec3(1.5975625, -0.8125, 0.0)
> > > >
> > > > Where are the coefficients from ?
> https://en.wikipedia.org/wiki/YCbCr
> > > > lists values very similar but slightly different. Are the values here
> > > > rounded to a multiple of 1/256 on purpose (except for 1.5975625) ?
> >
> > The coefficients matrix refer from the yuv_to_rgb function in
> > format_converter.cpp file
> > 298/256=1.1640625
> > I just conert the elements into floating by dividing 256.
>
> Of course. I should have looked there :-) That code uses fixed-point
> calculations to speed things up. As we can use floating point values
> here, let's specify the exact values where possible.
>
> Can we use arithemtic expression to let the shader compiler pick the
> highest precision ?
>
> vec3(255.0/219.0, 255.0/219.0, 255.0/219.0),
> vec3(0.0, -255.0/224.0*1.772*0.114/0.587,
> 255.0/224.0*1.772),
> vec3(255.0/224.0*1.402, -255.0/224.0*1.402*0.299/0.587,
> 0.0)
>
> or should we use numerical values directly ? I suppose a 8-bit precision
> will be enough in practice as that's what we have in the input image and
> what we will render, so rounding after 3 decimal points should be fine
> and would probably be more readable.
>
> vec3(1.164, 1.164, 1.164),
> vec3(0.000, -0.392, 2.017),
> vec3(1.596, -0.813, 0.000)
>
> What do you think ?
>
Sure. I will try to apply it on the next version.
Thanks,
Show
>
> > > > > + );
> > > >
> > > > The indentation is a bit weird.
> > > >
> > > > > +
> > > > > + yuv.x = texture2D(tex_y, textureOut).r - 0.0625;
> > > >
> > > > Is 0.0625 (16/256) the right value, or should it be 16/255 ? The Y
> range
> > > > on 8-bit is 0-255, which is mapped to 0.0-1.0, right ?
> > > >
> > > > > + yuv.y = texture2D(tex_u, textureOut).r - 0.5;
> > > > > + yuv.z = texture2D(tex_u, textureOut).g - 0.5;
> > > > > +
> > > > > + rgb = convert_mat * yuv;
> > > > > + gl_FragColor = vec4(rgb, 1.0);
> > > > > +}
> > > > > diff --git a/src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> b/src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> > > > > new file mode 100644
> > > > > index 0000000..aae12de
> > > > > --- /dev/null
> > > > > +++ b/src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> > > > > @@ -0,0 +1,32 @@
> > > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */
> > > > > +/*
> > > > > + * Copyright (C) 2020, Linaro
> > > > > + *
> > > > > + * NV_2_planes_VU_f.glsl - Fragment shader code for NV21, NV61
> and NV42 formats
> > > > > + */
> > > > > +
> > > > > +#ifdef GL_ES
> > > > > +precision highp float;
> > > > > +#endif
> > > > > +
> > > > > +varying vec2 textureOut;
> > > > > +uniform sampler2D tex_y;
> > > > > +uniform sampler2D tex_u;
> > > > > +
> > > > > +void main(void)
> > > > > +{
> > > > > + vec3 yuv;
> > > > > + vec3 rgb;
> > > > > + mat3 convert_mat = mat3(
> > > > > +
> vec3(1.1640625, 1.1640625, 1.1640625),
> > > > > + vec3(0.0,
> -0.390625, 2.015625),
> > > > > +
> vec3(1.5975625, -0.8125, 0.0)
> > > > > + );
> > > >
> > > > Here too.
> > > >
> > > > > +
> > > > > + yuv.x = texture2D(tex_y, textureOut).r - 0.0625;
> > > > > + yuv.y = texture2D(tex_u, textureOut).g - 0.5;
> > > > > + yuv.z = texture2D(tex_u, textureOut).r - 0.5;
> > > > > +
> > > >
> > > > And there are some white spaces at the end of lines (below too).
> >
> > ok. I will remove it.
> >
> > > >
> > > > We'll have to make the coefficients configurable to support different
> > > > colorspaces, but that's for later.
> > > >
> > > > > + rgb = convert_mat * yuv;
> > > > > + gl_FragColor = vec4(rgb, 1.0);
> > > > > +}
> > > > > diff --git a/src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> b/src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> > > > > new file mode 100644
> > > > > index 0000000..21fff3a
> > > > > --- /dev/null
> > > > > +++ b/src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> > > > > @@ -0,0 +1,33 @@
> > > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */
> > > > > +/*
> > > > > + * Copyright (C) 2020, Linaro
> > > > > + *
> > > > > + * NV_3_planes_UV_f.glsl - Fragment shader code for YUV420 format
> > > > > + */
> > > > > +
> > > > > +#ifdef GL_ES
> > > > > +precision highp float;
> > > > > +#endif
> > > > > +
> > > > > +varying vec2 textureOut;
> > > > > +uniform sampler2D tex_y;
> > > > > +uniform sampler2D tex_u;
> > > > > +uniform sampler2D tex_v;
> > > > > +
> > > > > +void main(void)
> > > > > +{
> > > > > + vec3 yuv;
> > > > > + vec3 rgb;
> > > > > + mat3 convert_mat = mat3(
> > > > > +
> vec3(1.1640625, 1.1640625, 1.1640625),
> > > > > + vec3(0.0,
> -0.390625, 2.015625),
> > > > > +
> vec3(1.5975625, -0.8125, 0.0)
> > > > > + );
> > > > > +
> > > > > + yuv.x = texture2D(tex_y, textureOut).r - 0.0625;
> > > > > + yuv.y = texture2D(tex_u, textureOut).r - 0.5;
> > > > > + yuv.z = texture2D(tex_v, textureOut).g - 0.5;
> > > >
> > > > I'm not sure to understand the .g here, with YUV420 having three
> planes,
> > > > shouldn't it be .r ?
> >
> > Yes. you are right. it's a problem here. I will fix it.
> >
> > > I could be wrong, my GLSL knowledge is very very rudimentary :-)
> >
> > You definitely are a GLSL expert...don't cheat me, I know that. :-)
> >
> > > > > +
> > > > > + rgb = convert_mat * yuv;
> > > > > + gl_FragColor = vec4(rgb, 1.0);
> > > > > +}
> > > > > diff --git a/src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> b/src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> > > > > new file mode 100644
> > > > > index 0000000..df00170
> > > > > --- /dev/null
> > > > > +++ b/src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> > > > > @@ -0,0 +1,33 @@
> > > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */
> > > > > +/*
> > > > > + * Copyright (C) 2020, Linaro
> > > > > + *
> > > > > + * NV_3_planes_VU_f.glsl - Fragment shader code for YVU420 format
> > > > > + */
> > > > > +
> > > > > +#ifdef GL_ES
> > > > > +precision highp float;
> > > > > +#endif
> > > > > +
> > > > > +varying vec2 textureOut;
> > > > > +uniform sampler2D tex_y;
> > > > > +uniform sampler2D tex_u;
> > > > > +uniform sampler2D tex_v;
> > > > > +
> > > > > +void main(void)
> > > > > +{
> > > > > + vec3 yuv;
> > > > > + vec3 rgb;
> > > > > + mat3 convert_mat = mat3(
> > > > > +
> vec3(1.1640625, 1.1640625, 1.1640625),
> > > > > + vec3(0.0,
> -0.390625, 2.015625),
> > > > > +
> vec3(1.5975625, -0.8125, 0.0)
> > > > > + );
> > > > > +
> > > > > + yuv.x = texture2D(tex_y, textureOut).r - 0.0625;
> > > > > + yuv.y = texture2D(tex_u, textureOut).g - 0.5;
> > > > > + yuv.z = texture2D(tex_v, textureOut).r - 0.5;
> > > >
> > > > Same here, I was expecting
> >
> > will fix it.
> >
> > > > yuv.x = texture2D(tex_y, textureOut).r - 0.0625;
> > > > yuv.y = texture2D(tex_v, textureOut).r - 0.5;
> > > > yuv.z = texture2D(tex_u, textureOut).r - 0.5;
> > > >
> > > > > +
> > > > > + rgb = convert_mat * yuv;
> > > > > + gl_FragColor = vec4(rgb, 1.0);
> > > > > +}
> > > > > diff --git a/src/qcam/assets/shader/NV_vertex_shader.glsl
> > > b/src/qcam/assets/shader/NV_vertex_shader.glsl
> > > > > new file mode 100644
> > > > > index 0000000..403b791
> > > > > --- /dev/null
> > > > > +++ b/src/qcam/assets/shader/NV_vertex_shader.glsl
> > > > > @@ -0,0 +1,16 @@
> > > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */
> > > > > +/*
> > > > > + * Copyright (C) 2020, Linaro
> > > > > + *
> > > > > + * NV_vertex_shader.glsl - Vertex shader code for NV family
> > > > > + */
> > > > > +
> > > > > +attribute vec4 vertexIn;
> > > > > +attribute vec2 textureIn;
> > > > > +varying vec2 textureOut;
> > > > > +
> > > > > +void main(void)
> > > > > +{
> > > > > + gl_Position = vertexIn;
> > > > > + textureOut = textureIn;
> > > > > +}
> > > > > diff --git a/src/qcam/assets/shader/shaders.qrc
> > > b/src/qcam/assets/shader/shaders.qrc
> > > > > new file mode 100644
> > > > > index 0000000..6fe4c7f
> > > > > --- /dev/null
> > > > > +++ b/src/qcam/assets/shader/shaders.qrc
> > > > > @@ -0,0 +1,10 @@
> > > > > +<!-- SPDX-License-Identifier: GPL-2.0-or-later -->
> > > > > +<!DOCTYPE RCC><RCC version="1.0">
> > > > > +<qresource>
> > > > > +<file>./NV_vertex_shader.glsl</file>
> > > > > +<file>./NV_2_planes_UV_f.glsl</file>
> > > > > +<file>./NV_2_planes_VU_f.glsl</file>
> > > > > +<file>./NV_3_planes_UV_f.glsl</file>
> > > > > +<file>./NV_3_planes_VU_f.glsl</file>
> > > > > +</qresource>
> > > > > +</RCC>
> > > > > diff --git a/src/qcam/meson.build b/src/qcam/meson.build
> > > > > index 6ea886a..e0c6f26 100644
> > > > > --- a/src/qcam/meson.build
> > > > > +++ b/src/qcam/meson.build
> > > > > @@ -16,6 +16,7 @@ qcam_moc_headers = files([
> > > > >
> > > > > qcam_resources = files([
> > > > > 'assets/feathericons/feathericons.qrc',
> > > > > + 'assets/shader/shaders.qrc'
> > > > > ])
> > > > >
> > > > > qt5 = import('qt5')
>
> --
> Regards,
>
> Laurent Pinchart
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.libcamera.org/pipermail/libcamera-devel/attachments/20200826/91072346/attachment-0001.htm>
More information about the libcamera-devel
mailing list