[libcamera-devel] [PATCH v4 1/3] qcam: add OpenGL shader code as QT resource
Laurent Pinchart
laurent.pinchart at ideasonboard.com
Tue Aug 25 21:37:22 CEST 2020
Hi Show,
On Tue, Aug 25, 2020 at 05:38:55PM +0800, Show Liu wrote:
> On Mon, Aug 24, 2020 at 8:07 AM Laurent Pinchart wrote:
> > On Mon, Aug 24, 2020 at 03:04:56AM +0300, Laurent Pinchart wrote:
> > > On Sat, Aug 22, 2020 at 12:16:00AM +0800, Show Liu wrote:
> > > > qcam: add OpenGL shader code as QT resource
> > >
> > > s/QT/Qt/
> > >
> > > > Signed-off-by: Show Liu <show.liu at linaro.org>
> > > > ---
> > > > src/qcam/assets/shader/NV_2_planes_UV_f.glsl | 32 +++++++++++++++++++
> > > > src/qcam/assets/shader/NV_2_planes_VU_f.glsl | 32 +++++++++++++++++++
> > > > src/qcam/assets/shader/NV_3_planes_UV_f.glsl | 33 ++++++++++++++++++++
> > > > src/qcam/assets/shader/NV_3_planes_VU_f.glsl | 33 ++++++++++++++++++++
> > > > src/qcam/assets/shader/NV_vertex_shader.glsl | 16 ++++++++++
> > > > src/qcam/assets/shader/shaders.qrc | 10 ++++++
> > > > src/qcam/meson.build | 1 +
> > > > 7 files changed, 157 insertions(+)
> > > > create mode 100644 src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> > > > create mode 100644 src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> > > > create mode 100644 src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> > > > create mode 100644 src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> > > > create mode 100644 src/qcam/assets/shader/NV_vertex_shader.glsl
> > > > create mode 100644 src/qcam/assets/shader/shaders.qrc
> > > >
> > > > diff --git a/src/qcam/assets/shader/NV_2_planes_UV_f.glsl b/src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> > > > new file mode 100644
> > > > index 0000000..32c6e90
> > > > --- /dev/null
> > > > +++ b/src/qcam/assets/shader/NV_2_planes_UV_f.glsl
> > > > @@ -0,0 +1,32 @@
> > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */
> > > > +/*
> > > > + * Copyright (C) 2020, Linaro
> > > > + *
> > > > + * NV_2_planes_UV_f.glsl - Fragment shader code for NV12, NV16 and NV24 formats
> > > > + */
> > > > +
> > > > +#ifdef GL_ES
> > > > +precision highp float;
> > > > +#endif
> > > > +
> > > > +varying vec2 textureOut;
> > > > +uniform sampler2D tex_y;
> > > > +uniform sampler2D tex_u;
> > > > +
> > > > +void main(void)
> > > > +{
> > > > + vec3 yuv;
> > > > + vec3 rgb;
> > > > + mat3 convert_mat = mat3(
> > >
> > > Maybe yuv2rgb or yuv2rgb_mat instead of convert_mat ? Or yuv2rgb_bt601
> > > or yuv2rgb_bt601_mat ?
>
> sure, I will fix it.
>
> > > > + vec3(1.1640625, 1.1640625, 1.1640625),
> > > > + vec3(0.0, -0.390625, 2.015625),
> > > > + vec3(1.5975625, -0.8125, 0.0)
> > >
> > > Where are the coefficients from ? https://en.wikipedia.org/wiki/YCbCr
> > > lists values very similar but slightly different. Are the values here
> > > rounded to a multiple of 1/256 on purpose (except for 1.5975625) ?
>
> The coefficients matrix refer from the yuv_to_rgb function in
> format_converter.cpp file
> 298/256=1.1640625
> I just conert the elements into floating by dividing 256.
Of course. I should have looked there :-) That code uses fixed-point
calculations to speed things up. As we can use floating point values
here, let's specify the exact values where possible.
Can we use arithemtic expression to let the shader compiler pick the
highest precision ?
vec3(255.0/219.0, 255.0/219.0, 255.0/219.0),
vec3(0.0, -255.0/224.0*1.772*0.114/0.587, 255.0/224.0*1.772),
vec3(255.0/224.0*1.402, -255.0/224.0*1.402*0.299/0.587, 0.0)
or should we use numerical values directly ? I suppose a 8-bit precision
will be enough in practice as that's what we have in the input image and
what we will render, so rounding after 3 decimal points should be fine
and would probably be more readable.
vec3(1.164, 1.164, 1.164),
vec3(0.000, -0.392, 2.017),
vec3(1.596, -0.813, 0.000)
What do you think ?
> > > > + );
> > >
> > > The indentation is a bit weird.
> > >
> > > > +
> > > > + yuv.x = texture2D(tex_y, textureOut).r - 0.0625;
> > >
> > > Is 0.0625 (16/256) the right value, or should it be 16/255 ? The Y range
> > > on 8-bit is 0-255, which is mapped to 0.0-1.0, right ?
> > >
> > > > + yuv.y = texture2D(tex_u, textureOut).r - 0.5;
> > > > + yuv.z = texture2D(tex_u, textureOut).g - 0.5;
> > > > +
> > > > + rgb = convert_mat * yuv;
> > > > + gl_FragColor = vec4(rgb, 1.0);
> > > > +}
> > > > diff --git a/src/qcam/assets/shader/NV_2_planes_VU_f.glsl b/src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> > > > new file mode 100644
> > > > index 0000000..aae12de
> > > > --- /dev/null
> > > > +++ b/src/qcam/assets/shader/NV_2_planes_VU_f.glsl
> > > > @@ -0,0 +1,32 @@
> > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */
> > > > +/*
> > > > + * Copyright (C) 2020, Linaro
> > > > + *
> > > > + * NV_2_planes_VU_f.glsl - Fragment shader code for NV21, NV61 and NV42 formats
> > > > + */
> > > > +
> > > > +#ifdef GL_ES
> > > > +precision highp float;
> > > > +#endif
> > > > +
> > > > +varying vec2 textureOut;
> > > > +uniform sampler2D tex_y;
> > > > +uniform sampler2D tex_u;
> > > > +
> > > > +void main(void)
> > > > +{
> > > > + vec3 yuv;
> > > > + vec3 rgb;
> > > > + mat3 convert_mat = mat3(
> > > > + vec3(1.1640625, 1.1640625, 1.1640625),
> > > > + vec3(0.0, -0.390625, 2.015625),
> > > > + vec3(1.5975625, -0.8125, 0.0)
> > > > + );
> > >
> > > Here too.
> > >
> > > > +
> > > > + yuv.x = texture2D(tex_y, textureOut).r - 0.0625;
> > > > + yuv.y = texture2D(tex_u, textureOut).g - 0.5;
> > > > + yuv.z = texture2D(tex_u, textureOut).r - 0.5;
> > > > +
> > >
> > > And there are some white spaces at the end of lines (below too).
>
> ok. I will remove it.
>
> > >
> > > We'll have to make the coefficients configurable to support different
> > > colorspaces, but that's for later.
> > >
> > > > + rgb = convert_mat * yuv;
> > > > + gl_FragColor = vec4(rgb, 1.0);
> > > > +}
> > > > diff --git a/src/qcam/assets/shader/NV_3_planes_UV_f.glsl b/src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> > > > new file mode 100644
> > > > index 0000000..21fff3a
> > > > --- /dev/null
> > > > +++ b/src/qcam/assets/shader/NV_3_planes_UV_f.glsl
> > > > @@ -0,0 +1,33 @@
> > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */
> > > > +/*
> > > > + * Copyright (C) 2020, Linaro
> > > > + *
> > > > + * NV_3_planes_UV_f.glsl - Fragment shader code for YUV420 format
> > > > + */
> > > > +
> > > > +#ifdef GL_ES
> > > > +precision highp float;
> > > > +#endif
> > > > +
> > > > +varying vec2 textureOut;
> > > > +uniform sampler2D tex_y;
> > > > +uniform sampler2D tex_u;
> > > > +uniform sampler2D tex_v;
> > > > +
> > > > +void main(void)
> > > > +{
> > > > + vec3 yuv;
> > > > + vec3 rgb;
> > > > + mat3 convert_mat = mat3(
> > > > + vec3(1.1640625, 1.1640625, 1.1640625),
> > > > + vec3(0.0, -0.390625, 2.015625),
> > > > + vec3(1.5975625, -0.8125, 0.0)
> > > > + );
> > > > +
> > > > + yuv.x = texture2D(tex_y, textureOut).r - 0.0625;
> > > > + yuv.y = texture2D(tex_u, textureOut).r - 0.5;
> > > > + yuv.z = texture2D(tex_v, textureOut).g - 0.5;
> > >
> > > I'm not sure to understand the .g here, with YUV420 having three planes,
> > > shouldn't it be .r ?
>
> Yes. you are right. it's a problem here. I will fix it.
>
> > I could be wrong, my GLSL knowledge is very very rudimentary :-)
>
> You definitely are a GLSL expert...don't cheat me, I know that. :-)
>
> > > > +
> > > > + rgb = convert_mat * yuv;
> > > > + gl_FragColor = vec4(rgb, 1.0);
> > > > +}
> > > > diff --git a/src/qcam/assets/shader/NV_3_planes_VU_f.glsl b/src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> > > > new file mode 100644
> > > > index 0000000..df00170
> > > > --- /dev/null
> > > > +++ b/src/qcam/assets/shader/NV_3_planes_VU_f.glsl
> > > > @@ -0,0 +1,33 @@
> > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */
> > > > +/*
> > > > + * Copyright (C) 2020, Linaro
> > > > + *
> > > > + * NV_3_planes_VU_f.glsl - Fragment shader code for YVU420 format
> > > > + */
> > > > +
> > > > +#ifdef GL_ES
> > > > +precision highp float;
> > > > +#endif
> > > > +
> > > > +varying vec2 textureOut;
> > > > +uniform sampler2D tex_y;
> > > > +uniform sampler2D tex_u;
> > > > +uniform sampler2D tex_v;
> > > > +
> > > > +void main(void)
> > > > +{
> > > > + vec3 yuv;
> > > > + vec3 rgb;
> > > > + mat3 convert_mat = mat3(
> > > > + vec3(1.1640625, 1.1640625, 1.1640625),
> > > > + vec3(0.0, -0.390625, 2.015625),
> > > > + vec3(1.5975625, -0.8125, 0.0)
> > > > + );
> > > > +
> > > > + yuv.x = texture2D(tex_y, textureOut).r - 0.0625;
> > > > + yuv.y = texture2D(tex_u, textureOut).g - 0.5;
> > > > + yuv.z = texture2D(tex_v, textureOut).r - 0.5;
> > >
> > > Same here, I was expecting
>
> will fix it.
>
> > > yuv.x = texture2D(tex_y, textureOut).r - 0.0625;
> > > yuv.y = texture2D(tex_v, textureOut).r - 0.5;
> > > yuv.z = texture2D(tex_u, textureOut).r - 0.5;
> > >
> > > > +
> > > > + rgb = convert_mat * yuv;
> > > > + gl_FragColor = vec4(rgb, 1.0);
> > > > +}
> > > > diff --git a/src/qcam/assets/shader/NV_vertex_shader.glsl
> > b/src/qcam/assets/shader/NV_vertex_shader.glsl
> > > > new file mode 100644
> > > > index 0000000..403b791
> > > > --- /dev/null
> > > > +++ b/src/qcam/assets/shader/NV_vertex_shader.glsl
> > > > @@ -0,0 +1,16 @@
> > > > +/* SPDX-License-Identifier: GPL-2.0-or-later */
> > > > +/*
> > > > + * Copyright (C) 2020, Linaro
> > > > + *
> > > > + * NV_vertex_shader.glsl - Vertex shader code for NV family
> > > > + */
> > > > +
> > > > +attribute vec4 vertexIn;
> > > > +attribute vec2 textureIn;
> > > > +varying vec2 textureOut;
> > > > +
> > > > +void main(void)
> > > > +{
> > > > + gl_Position = vertexIn;
> > > > + textureOut = textureIn;
> > > > +}
> > > > diff --git a/src/qcam/assets/shader/shaders.qrc
> > b/src/qcam/assets/shader/shaders.qrc
> > > > new file mode 100644
> > > > index 0000000..6fe4c7f
> > > > --- /dev/null
> > > > +++ b/src/qcam/assets/shader/shaders.qrc
> > > > @@ -0,0 +1,10 @@
> > > > +<!-- SPDX-License-Identifier: GPL-2.0-or-later -->
> > > > +<!DOCTYPE RCC><RCC version="1.0">
> > > > +<qresource>
> > > > +<file>./NV_vertex_shader.glsl</file>
> > > > +<file>./NV_2_planes_UV_f.glsl</file>
> > > > +<file>./NV_2_planes_VU_f.glsl</file>
> > > > +<file>./NV_3_planes_UV_f.glsl</file>
> > > > +<file>./NV_3_planes_VU_f.glsl</file>
> > > > +</qresource>
> > > > +</RCC>
> > > > diff --git a/src/qcam/meson.build b/src/qcam/meson.build
> > > > index 6ea886a..e0c6f26 100644
> > > > --- a/src/qcam/meson.build
> > > > +++ b/src/qcam/meson.build
> > > > @@ -16,6 +16,7 @@ qcam_moc_headers = files([
> > > >
> > > > qcam_resources = files([
> > > > 'assets/feathericons/feathericons.qrc',
> > > > + 'assets/shader/shaders.qrc'
> > > > ])
> > > >
> > > > qt5 = import('qt5')
--
Regards,
Laurent Pinchart
More information about the libcamera-devel
mailing list