<div dir="ltr">Hi Laurent and Umang,<br><br>> Hi Umang,<br>> The quantization of the Cb and Cr values in all relevant color spaces<br>> (ITU-R BT.601, BT.709, BT.2020, ...) add an offset of 128 (for 8-bit<br>> values). For instance, in BT.709, we have<br>><br>> D'Cb = INT[(224*E'Cb + 128)*2^(n-8)]<br>><br>> where D'Cb is the Cb signal after quantization, E'Cb the Cb signal<br>> before quantization (in the [-0.5, 0.5] range), and n the number of<br>> bits). INT[] denotes rounding to the closest integer.<br>><br>> The 224 multiplier creates a limited quantization range, following the<br>> above formula, -0.5 will be quantized to INT[224 * -0.5 + 128] = 16, and<br>> 0.5 to INT[224 * 0.5 + 128] = 240. The values are then stored as 8-bit<br>> unsigned integers in memory.<br>><br>> For full range quantization, the same applies, with a multiplier equal<br>> to 255 instead of 224. [-0.5, 0.5] is thus mapped to [0, 255].<br>><br>> We need to apply the reverse quantization on D'Y, D'Cb and D'Cr in order<br>> to get the original E'Y, E'Cb and E'Cr values (in the [0.0, 1.0] and<br>> [-0.5, 0.5] ranges respectively for E'Y and E'C[br]. Starting with full<br>> range, given<br>><br>> D'Cb = INT[(255*E'Cb + 128)] (for 8-bit data)<br>><br>> the inverse is given by<br>><br>> E'Cb = (D'Cb - 128) / 255<br>><br>> or<br>><br>> E'Cb = D'Cb / 255 - 128 / 255<br>><br>> OpenGL, when reading texture data through a floating point texture<br>> sampler (which we do in the shader by calling texture2D on a sampler2D<br>> variable), normalizes the values stored in memory ([0, 255]) to the<br>> [0.0, 1.0] range. This means that the D'Cb value is already divided by<br>> 255 by the GPU. We only need to subtract 128 / 255 to get the original<br>> E'Cb value.<br>><br>> In the limited quantization range case, we have<br>><br>> D'Cb = INT[(225*E'Cb + 128)] (for 8-bit data)<br>><br>> the inverse is given by<br>><br>> E'Cb = (D'Cb - 128) / 224<br>><br>> Let's introduce the 255 factor:<br>><br>> E'Cb = (D'Cb - 128) / 255 * 255 / 224<br>><br>> which can also be written as<br>><br>> E'Cb = (D'Cb / 255 - 128 / 255) * 255 / 224<br>><br>> We thus have<br>><br>> E'Cb(lim) = E'Cb(full) * 255 / 224<br>><br>> The shader doesn't include the 255 / 224 multiplier directly, it gets<br>> included by the C++ code in the yuv2rgb matrix, and there's no need for<br>> a different offset between the limited and full range quantization.<br>><br>> I hope this helps clarifying the implementation.<br>> --<br>> Regards,<br>><br>> Laurent Pinchart<br><br>I had gone through this conversion on multiple resources. <br>The implementation looks correct.<br><br>Reviewed-by: Kunal Agarwal <<a href="mailto:kunalagarwal1072002@gmail.com">kunalagarwal1072002@gmail.com</a>><br><br>Regards,<br><br>Kunal Agarwal</div><br><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Aug 30, 2022 at 10:57 PM Laurent Pinchart via libcamera-devel <<a href="mailto:libcamera-devel@lists.libcamera.org">libcamera-devel@lists.libcamera.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi Umang,<br>
<br>
On Tue, Aug 30, 2022 at 07:43:12PM +0530, Umang Jain wrote:<br>
> On 8/29/22 3:34 PM, Laurent Pinchart via libcamera-devel wrote:<br>
> > Update the YUV shaders and the viewfinder_gl to correctly take the<br>
> > Y'CbCr encoding and the quantization range into account when rendering<br>
> > YUV formats to RGB. Support for the primaries and transfer function will<br>
> > be added in a subsequent step.<br>
> ><br>
> > Signed-off-by: Laurent Pinchart <<a href="mailto:laurent.pinchart@ideasonboard.com" target="_blank">laurent.pinchart@ideasonboard.com</a>><br>
> <br>
> Patch looks good and straight forward for most parts, however few <br>
> specifics are still a bit unclear to me<br>
> <br>
> > ---<br>
> > src/qcam/assets/shader/YUV_2_planes.frag | 27 ++++----<br>
> > src/qcam/assets/shader/YUV_3_planes.frag | 23 ++++---<br>
> > src/qcam/assets/shader/YUV_packed.frag | 17 ++---<br>
> > src/qcam/viewfinder_gl.cpp | 79 +++++++++++++++++++++++-<br>
> > src/qcam/viewfinder_gl.h | 2 +<br>
> > 5 files changed, 115 insertions(+), 33 deletions(-)<br>
> ><br>
> > diff --git a/src/qcam/assets/shader/YUV_2_planes.frag b/src/qcam/assets/shader/YUV_2_planes.frag<br>
> > index 254463c05cac..da8dbcc5f801 100644<br>
> > --- a/src/qcam/assets/shader/YUV_2_planes.frag<br>
> > +++ b/src/qcam/assets/shader/YUV_2_planes.frag<br>
> > @@ -13,27 +13,30 @@ varying vec2 textureOut;<br>
> > uniform sampler2D tex_y;<br>
> > uniform sampler2D tex_u;<br>
> > <br>
> > +const mat3 yuv2rgb_matrix = mat3(<br>
> > + YUV2RGB_MATRIX<br>
> > +);<br>
> > +<br>
> > +const vec3 yuv2rgb_offset = vec3(<br>
> > + YUV2RGB_Y_OFFSET / 255.0, 128.0 / 255.0, 128.0 / 255.0<br>
> <br>
> I understood the YUV2RGB_Y_OFFSET #define but don't understand where <br>
> other values come from (or why they exist :D)<br>
<br>
The quantization of the Cb and Cr values in all relevant color spaces<br>
(ITU-R BT.601, BT.709, BT.2020, ...) add an offset of 128 (for 8-bit<br>
values). For instance, in BT.709, we have<br>
<br>
D'Cb = INT[(224*E'Cb + 128)*2^(n-8)]<br>
<br>
where D'Cb is the Cb signal after quantization, E'Cb the Cb signal<br>
before quantization (in the [-0.5, 0.5] range), and n the number of<br>
bits). INT[] denotes rounding to the closest integer.<br>
<br>
The 224 multiplier creates a limited quantization range, following the<br>
above formula, -0.5 will be quantized to INT[224 * -0.5 + 128] = 16, and<br>
0.5 to INT[224 * 0.5 + 128] = 240. The values are then stored as 8-bit<br>
unsigned integers in memory.<br>
<br>
For full range quantization, the same applies, with a multiplier equal<br>
to 255 instead of 224. [-0.5, 0.5] is thus mapped to [0, 255].<br>
<br>
We need to apply the reverse quantization on D'Y, D'Cb and D'Cr in order<br>
to get the original E'Y, E'Cb and E'Cr values (in the [0.0, 1.0] and<br>
[-0.5, 0.5] ranges respectively for E'Y and E'C[br]. Starting with full<br>
range, given<br>
<br>
D'Cb = INT[(255*E'Cb + 128)] (for 8-bit data)<br>
<br>
the inverse is given by<br>
<br>
E'Cb = (D'Cb - 128) / 255<br>
<br>
or<br>
<br>
E'Cb = D'Cb / 255 - 128 / 255<br>
<br>
OpenGL, when reading texture data through a floating point texture<br>
sampler (which we do in the shader by calling texture2D on a sampler2D<br>
variable), normalizes the values stored in memory ([0, 255]) to the<br>
[0.0, 1.0] range. This means that the D'Cb value is already divided by<br>
255 by the GPU. We only need to subtract 128 / 255 to get the original<br>
E'Cb value.<br>
<br>
In the limited quantization range case, we have<br>
<br>
D'Cb = INT[(225*E'Cb + 128)] (for 8-bit data)<br>
<br>
the inverse is given by<br>
<br>
E'Cb = (D'Cb - 128) / 224<br>
<br>
Let's introduce the 255 factor:<br>
<br>
E'Cb = (D'Cb - 128) / 255 * 255 / 224<br>
<br>
which can also be written as<br>
<br>
E'Cb = (D'Cb / 255 - 128 / 255) * 255 / 224<br>
<br>
We thus have<br>
<br>
E'Cb(lim) = E'Cb(full) * 255 / 224<br>
<br>
The shader doesn't include the 255 / 224 multiplier directly, it gets<br>
included by the C++ code in the yuv2rgb matrix, and there's no need for<br>
a different offset between the limited and full range quantization.<br>
<br>
I hope this helps clarifying the implementation.<br>
<br>
> Maybe I should start learning shaders programming ;-)<br>
> <br>
> Reviewed-by: Umang Jain <<a href="mailto:umang.jain@ideasonboard.com" target="_blank">umang.jain@ideasonboard.com</a>><br>
> <br>
> > +);<br>
> > +<br>
> > void main(void)<br>
> > {<br>
> > vec3 yuv;<br>
> > - vec3 rgb;<br>
> > - mat3 yuv2rgb_bt601_mat = mat3(<br>
> > - vec3(1.164, 1.164, 1.164),<br>
> > - vec3(0.000, -0.392, 2.017),<br>
> > - vec3(1.596, -0.813, 0.000)<br>
> > - );<br>
> > <br>
> > - yuv.x = texture2D(tex_y, textureOut).r - 0.063;<br>
> > + yuv.x = texture2D(tex_y, textureOut).r;<br>
> > #if defined(YUV_PATTERN_UV)<br>
> > - yuv.y = texture2D(tex_u, textureOut).r - 0.500;<br>
> > - yuv.z = texture2D(tex_u, textureOut).a - 0.500;<br>
> > + yuv.y = texture2D(tex_u, textureOut).r;<br>
> > + yuv.z = texture2D(tex_u, textureOut).a;<br>
> > #elif defined(YUV_PATTERN_VU)<br>
> > - yuv.y = texture2D(tex_u, textureOut).a - 0.500;<br>
> > - yuv.z = texture2D(tex_u, textureOut).r - 0.500;<br>
> > + yuv.y = texture2D(tex_u, textureOut).a;<br>
> > + yuv.z = texture2D(tex_u, textureOut).r;<br>
> > #else<br>
> > #error Invalid pattern<br>
> > #endif<br>
> > <br>
> > - rgb = yuv2rgb_bt601_mat * yuv;<br>
> > + vec3 rgb = yuv2rgb_matrix * (vec3(y, uv) - yuv2rgb_offset);<br>
> > +<br>
> > gl_FragColor = vec4(rgb, 1.0);<br>
> > }<br>
> > diff --git a/src/qcam/assets/shader/YUV_3_planes.frag b/src/qcam/assets/shader/YUV_3_planes.frag<br>
> > index 2be74b5d2a9d..e754129d74d1 100644<br>
> > --- a/src/qcam/assets/shader/YUV_3_planes.frag<br>
> > +++ b/src/qcam/assets/shader/YUV_3_planes.frag<br>
> > @@ -14,20 +14,23 @@ uniform sampler2D tex_y;<br>
> > uniform sampler2D tex_u;<br>
> > uniform sampler2D tex_v;<br>
> > <br>
> > +const mat3 yuv2rgb_matrix = mat3(<br>
> > + YUV2RGB_MATRIX<br>
> > +);<br>
> > +<br>
> > +const vec3 yuv2rgb_offset = vec3(<br>
> > + YUV2RGB_Y_OFFSET / 255.0, 128.0 / 255.0, 128.0 / 255.0<br>
> > +);<br>
> > +<br>
> > void main(void)<br>
> > {<br>
> > vec3 yuv;<br>
> > - vec3 rgb;<br>
> > - mat3 yuv2rgb_bt601_mat = mat3(<br>
> > - vec3(1.164, 1.164, 1.164),<br>
> > - vec3(0.000, -0.392, 2.017),<br>
> > - vec3(1.596, -0.813, 0.000)<br>
> > - );<br>
> > <br>
> > - yuv.x = texture2D(tex_y, textureOut).r - 0.063;<br>
> > - yuv.y = texture2D(tex_u, textureOut).r - 0.500;<br>
> > - yuv.z = texture2D(tex_v, textureOut).r - 0.500;<br>
> > + yuv.x = texture2D(tex_y, textureOut).r;<br>
> > + yuv.y = texture2D(tex_u, textureOut).r;<br>
> > + yuv.z = texture2D(tex_v, textureOut).r;<br>
> > +<br>
> > + vec3 rgb = yuv2rgb_matrix * (vec3(y, uv) - yuv2rgb_offset);<br>
> > <br>
> > - rgb = yuv2rgb_bt601_mat * yuv;<br>
> > gl_FragColor = vec4(rgb, 1.0);<br>
> > }<br>
> > diff --git a/src/qcam/assets/shader/YUV_packed.frag b/src/qcam/assets/shader/YUV_packed.frag<br>
> > index d6efd4ce92a9..b9ef9d41beae 100644<br>
> > --- a/src/qcam/assets/shader/YUV_packed.frag<br>
> > +++ b/src/qcam/assets/shader/YUV_packed.frag<br>
> > @@ -14,15 +14,16 @@ varying vec2 textureOut;<br>
> > uniform sampler2D tex_y;<br>
> > uniform vec2 tex_step;<br>
> > <br>
> > +const mat3 yuv2rgb_matrix = mat3(<br>
> > + YUV2RGB_MATRIX<br>
> > +);<br>
> > +<br>
> > +const vec3 yuv2rgb_offset = vec3(<br>
> > + YUV2RGB_Y_OFFSET / 255.0, 128.0 / 255.0, 128.0 / 255.0<br>
> > +);<br>
> > +<br>
> > void main(void)<br>
> > {<br>
> > - mat3 yuv2rgb_bt601_mat = mat3(<br>
> > - vec3(1.164, 1.164, 1.164),<br>
> > - vec3(0.000, -0.392, 2.017),<br>
> > - vec3(1.596, -0.813, 0.000)<br>
> > - );<br>
> > - vec3 yuv2rgb_bt601_offset = vec3(0.063, 0.500, 0.500);<br>
> > -<br>
> > /*<br>
> > * The sampler won't interpolate the texture correctly along the X axis,<br>
> > * as each RGBA pixel effectively stores two pixels. We thus need to<br>
> > @@ -76,7 +77,7 @@ void main(void)<br>
> > <br>
> > float y = mix(y_left, y_right, step(0.5, f_x));<br>
> > <br>
> > - vec3 rgb = yuv2rgb_bt601_mat * (vec3(y, uv) - yuv2rgb_bt601_offset);<br>
> > + vec3 rgb = yuv2rgb_matrix * (vec3(y, uv) - yuv2rgb_offset);<br>
> > <br>
> > gl_FragColor = vec4(rgb, 1.0);<br>
> > }<br>
> > diff --git a/src/qcam/viewfinder_gl.cpp b/src/qcam/viewfinder_gl.cpp<br>
> > index ec295b6de0dd..e2aa24703ff0 100644<br>
> > --- a/src/qcam/viewfinder_gl.cpp<br>
> > +++ b/src/qcam/viewfinder_gl.cpp<br>
> > @@ -7,9 +7,12 @@<br>
> > <br>
> > #include "viewfinder_gl.h"<br>
> > <br>
> > +#include <array><br>
> > +<br>
> > #include <QByteArray><br>
> > #include <QFile><br>
> > #include <QImage><br>
> > +#include <QStringList><br>
> > <br>
> > #include <libcamera/formats.h><br>
> > <br>
> > @@ -56,7 +59,8 @@ static const QList<libcamera::PixelFormat> supportedFormats{<br>
> > };<br>
> > <br>
> > ViewFinderGL::ViewFinderGL(QWidget *parent)<br>
> > - : QOpenGLWidget(parent), buffer_(nullptr), image_(nullptr),<br>
> > + : QOpenGLWidget(parent), buffer_(nullptr),<br>
> > + colorSpace_(libcamera::ColorSpace::Raw), image_(nullptr),<br>
> > vertexBuffer_(QOpenGLBuffer::VertexBuffer)<br>
> > {<br>
> > }<br>
> > @@ -72,10 +76,10 @@ const QList<libcamera::PixelFormat> &ViewFinderGL::nativeFormats() const<br>
> > }<br>
> > <br>
> > int ViewFinderGL::setFormat(const libcamera::PixelFormat &format, const QSize &size,<br>
> > - [[maybe_unused]] const libcamera::ColorSpace &colorSpace,<br>
> > + const libcamera::ColorSpace &colorSpace,<br>
> > unsigned int stride)<br>
> > {<br>
> > - if (format != format_) {<br>
> > + if (format != format_ || colorSpace != colorSpace_) {<br>
> > /*<br>
> > * If the fragment already exists, remove it and create a new<br>
> > * one for the new format.<br>
> > @@ -89,7 +93,10 @@ int ViewFinderGL::setFormat(const libcamera::PixelFormat &format, const QSize &s<br>
> > if (!selectFormat(format))<br>
> > return -1;<br>
> > <br>
> > + selectColorSpace(colorSpace);<br>
> > +<br>
> > format_ = format;<br>
> > + colorSpace_ = colorSpace;<br>
> > }<br>
> > <br>
> > size_ = size;<br>
> > @@ -318,6 +325,72 @@ bool ViewFinderGL::selectFormat(const libcamera::PixelFormat &format)<br>
> > return ret;<br>
> > }<br>
> > <br>
> > +void ViewFinderGL::selectColorSpace(const libcamera::ColorSpace &colorSpace)<br>
> > +{<br>
> > + std::array<double, 9> yuv2rgb;<br>
> > +<br>
> > + /* OpenGL stores arrays in column-major order. */<br>
> > + switch (colorSpace.ycbcrEncoding) {<br>
> > + case libcamera::ColorSpace::YcbcrEncoding::None:<br>
> > + yuv2rgb = {<br>
> > + 1.0000, 0.0000, 0.0000,<br>
> > + 0.0000, 1.0000, 0.0000,<br>
> > + 0.0000, 0.0000, 1.0000,<br>
> > + };<br>
> > + break;<br>
> > +<br>
> > + case libcamera::ColorSpace::YcbcrEncoding::Rec601:<br>
> > + yuv2rgb = {<br>
> > + 1.0000, 1.0000, 1.0000,<br>
> > + 0.0000, -0.3441, 1.7720,<br>
> > + 1.4020, -0.7141, 0.0000,<br>
> > + };<br>
> > + break;<br>
> > +<br>
> > + case libcamera::ColorSpace::YcbcrEncoding::Rec709:<br>
> > + yuv2rgb = {<br>
> > + 1.0000, 1.0000, 1.0000,<br>
> > + 0.0000, -0.1873, 1.8856,<br>
> > + 1.5748, -0.4681, 0.0000,<br>
> > + };<br>
> > + break;<br>
> > +<br>
> > + case libcamera::ColorSpace::YcbcrEncoding::Rec2020:<br>
> > + yuv2rgb = {<br>
> > + 1.0000, 1.0000, 1.0000,<br>
> > + 0.0000, -0.1646, 1.8814,<br>
> > + 1.4746, -0.5714, 0.0000,<br>
> > + };<br>
> > + break;<br>
> > + }<br>
> > +<br>
> > + double offset;<br>
> > +<br>
> > + switch (colorSpace.range) {<br>
> > + case libcamera::ColorSpace::Range::Full:<br>
> > + offset = 0.0;<br>
> > + break;<br>
> > +<br>
> > + case libcamera::ColorSpace::Range::Limited:<br>
> > + offset = 16.0;<br>
> > +<br>
> > + for (unsigned int i = 0; i < 3; ++i)<br>
> > + yuv2rgb[i] *= 255.0 / 219.0;<br>
> > + for (unsigned int i = 4; i < 9; ++i)<br>
> > + yuv2rgb[i] *= 255.0 / 224.0;<br>
> > + break;<br>
> > + }<br>
> > +<br>
> > + QStringList matrix;<br>
> > +<br>
> > + for (double coeff : yuv2rgb)<br>
> > + matrix.append(QString::number(coeff, 'f'));<br>
> > +<br>
> > + fragmentShaderDefines_.append("#define YUV2RGB_MATRIX " + matrix.join(", "));<br>
> > + fragmentShaderDefines_.append(QString("#define YUV2RGB_Y_OFFSET %1")<br>
> > + .arg(offset, 0, 'f', 1));<br>
> > +}<br>
> > +<br>
> > bool ViewFinderGL::createVertexShader()<br>
> > {<br>
> > /* Create Vertex Shader */<br>
> > diff --git a/src/qcam/viewfinder_gl.h b/src/qcam/viewfinder_gl.h<br>
> > index 798830a31cd2..68c2912df12f 100644<br>
> > --- a/src/qcam/viewfinder_gl.h<br>
> > +++ b/src/qcam/viewfinder_gl.h<br>
> > @@ -57,6 +57,7 @@ protected:<br>
> > <br>
> > private:<br>
> > bool selectFormat(const libcamera::PixelFormat &format);<br>
> > + void selectColorSpace(const libcamera::ColorSpace &colorSpace);<br>
> > <br>
> > void configureTexture(QOpenGLTexture &texture);<br>
> > bool createFragmentShader();<br>
> > @@ -67,6 +68,7 @@ private:<br>
> > /* Captured image size, format and buffer */<br>
> > libcamera::FrameBuffer *buffer_;<br>
> > libcamera::PixelFormat format_;<br>
> > + libcamera::ColorSpace colorSpace_;<br>
> > QSize size_;<br>
> > unsigned int stride_;<br>
> > Image *image_;<br>
<br>
-- <br>
Regards,<br>
<br>
Laurent Pinchart<br>
</blockquote></div>