[libcamera-devel] JPEG support for libcamera

[Dave Stevenson]

On Tue, 5 Jul 2022 at 15:48, Pavel Machek via libcamera-devel
<libcamera-devel at lists.libcamera.org> wrote:
>
> Hi!
>
> > Hi Pavel,
> >    I discussed it with Rafael (see message id
> > <20220701075644.mcjbcwthzsofmssx at uno.localdomain>) and my take was
> > slightly different: the driver should be fixed to report MJPEG:
> >
> > Let me re-paste it here, mostly to validate my understanding as I
> > might have got this very wrong.
> >
> > -------------------------------------------------------------------------------
> >
> > Libcamera only supports:
> > #define V4L2_PIX_FMT_MJPEG    v4l2_fourcc('M', 'J', 'P', 'G') /* Motion-JPEG   */
> >
> > I admit I know nothing about JPEG related standards but looking at the
> > media drivers, it seems FMT_JPEG is used more often than MJPEG.
> >
> > However libcamera only seems to support MJPEG. You might have noticed
> > libcamera uses the DRM pixelformat identifiers, and it seems that in
> > the drm_fourcc.h header we ship there's no JPEG format at all, but
> > only MJPEG. If my understanding is correct this makes sense, as the
> > MJPEG standard defines a video container format for streaming
> > JPEG-compressed frames while JFIF JPEG defines a file-encoding scheme
> > for exchanging single images. Being DRM about video output devices
> > only MJPEG applies there, so there's no "JIF/JFIF JPEG" format in
> > there.
> >
> > Now, I would not be surprised if the multiple users of FMT_JPEG in
> > drivers/media/ are actually sending MJPEG but driver developers pick
> > FMT_JPEG because of some form of cargo-cult.
>
> Best explanation I found so far was that JPEG = EXIF + MJPEG. Kernel
> interface exposes JPEG and MJPEG as separate types, so IMO it makes
> sense for libcamera to support both.

TBH JPEG at least has a specification. MJPEG largely doesn't - it's
just a concatenation of JPEG encoded frames, and there seems to be no
real definition of which JPEG blocks are required and in which order.
Some MJPEG encoders drop the Huffman tables and rely on the decoder
assuming defaults.
There are then also MJPEG-A and MJPEG-B variants defined by Apple as
part of the Quicktime spec [1], with MJPEG-B not being compatible with
JPEG decoders due to changing headers.

Which flavour V4L2_PIX_FMT_MJPEG relates to seems to be undefined.

I've had issues in the past with what conditions VLC judges to be JPEG
vs MJPEG, and never found a definitive answer [2]

So how Libcamera should handle JPEG vs MJPEG probably doesn't have an
obvious answer. There's little point in encoding EXIF in a hardware
encoder as you have the overhead of passing all the parameters in.
For mapping into V4L2, just treat both of them the same.

  Dave

[1] https://developer.apple.com/standards/qtff-2001.pdf page 95.
[2] https://github.com/raspberrypi/userland/issues/345

> > To remove the above error I would change your driver to report
> > FMT_MJPEG for the time being until the above doesn't get clarified.
>
> I'm not really able to touch the kernel at the moment. Plus it is not
> "my" driver, it is driver in upstream kernel...
>
> If one of the formats is redundant and should be dropped, I guess we
> should have that discussion on the v4l2/kernel mailing lists...
>
> Best regards,
>                                                                 Pavel
> --
> People of Russia, stop Putin before his war on Ukraine escalates.