[libcamera-devel] [PATCH v2 00/27] libcamera: Handle fallout of FrameBuffer offset support
Umang Jain
umang.jain at ideasonboard.com
Mon Sep 6 12:08:04 CEST 2021
Hi Laurent
On 9/6/21 7:30 AM, Laurent Pinchart wrote:
> Hello everybody,
>
> This patch series started as an investigation of a qcam failure with
> IPU3 after the merge of the FrameBuffer offset support. While a hack in
> qcam would be possible, I decided to instead address the core issue and
> fix it in V4L2VideoDevice.
>
> Compared to v1, the series now includes fixes for cam and qcam in
> addition to the changes needed in the libcamera core. They have been
> tested with the Raspberry Pi, IPU3, VIMC and UVC pipeline handlers.
>
> The GStreamer element seems to work fine without any change required.
> The V4L2 compatibility layer is still broken, and I haven't tested the
> Android HAL yet (any volunteer ?).
I am applied this series on top of master and tested on nautilus.
Streaming seems fine to me, although it was appearing more choppy than
usual.
Now that I see, entire nautilus seems laggy to respond in general. One
"Chrome" process hogging 50% CPU, seems like a transient issue.
So, streaming seems fine on multiple runs, with multiple resolutions
selections (from the camera-app menu), however, requesting MJPEG
(shutter-click) results in a segfault.
The stack trace is below:
[0:24:11.415660787] [10635] DEBUG HAL camera_device.cpp:941
'\_SB_.PCI0.I2C2.CAM0': Queueing request 140736750235824 with 2 streams
[0:24:11.415858538] [10635] DEBUG Buffer framebuffer.cpp:249 Buffer is
not contiguous
[0:24:11.415905500] [10635] DEBUG HAL camera_device.cpp:980
'\_SB_.PCI0.I2C2.CAM0': 0 - (4160x3104)[0x00000023] -> (4160x3104)[NV12]
(direct)
[0:24:11.415943620] [10635] DEBUG HAL camera_device.cpp:966
'\_SB_.PCI0.I2C2.CAM0': 1 - (4160x3104)[0x00000021] -> (4160x3104)[NV12]
(mapped)
[0:24:11.416461639] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582
/dev/video0[37:cap]: Queueing buffer 3
[0:24:11.428903141] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video0[37:cap]: Dequeuing buffer 2
[0:24:11.429095316] [10643] DEBUG IPU3Awb awb.cpp:331 Color temperature
estimated: 6644
[0:24:11.429563781] [10629] DEBUG DelayedControls
delayed_controls.cpp:237 frame 193 started
[0:24:11.429627478] [10629] DEBUG DelayedControls
delayed_controls.cpp:272 Setting Exposure to 269 at index 193
[0:24:11.429683048] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582
/dev/video6[25:cap]: Queueing buffer 1
[0:24:11.436615191] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582
/dev/video5[27:out]: Queueing buffer 2
[0:24:11.436718629] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582
/dev/video8[28:cap]: Queueing buffer 2
[0:24:11.436799420] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582
/dev/video4[24:out]: Queueing buffer 2
[0:24:11.457388821] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video4[24:out]: Dequeuing buffer 1
[0:24:11.457948159] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video6[25:cap]: Dequeuing buffer 0
[0:24:11.458257692] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video5[27:out]: Dequeuing buffer 2
[0:24:11.458525315] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video8[28:cap]: Dequeuing buffer 1
[0:24:11.458968848] [10643] DEBUG IPU3Awb awb.cpp:270 Valid zones: 186
[0:24:11.459005634] [10643] DEBUG IPU3Awb awb.cpp:224 Grey world AWB
[0:24:11.459082715] [10643] DEBUG IPU3Awb awb.cpp:273 Gain found for
red: 1.16864 and for blue: 1.14863
[0:24:11.459325998] [10629] DEBUG DelayedControls
delayed_controls.cpp:179 Queuing Exposure to 269 at index 194
[0:24:11.459562485] [10629] DEBUG DelayedControls
delayed_controls.cpp:179 Queuing Analogue Gain to 477 at index 194
[0:24:11.459815347] [10629] DEBUG Request request.cpp:291
Request(129:C:0/1:140736750229136)
[0:24:11.460390126] [10629] DEBUG HAL camera_device.cpp:1117
'\_SB_.PCI0.I2C2.CAM0': Request 140736750229136 completed with 1 streams
[0:24:11.461855854] [10635] DEBUG Request request.cpp:94 Created request
- cookie: 140736750230064
[0:24:11.461939858] [10635] DEBUG HAL camera_device.cpp:941
'\_SB_.PCI0.I2C2.CAM0': Queueing request 140736750230064 with 1 streams
[0:24:11.462057435] [10635] DEBUG Buffer framebuffer.cpp:249 Buffer is
not contiguous
[0:24:11.462098906] [10635] DEBUG HAL camera_device.cpp:980
'\_SB_.PCI0.I2C2.CAM0': 0 - (4160x3104)[0x00000023] -> (4160x3104)[NV12]
(direct)
[0:24:11.463164722] [10629] DEBUG DelayedControls
delayed_controls.cpp:237 frame 194 started
[0:24:11.463730946] [10629] DEBUG DelayedControls
delayed_controls.cpp:272 Setting Analogue Gain to 477 at index 193
[0:24:11.464033686] [10629] DEBUG DelayedControls
delayed_controls.cpp:272 Setting Exposure to 269 at index 194
[0:24:11.464329869] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582
/dev/video0[37:cap]: Queueing buffer 0
[0:24:11.495739092] [10629] DEBUG DelayedControls
delayed_controls.cpp:237 frame 195 started
[0:24:11.496489311] [10629] DEBUG DelayedControls
delayed_controls.cpp:272 Setting Analogue Gain to 477 at index 194
[0:24:11.496737385] [10629] DEBUG DelayedControls
delayed_controls.cpp:285 Queue is empty, auto queue no-op.
[0:24:11.497044311] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video0[37:cap]: Dequeuing buffer 3
[0:24:11.497335155] [10643] DEBUG IPU3Awb awb.cpp:331 Color temperature
estimated: 6658
[0:24:11.497528845] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582
/dev/video6[25:cap]: Queueing buffer 2
[0:24:11.503589322] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582
/dev/video5[27:out]: Queueing buffer 3
[0:24:11.504027344] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582
/dev/video8[28:cap]: Queueing buffer 3
[0:24:11.504287330] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582
/dev/video4[24:out]: Queueing buffer 3
[0:24:11.504712501] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video4[24:out]: Dequeuing buffer 2
[0:24:11.505005096] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video6[25:cap]: Dequeuing buffer 1
[0:24:11.505260331] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video5[27:out]: Dequeuing buffer 3
[0:24:11.505506837] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video8[28:cap]: Dequeuing buffer 2
[0:24:11.505940926] [10643] DEBUG IPU3Awb awb.cpp:270 Valid zones: 186
[0:24:11.505976974] [10643] DEBUG IPU3Awb awb.cpp:224 Grey world AWB
[0:24:11.506057427] [10643] DEBUG IPU3Awb awb.cpp:273 Gain found for
red: 1.16797 and for blue: 1.14978
[0:24:11.506297848] [10629] DEBUG DelayedControls
delayed_controls.cpp:179 Queuing Exposure to 269 at index 196
[0:24:11.506547021] [10629] DEBUG DelayedControls
delayed_controls.cpp:179 Queuing Analogue Gain to 477 at index 196
[0:24:11.506796456] [10629] DEBUG Request request.cpp:291
Request(130:C:0/1:140736750225984)
[0:24:11.507374522] [10629] DEBUG HAL camera_device.cpp:1117
'\_SB_.PCI0.I2C2.CAM0': Request 140736750225984 completed with 1 streams
[0:24:11.509426987] [10635] DEBUG Request request.cpp:94 Created request
- cookie: 140736750227664
[0:24:11.509507039] [10635] DEBUG HAL camera_device.cpp:941
'\_SB_.PCI0.I2C2.CAM0': Queueing request 140736750227664 with 1 streams
[0:24:11.509630706] [10635] DEBUG Buffer framebuffer.cpp:249 Buffer is
not contiguous
[0:24:11.509675200] [10635] DEBUG HAL camera_device.cpp:980
'\_SB_.PCI0.I2C2.CAM0': 0 - (4160x3104)[0x00000023] -> (4160x3104)[NV12]
(direct)
[0:24:11.511309605] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582
/dev/video0[37:cap]: Queueing buffer 1
[0:24:11.528984601] [10629] DEBUG DelayedControls
delayed_controls.cpp:237 frame 196 started
[0:24:11.529446048] [10629] DEBUG DelayedControls
delayed_controls.cpp:272 Setting Exposure to 269 at index 196
[0:24:11.548431632] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video4[24:out]: Dequeuing buffer 3
[0:24:11.548993830] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video6[25:cap]: Dequeuing buffer 2
[0:24:11.549320802] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650
/dev/video8[28:cap]: Dequeuing buffer 3
[0:24:11.549807736] [10643] DEBUG IPU3Awb awb.cpp:270 Valid zones: 186
[0:24:11.549846777] [10643] DEBUG IPU3Awb awb.cpp:224 Grey world AWB
[0:24:11.549927350] [10643] DEBUG IPU3Awb awb.cpp:273 Gain found for
red: 1.16841 and for blue: 1.14912
[0:24:11.550170924] [10629] DEBUG DelayedControls
delayed_controls.cpp:179 Queuing Exposure to 269 at index 197
[0:24:11.550405629] [10629] DEBUG DelayedControls
delayed_controls.cpp:179 Queuing Analogue Gain to 477 at index 197
[0:24:11.550646460] [10629] DEBUG Request request.cpp:291
Request(131:C:0/1:140736750235824)
[0:24:11.551245104] [10629] DEBUG HAL camera_device.cpp:1117
'\_SB_.PCI0.I2C2.CAM0': Request 140736750235824 completed with 2 streams
[0:24:11.562307680] [10629] DEBUG EXIF exif.cpp:522 Created EXIF
instance (536 bytes)
[0:24:11.634800113] [10629] DEBUG JPEG encoder_libjpeg.cpp:220 JPEG
Encode Starting:4160x3104
Thread 6 "CameraModuleThr" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe7fff640 (LWP 10629)]
0x00007ffff597ead5 in
EncoderLibJpeg::compressNV(libcamera::Span<unsigned char const,
18446744073709551615ul>) () from /usr/lib64/camera_hal/libcamera-hal.so
(gdb) bt
#0 0x00007ffff597ead5 in
EncoderLibJpeg::compressNV(libcamera::Span<unsigned char const,
18446744073709551615ul>) () from /usr/lib64/camera_hal/libcamera-hal.so
#1 0x00007ffff597eda0 in
EncoderLibJpeg::encode(libcamera::Span<unsigned char const,
18446744073709551615ul>, libcamera::Span<unsigned char,
18446744073709551615ul>, libcamera::Span<unsigned char const,
18446744073709551615ul>, unsigned int) () from
/usr/lib64/camera_hal/libcamera-hal.so
#2 0x00007ffff597ec53 in EncoderLibJpeg::encode(libcamera::FrameBuffer
const&, libcamera::Span<unsigned char, 18446744073709551615ul>,
libcamera::Span<unsigned char const, 18446744073709551615ul>, unsigned
int) () from /usr/lib64/camera_hal/libcamera-hal.so
#3 0x00007ffff5981871 in
PostProcessorJpeg::process(libcamera::FrameBuffer const&, CameraBuffer*,
CameraMetadata const&, CameraMetadata*) ()
from /usr/lib64/camera_hal/libcamera-hal.so
#4 0x00007ffff597dd8d in CameraStream::process(libcamera::FrameBuffer
const&, native_handle const*, CameraMetadata const&, CameraMetadata*) ()
from /usr/lib64/camera_hal/libcamera-hal.so
#5 0x00007ffff59717b9 in
CameraDevice::requestComplete(libcamera::Request*) () from
/usr/lib64/camera_hal/libcamera-hal.so
#6 0x00007ffff5979a7c in libcamera::BoundMethodMember<CameraDevice,
void, libcamera::Request*>::activate(libcamera::Request*, bool) ()
from /usr/lib64/camera_hal/libcamera-hal.so
#7 0x00007ffff58f9051 in libcamera::Signal<libcamera::IPCMessage
const&>::emit(libcamera::IPCMessage const&) () from /usr/lib64/libcamera.so
#8 0x00007ffff58f8fe8 in
libcamera::Camera::requestComplete(libcamera::Request*) () from
/usr/lib64/libcamera.so
#9 0x00007ffff591f28a in
libcamera::PipelineHandler::completeRequest(libcamera::Request*) () from
/usr/lib64/libcamera.so
#10 0x00007ffff5935cef in
libcamera::IPU3CameraData::queueFrameAction(unsigned int,
libcamera::ipa::ipu3::IPU3Action const&) () from /usr/lib64/libcamera.so
#11 0x00007ffff58f0b02 in
libcamera::BoundMethodMember<libcamera::ipa::ipu3::IPAProxyIPU3, void,
unsigned int, libcamera::ipa::ipu3::IPU3Action
const&>::activate(unsigned int, libcamera::ipa::ipu3::IPU3Action const&,
bool) () from /usr/lib64/libcamera.so
#12 0x00007ffff58edf81 in libcamera::Signal<unsigned int,
libcamera::ipa::ipu3::IPU3Action const&>::emit(unsigned int,
libcamera::ipa::ipu3::IPU3Action const&) ()
from /usr/lib64/libcamera.so
#13 0x00007ffff58eb5b4 in
libcamera::ipa::ipu3::IPAProxyIPU3::queueFrameActionThread(unsigned int,
libcamera::ipa::ipu3::IPU3Action const&) ()
from /usr/lib64/libcamera.so
#14 0x00007ffff587caed in
libcamera::Object::message(libcamera::Message*) () from
/usr/lib64/libcamera-base.so
#15 0x00007ffff587e00b in
libcamera::Thread::dispatchMessages(libcamera::Message::Type) () from
/usr/lib64/libcamera-base.so
#16 0x00007ffff5876925 in
libcamera::EventDispatcherPoll::processEvents() () from
/usr/lib64/libcamera-base.so
#17 0x00007ffff587d6e2 in libcamera::Thread::exec() () from
/usr/lib64/libcamera-base.so
#18 0x00007ffff58fb582 in libcamera::CameraManager::Private::run() ()
from /usr/lib64/libcamera.so
#19 0x00007ffff587e43c in ?? () from /usr/lib64/libcamera-base.so
#20 0x0000000000000000 in ?? ()
>
> The most important change is in patches 13/27 and 14/27, which translate
> between V4L2 buffers and libcamera FrameBuffer to handle the case where
> a multi-planar frame buffer is used with the V4L2 single-planar API.
> It's working more or less by chance at the moment (except in qcam where
> it's broken, and possibly in other places I haven't tested). Patches
> 01/27 to 12/27 are cleanups and additions to prepare for the work in
> V4L2VideoDevice, and patch 15/27 is a small cleanup on top. Patches
> 16/27 and 17/27 then improve the FrameBuffer class API as a cleanup.
>
> Patches 18/27 to 27/27 fix the cam and qcam applications, as well as an
> issue in the Android HAL. Worth being noted is patch 19/27 that
> introduces an Image class shared by cam and qcam. The class duplicates
> the MappedFrameBuffer implementation private to libcamera. I've tried to
> rework MappedFrameBuffer into something I would be happy to see in the
> public API, but failed to do so in a reasonable amount of time, and I
> didn't want to delay this important regression fix.
>
> This series doesn't break any unit test, as vimc doesn't support NV12.
> Addition of NV12 support to the vimc kernel driver would be very nice,
> in order to test multi-planar support in our unit tests. Volunteers are
> welcome ;-)
>
> Laurent Pinchart (27):
> libcamera: base: utils: Use size_t for index in utils::enumerate()
> libcamera: file_descriptor: Add a function to retrieve the inode
> libcamera: v4l2_videodevice: Drop toV4L2PixelFormat()
> libcamera: Use V4L2PixelFormat::fromPixelFormat()
> libcamera: formats: Move plane info structure to PixelFormatInfo
> libcamera: formats: Add planeSize() helpers to PixelFormatInfo
> libcamera: formats: Support V4L2 non-contiguous formats
> libcamera: framebuffer: Move planes check to constructor
> libcamera: framebuffer: Add a function to check if planes are
> contiguous
> libcamera: v4l2_videodevice: Cache PixelFormatInfo
> libcamera: v4l2_videodevice: Document plane handling in createBuffer()
> libcamera: v4l2_videodevice: Take stride into account to compute
> offsets
> libcamera: v4l2_videodevice: Coalesce planes when queuing buffer
> libcamera: v4l2_videodevice: Split planes when dequeuing buffer
> libcamera: v4l2_videodevice: Use utils::enumerate()
> libcamera: framebuffer: Allocate metadata planes at construction time
> libcamera: framebuffer: Prevent modifying the number of metadata
> planes
> android: camera_device: Don't assume all planes use the same fd
> cam: Add Image class
> cam: file_sink: Use Image class to access pixel data
> cam: drm: Support per-plane stride values
> cam: drm: Set per-plane offsets when creating DRM frame buffer
> cam: drm: Avoid importing the same dmabuf multiple times
> qcam: Print bytesused for all planes
> qcam: Use Image class to access pixel data
> qcam: viewfinder_gl: Support multi-planar buffers
> qcam: viewfinder_qt: Support multi-planar buffers
>
> include/libcamera/base/utils.h | 4 +-
> include/libcamera/file_descriptor.h | 3 +
> include/libcamera/framebuffer.h | 19 +-
> include/libcamera/internal/formats.h | 22 +-
> include/libcamera/internal/framebuffer.h | 2 +
> include/libcamera/internal/v4l2_pixelformat.h | 2 +-
> include/libcamera/internal/v4l2_videodevice.h | 3 +-
> src/android/camera_device.cpp | 25 +-
> src/android/mm/generic_camera_buffer.cpp | 11 +-
> src/android/yuv/post_processor_yuv.cpp | 10 +-
> src/cam/camera_session.cpp | 4 +-
> src/cam/drm.cpp | 38 +-
> src/cam/drm.h | 7 +-
> src/cam/file_sink.cpp | 44 +--
> src/cam/file_sink.h | 6 +-
> src/cam/image.cpp | 107 +++++
> src/cam/image.h | 52 +++
> src/cam/kms_sink.cpp | 28 +-
> src/cam/meson.build | 1 +
> src/libcamera/file_descriptor.cpp | 26 ++
> src/libcamera/formats.cpp | 373 ++++++++++++++----
> src/libcamera/framebuffer.cpp | 57 ++-
> src/libcamera/pipeline/ipu3/cio2.cpp | 4 +-
> src/libcamera/pipeline/ipu3/imgu.cpp | 2 +-
> .../pipeline/raspberrypi/raspberrypi.cpp | 8 +-
> src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 6 +-
> src/libcamera/pipeline/simple/converter.cpp | 8 +-
> src/libcamera/pipeline/simple/simple.cpp | 4 +-
> src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 6 +-
> src/libcamera/pipeline/vimc/vimc.cpp | 8 +-
> src/libcamera/v4l2_pixelformat.cpp | 11 +-
> src/libcamera/v4l2_videodevice.cpp | 196 ++++++---
> src/qcam/format_converter.cpp | 18 +-
> src/qcam/format_converter.h | 9 +-
> src/qcam/main_window.cpp | 38 +-
> src/qcam/main_window.h | 4 +-
> src/qcam/meson.build | 1 +
> src/qcam/viewfinder.h | 6 +-
> src/qcam/viewfinder_gl.cpp | 45 +--
> src/qcam/viewfinder_gl.h | 4 +-
> src/qcam/viewfinder_qt.cpp | 20 +-
> src/qcam/viewfinder_qt.h | 2 +-
> src/v4l2/v4l2_camera_proxy.cpp | 11 +-
> test/libtest/buffer_source.cpp | 3 +-
> test/utils.cpp | 10 +-
> 45 files changed, 911 insertions(+), 357 deletions(-)
> create mode 100644 src/cam/image.cpp
> create mode 100644 src/cam/image.h
>
More information about the libcamera-devel
mailing list