[libcamera-devel] [PATCH v2 00/27] libcamera: Handle fallout of FrameBuffer offset support
Laurent Pinchart
laurent.pinchart at ideasonboard.com
Mon Sep 6 14:24:48 CEST 2021
Hi Umang,
On Mon, Sep 06, 2021 at 03:38:04PM +0530, Umang Jain wrote:
> On 9/6/21 7:30 AM, Laurent Pinchart wrote:
> > Hello everybody,
> >
> > This patch series started as an investigation of a qcam failure with
> > IPU3 after the merge of the FrameBuffer offset support. While a hack in
> > qcam would be possible, I decided to instead address the core issue and
> > fix it in V4L2VideoDevice.
> >
> > Compared to v1, the series now includes fixes for cam and qcam in
> > addition to the changes needed in the libcamera core. They have been
> > tested with the Raspberry Pi, IPU3, VIMC and UVC pipeline handlers.
> >
> > The GStreamer element seems to work fine without any change required.
> > The V4L2 compatibility layer is still broken, and I haven't tested the
> > Android HAL yet (any volunteer ?).
>
> I am applied this series on top of master and tested on nautilus.
> Streaming seems fine to me, although it was appearing more choppy than
> usual.
>
> Now that I see, entire nautilus seems laggy to respond in general. One
> "Chrome" process hogging 50% CPU, seems like a transient issue.
>
> So, streaming seems fine on multiple runs, with multiple resolutions
> selections (from the camera-app menu), however, requesting MJPEG
> (shutter-click) results in a segfault.
Thanks for testing !
> The stack trace is below:
>
> [0:24:11.415660787] [10635] DEBUG HAL camera_device.cpp:941 '\_SB_.PCI0.I2C2.CAM0': Queueing request 140736750235824 with 2 streams
> [0:24:11.415858538] [10635] DEBUG Buffer framebuffer.cpp:249 Buffer is not contiguous
> [0:24:11.415905500] [10635] DEBUG HAL camera_device.cpp:980 '\_SB_.PCI0.I2C2.CAM0': 0 - (4160x3104)[0x00000023] -> (4160x3104)[NV12] (direct)
> [0:24:11.415943620] [10635] DEBUG HAL camera_device.cpp:966 '\_SB_.PCI0.I2C2.CAM0': 1 - (4160x3104)[0x00000021] -> (4160x3104)[NV12] (mapped)
> [0:24:11.416461639] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video0[37:cap]: Queueing buffer 3
> [0:24:11.428903141] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video0[37:cap]: Dequeuing buffer 2
> [0:24:11.429095316] [10643] DEBUG IPU3Awb awb.cpp:331 Color temperature estimated: 6644
> [0:24:11.429563781] [10629] DEBUG DelayedControls delayed_controls.cpp:237 frame 193 started
> [0:24:11.429627478] [10629] DEBUG DelayedControls delayed_controls.cpp:272 Setting Exposure to 269 at index 193
> [0:24:11.429683048] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video6[25:cap]: Queueing buffer 1
> [0:24:11.436615191] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video5[27:out]: Queueing buffer 2
> [0:24:11.436718629] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video8[28:cap]: Queueing buffer 2
> [0:24:11.436799420] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video4[24:out]: Queueing buffer 2
> [0:24:11.457388821] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video4[24:out]: Dequeuing buffer 1
> [0:24:11.457948159] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video6[25:cap]: Dequeuing buffer 0
> [0:24:11.458257692] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video5[27:out]: Dequeuing buffer 2
> [0:24:11.458525315] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video8[28:cap]: Dequeuing buffer 1
> [0:24:11.458968848] [10643] DEBUG IPU3Awb awb.cpp:270 Valid zones: 186
> [0:24:11.459005634] [10643] DEBUG IPU3Awb awb.cpp:224 Grey world AWB
> [0:24:11.459082715] [10643] DEBUG IPU3Awb awb.cpp:273 Gain found for red: 1.16864 and for blue: 1.14863
> [0:24:11.459325998] [10629] DEBUG DelayedControls delayed_controls.cpp:179 Queuing Exposure to 269 at index 194
> [0:24:11.459562485] [10629] DEBUG DelayedControls delayed_controls.cpp:179 Queuing Analogue Gain to 477 at index 194
> [0:24:11.459815347] [10629] DEBUG Request request.cpp:291 Request(129:C:0/1:140736750229136)
> [0:24:11.460390126] [10629] DEBUG HAL camera_device.cpp:1117 '\_SB_.PCI0.I2C2.CAM0': Request 140736750229136 completed with 1 streams
> [0:24:11.461855854] [10635] DEBUG Request request.cpp:94 Created request - cookie: 140736750230064
> [0:24:11.461939858] [10635] DEBUG HAL camera_device.cpp:941 '\_SB_.PCI0.I2C2.CAM0': Queueing request 140736750230064 with 1 streams
> [0:24:11.462057435] [10635] DEBUG Buffer framebuffer.cpp:249 Buffer is not contiguous
> [0:24:11.462098906] [10635] DEBUG HAL camera_device.cpp:980 '\_SB_.PCI0.I2C2.CAM0': 0 - (4160x3104)[0x00000023] -> (4160x3104)[NV12] (direct)
> [0:24:11.463164722] [10629] DEBUG DelayedControls delayed_controls.cpp:237 frame 194 started
> [0:24:11.463730946] [10629] DEBUG DelayedControls delayed_controls.cpp:272 Setting Analogue Gain to 477 at index 193
> [0:24:11.464033686] [10629] DEBUG DelayedControls delayed_controls.cpp:272 Setting Exposure to 269 at index 194
> [0:24:11.464329869] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video0[37:cap]: Queueing buffer 0
> [0:24:11.495739092] [10629] DEBUG DelayedControls delayed_controls.cpp:237 frame 195 started
> [0:24:11.496489311] [10629] DEBUG DelayedControls delayed_controls.cpp:272 Setting Analogue Gain to 477 at index 194
> [0:24:11.496737385] [10629] DEBUG DelayedControls delayed_controls.cpp:285 Queue is empty, auto queue no-op.
> [0:24:11.497044311] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video0[37:cap]: Dequeuing buffer 3
> [0:24:11.497335155] [10643] DEBUG IPU3Awb awb.cpp:331 Color temperature estimated: 6658
> [0:24:11.497528845] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video6[25:cap]: Queueing buffer 2
> [0:24:11.503589322] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video5[27:out]: Queueing buffer 3
> [0:24:11.504027344] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video8[28:cap]: Queueing buffer 3
> [0:24:11.504287330] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video4[24:out]: Queueing buffer 3
> [0:24:11.504712501] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video4[24:out]: Dequeuing buffer 2
> [0:24:11.505005096] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video6[25:cap]: Dequeuing buffer 1
> [0:24:11.505260331] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video5[27:out]: Dequeuing buffer 3
> [0:24:11.505506837] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video8[28:cap]: Dequeuing buffer 2
> [0:24:11.505940926] [10643] DEBUG IPU3Awb awb.cpp:270 Valid zones: 186
> [0:24:11.505976974] [10643] DEBUG IPU3Awb awb.cpp:224 Grey world AWB
> [0:24:11.506057427] [10643] DEBUG IPU3Awb awb.cpp:273 Gain found for red: 1.16797 and for blue: 1.14978
> [0:24:11.506297848] [10629] DEBUG DelayedControls delayed_controls.cpp:179 Queuing Exposure to 269 at index 196
> [0:24:11.506547021] [10629] DEBUG DelayedControls delayed_controls.cpp:179 Queuing Analogue Gain to 477 at index 196
> [0:24:11.506796456] [10629] DEBUG Request request.cpp:291 Request(130:C:0/1:140736750225984)
> [0:24:11.507374522] [10629] DEBUG HAL camera_device.cpp:1117 '\_SB_.PCI0.I2C2.CAM0': Request 140736750225984 completed with 1 streams
> [0:24:11.509426987] [10635] DEBUG Request request.cpp:94 Created request - cookie: 140736750227664
> [0:24:11.509507039] [10635] DEBUG HAL camera_device.cpp:941 '\_SB_.PCI0.I2C2.CAM0': Queueing request 140736750227664 with 1 streams
> [0:24:11.509630706] [10635] DEBUG Buffer framebuffer.cpp:249 Buffer is not contiguous
> [0:24:11.509675200] [10635] DEBUG HAL camera_device.cpp:980 '\_SB_.PCI0.I2C2.CAM0': 0 - (4160x3104)[0x00000023] -> (4160x3104)[NV12] (direct)
> [0:24:11.511309605] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video0[37:cap]: Queueing buffer 1
> [0:24:11.528984601] [10629] DEBUG DelayedControls delayed_controls.cpp:237 frame 196 started
> [0:24:11.529446048] [10629] DEBUG DelayedControls delayed_controls.cpp:272 Setting Exposure to 269 at index 196
> [0:24:11.548431632] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video4[24:out]: Dequeuing buffer 3
> [0:24:11.548993830] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video6[25:cap]: Dequeuing buffer 2
> [0:24:11.549320802] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video8[28:cap]: Dequeuing buffer 3
> [0:24:11.549807736] [10643] DEBUG IPU3Awb awb.cpp:270 Valid zones: 186
> [0:24:11.549846777] [10643] DEBUG IPU3Awb awb.cpp:224 Grey world AWB
> [0:24:11.549927350] [10643] DEBUG IPU3Awb awb.cpp:273 Gain found for red: 1.16841 and for blue: 1.14912
> [0:24:11.550170924] [10629] DEBUG DelayedControls delayed_controls.cpp:179 Queuing Exposure to 269 at index 197
> [0:24:11.550405629] [10629] DEBUG DelayedControls delayed_controls.cpp:179 Queuing Analogue Gain to 477 at index 197
> [0:24:11.550646460] [10629] DEBUG Request request.cpp:291 Request(131:C:0/1:140736750235824)
> [0:24:11.551245104] [10629] DEBUG HAL camera_device.cpp:1117 '\_SB_.PCI0.I2C2.CAM0': Request 140736750235824 completed with 2 streams
> [0:24:11.562307680] [10629] DEBUG EXIF exif.cpp:522 Created EXIF instance (536 bytes)
> [0:24:11.634800113] [10629] DEBUG JPEG encoder_libjpeg.cpp:220 JPEG Encode Starting:4160x3104
>
> Thread 6 "CameraModuleThr" received signal SIGSEGV, Segmentation fault.
> [Switching to Thread 0x7fffe7fff640 (LWP 10629)]
> 0x00007ffff597ead5 in EncoderLibJpeg::compressNV(libcamera::Span<unsigned char const, 18446744073709551615ul>) () from /usr/lib64/camera_hal/libcamera-hal.so
Not nice :-S Could you please print frame.size() here ?
> (gdb) bt
> #0 0x00007ffff597ead5 in EncoderLibJpeg::compressNV(libcamera::Span<unsigned char const, 18446744073709551615ul>) () from /usr/lib64/camera_hal/libcamera-hal.so
> #1 0x00007ffff597eda0 in EncoderLibJpeg::encode(libcamera::Span<unsigned char const, 18446744073709551615ul>, libcamera::Span<unsigned char, 18446744073709551615ul>, libcamera::Span<unsigned char const, 18446744073709551615ul>, unsigned int) () from /usr/lib64/camera_hal/libcamera-hal.so
> #2 0x00007ffff597ec53 in EncoderLibJpeg::encode(libcamera::FrameBuffer const&, libcamera::Span<unsigned char, 18446744073709551615ul>, libcamera::Span<unsigned char const, 18446744073709551615ul>, unsigned int) () from /usr/lib64/camera_hal/libcamera-hal.so
> #3 0x00007ffff5981871 in PostProcessorJpeg::process(libcamera::FrameBuffer const&, CameraBuffer*, CameraMetadata const&, CameraMetadata*) () from /usr/lib64/camera_hal/libcamera-hal.so
> #4 0x00007ffff597dd8d in CameraStream::process(libcamera::FrameBuffer const&, native_handle const*, CameraMetadata const&, CameraMetadata*) () from /usr/lib64/camera_hal/libcamera-hal.so
> #5 0x00007ffff59717b9 in CameraDevice::requestComplete(libcamera::Request*) () from /usr/lib64/camera_hal/libcamera-hal.so
> #6 0x00007ffff5979a7c in libcamera::BoundMethodMember<CameraDevice, void, libcamera::Request*>::activate(libcamera::Request*, bool) () from /usr/lib64/camera_hal/libcamera-hal.so
> #7 0x00007ffff58f9051 in libcamera::Signal<libcamera::IPCMessage const&>::emit(libcamera::IPCMessage const&) () from /usr/lib64/libcamera.so
> #8 0x00007ffff58f8fe8 in libcamera::Camera::requestComplete(libcamera::Request*) () from /usr/lib64/libcamera.so
> #9 0x00007ffff591f28a in libcamera::PipelineHandler::completeRequest(libcamera::Request*) () from /usr/lib64/libcamera.so
> #10 0x00007ffff5935cef in libcamera::IPU3CameraData::queueFrameAction(unsigned int, libcamera::ipa::ipu3::IPU3Action const&) () from /usr/lib64/libcamera.so
> #11 0x00007ffff58f0b02 in libcamera::BoundMethodMember<libcamera::ipa::ipu3::IPAProxyIPU3, void, unsigned int, libcamera::ipa::ipu3::IPU3Action const&>::activate(unsigned int, libcamera::ipa::ipu3::IPU3Action const&, bool) () from /usr/lib64/libcamera.so
> #12 0x00007ffff58edf81 in libcamera::Signal<unsigned int, libcamera::ipa::ipu3::IPU3Action const&>::emit(unsigned int, libcamera::ipa::ipu3::IPU3Action const&) () from /usr/lib64/libcamera.so
> #13 0x00007ffff58eb5b4 in libcamera::ipa::ipu3::IPAProxyIPU3::queueFrameActionThread(unsigned int, libcamera::ipa::ipu3::IPU3Action const&) () from /usr/lib64/libcamera.so
> #14 0x00007ffff587caed in libcamera::Object::message(libcamera::Message*) () from /usr/lib64/libcamera-base.so
> #15 0x00007ffff587e00b in libcamera::Thread::dispatchMessages(libcamera::Message::Type) () from /usr/lib64/libcamera-base.so
> #16 0x00007ffff5876925 in libcamera::EventDispatcherPoll::processEvents() () from /usr/lib64/libcamera-base.so
> #17 0x00007ffff587d6e2 in libcamera::Thread::exec() () from /usr/lib64/libcamera-base.so
> #18 0x00007ffff58fb582 in libcamera::CameraManager::Private::run() () from /usr/lib64/libcamera.so
> #19 0x00007ffff587e43c in ?? () from /usr/lib64/libcamera-base.so
> #20 0x0000000000000000 in ?? ()
>
> > The most important change is in patches 13/27 and 14/27, which translate
> > between V4L2 buffers and libcamera FrameBuffer to handle the case where
> > a multi-planar frame buffer is used with the V4L2 single-planar API.
> > It's working more or less by chance at the moment (except in qcam where
> > it's broken, and possibly in other places I haven't tested). Patches
> > 01/27 to 12/27 are cleanups and additions to prepare for the work in
> > V4L2VideoDevice, and patch 15/27 is a small cleanup on top. Patches
> > 16/27 and 17/27 then improve the FrameBuffer class API as a cleanup.
> >
> > Patches 18/27 to 27/27 fix the cam and qcam applications, as well as an
> > issue in the Android HAL. Worth being noted is patch 19/27 that
> > introduces an Image class shared by cam and qcam. The class duplicates
> > the MappedFrameBuffer implementation private to libcamera. I've tried to
> > rework MappedFrameBuffer into something I would be happy to see in the
> > public API, but failed to do so in a reasonable amount of time, and I
> > didn't want to delay this important regression fix.
> >
> > This series doesn't break any unit test, as vimc doesn't support NV12.
> > Addition of NV12 support to the vimc kernel driver would be very nice,
> > in order to test multi-planar support in our unit tests. Volunteers are
> > welcome ;-)
> >
> > Laurent Pinchart (27):
> > libcamera: base: utils: Use size_t for index in utils::enumerate()
> > libcamera: file_descriptor: Add a function to retrieve the inode
> > libcamera: v4l2_videodevice: Drop toV4L2PixelFormat()
> > libcamera: Use V4L2PixelFormat::fromPixelFormat()
> > libcamera: formats: Move plane info structure to PixelFormatInfo
> > libcamera: formats: Add planeSize() helpers to PixelFormatInfo
> > libcamera: formats: Support V4L2 non-contiguous formats
> > libcamera: framebuffer: Move planes check to constructor
> > libcamera: framebuffer: Add a function to check if planes are
> > contiguous
> > libcamera: v4l2_videodevice: Cache PixelFormatInfo
> > libcamera: v4l2_videodevice: Document plane handling in createBuffer()
> > libcamera: v4l2_videodevice: Take stride into account to compute
> > offsets
> > libcamera: v4l2_videodevice: Coalesce planes when queuing buffer
> > libcamera: v4l2_videodevice: Split planes when dequeuing buffer
> > libcamera: v4l2_videodevice: Use utils::enumerate()
> > libcamera: framebuffer: Allocate metadata planes at construction time
> > libcamera: framebuffer: Prevent modifying the number of metadata
> > planes
> > android: camera_device: Don't assume all planes use the same fd
> > cam: Add Image class
> > cam: file_sink: Use Image class to access pixel data
> > cam: drm: Support per-plane stride values
> > cam: drm: Set per-plane offsets when creating DRM frame buffer
> > cam: drm: Avoid importing the same dmabuf multiple times
> > qcam: Print bytesused for all planes
> > qcam: Use Image class to access pixel data
> > qcam: viewfinder_gl: Support multi-planar buffers
> > qcam: viewfinder_qt: Support multi-planar buffers
> >
> > include/libcamera/base/utils.h | 4 +-
> > include/libcamera/file_descriptor.h | 3 +
> > include/libcamera/framebuffer.h | 19 +-
> > include/libcamera/internal/formats.h | 22 +-
> > include/libcamera/internal/framebuffer.h | 2 +
> > include/libcamera/internal/v4l2_pixelformat.h | 2 +-
> > include/libcamera/internal/v4l2_videodevice.h | 3 +-
> > src/android/camera_device.cpp | 25 +-
> > src/android/mm/generic_camera_buffer.cpp | 11 +-
> > src/android/yuv/post_processor_yuv.cpp | 10 +-
> > src/cam/camera_session.cpp | 4 +-
> > src/cam/drm.cpp | 38 +-
> > src/cam/drm.h | 7 +-
> > src/cam/file_sink.cpp | 44 +--
> > src/cam/file_sink.h | 6 +-
> > src/cam/image.cpp | 107 +++++
> > src/cam/image.h | 52 +++
> > src/cam/kms_sink.cpp | 28 +-
> > src/cam/meson.build | 1 +
> > src/libcamera/file_descriptor.cpp | 26 ++
> > src/libcamera/formats.cpp | 373 ++++++++++++++----
> > src/libcamera/framebuffer.cpp | 57 ++-
> > src/libcamera/pipeline/ipu3/cio2.cpp | 4 +-
> > src/libcamera/pipeline/ipu3/imgu.cpp | 2 +-
> > .../pipeline/raspberrypi/raspberrypi.cpp | 8 +-
> > src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 6 +-
> > src/libcamera/pipeline/simple/converter.cpp | 8 +-
> > src/libcamera/pipeline/simple/simple.cpp | 4 +-
> > src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 6 +-
> > src/libcamera/pipeline/vimc/vimc.cpp | 8 +-
> > src/libcamera/v4l2_pixelformat.cpp | 11 +-
> > src/libcamera/v4l2_videodevice.cpp | 196 ++++++---
> > src/qcam/format_converter.cpp | 18 +-
> > src/qcam/format_converter.h | 9 +-
> > src/qcam/main_window.cpp | 38 +-
> > src/qcam/main_window.h | 4 +-
> > src/qcam/meson.build | 1 +
> > src/qcam/viewfinder.h | 6 +-
> > src/qcam/viewfinder_gl.cpp | 45 +--
> > src/qcam/viewfinder_gl.h | 4 +-
> > src/qcam/viewfinder_qt.cpp | 20 +-
> > src/qcam/viewfinder_qt.h | 2 +-
> > src/v4l2/v4l2_camera_proxy.cpp | 11 +-
> > test/libtest/buffer_source.cpp | 3 +-
> > test/utils.cpp | 10 +-
> > 45 files changed, 911 insertions(+), 357 deletions(-)
> > create mode 100644 src/cam/image.cpp
> > create mode 100644 src/cam/image.h
--
Regards,
Laurent Pinchart
More information about the libcamera-devel
mailing list