[libcamera-devel] [PATCH v2 00/27] libcamera: Handle fallout of FrameBuffer offset support
Umang Jain
umang.jain at ideasonboard.com
Mon Sep 6 14:54:21 CEST 2021
Hi Laurent,
On 9/6/21 5:54 PM, Laurent Pinchart wrote:
> Hi Umang,
>
> On Mon, Sep 06, 2021 at 03:38:04PM +0530, Umang Jain wrote:
>> On 9/6/21 7:30 AM, Laurent Pinchart wrote:
>>> Hello everybody,
>>>
>>> This patch series started as an investigation of a qcam failure with
>>> IPU3 after the merge of the FrameBuffer offset support. While a hack in
>>> qcam would be possible, I decided to instead address the core issue and
>>> fix it in V4L2VideoDevice.
>>>
>>> Compared to v1, the series now includes fixes for cam and qcam in
>>> addition to the changes needed in the libcamera core. They have been
>>> tested with the Raspberry Pi, IPU3, VIMC and UVC pipeline handlers.
>>>
>>> The GStreamer element seems to work fine without any change required.
>>> The V4L2 compatibility layer is still broken, and I haven't tested the
>>> Android HAL yet (any volunteer ?).
>> I am applied this series on top of master and tested on nautilus.
>> Streaming seems fine to me, although it was appearing more choppy than
>> usual.
>>
>> Now that I see, entire nautilus seems laggy to respond in general. One
>> "Chrome" process hogging 50% CPU, seems like a transient issue.
>>
>> So, streaming seems fine on multiple runs, with multiple resolutions
>> selections (from the camera-app menu), however, requesting MJPEG
>> (shutter-click) results in a segfault.
> Thanks for testing !
>
>> The stack trace is below:
>>
>> [0:24:11.415660787] [10635] DEBUG HAL camera_device.cpp:941 '\_SB_.PCI0.I2C2.CAM0': Queueing request 140736750235824 with 2 streams
>> [0:24:11.415858538] [10635] DEBUG Buffer framebuffer.cpp:249 Buffer is not contiguous
>> [0:24:11.415905500] [10635] DEBUG HAL camera_device.cpp:980 '\_SB_.PCI0.I2C2.CAM0': 0 - (4160x3104)[0x00000023] -> (4160x3104)[NV12] (direct)
>> [0:24:11.415943620] [10635] DEBUG HAL camera_device.cpp:966 '\_SB_.PCI0.I2C2.CAM0': 1 - (4160x3104)[0x00000021] -> (4160x3104)[NV12] (mapped)
>> [0:24:11.416461639] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video0[37:cap]: Queueing buffer 3
>> [0:24:11.428903141] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video0[37:cap]: Dequeuing buffer 2
>> [0:24:11.429095316] [10643] DEBUG IPU3Awb awb.cpp:331 Color temperature estimated: 6644
>> [0:24:11.429563781] [10629] DEBUG DelayedControls delayed_controls.cpp:237 frame 193 started
>> [0:24:11.429627478] [10629] DEBUG DelayedControls delayed_controls.cpp:272 Setting Exposure to 269 at index 193
>> [0:24:11.429683048] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video6[25:cap]: Queueing buffer 1
>> [0:24:11.436615191] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video5[27:out]: Queueing buffer 2
>> [0:24:11.436718629] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video8[28:cap]: Queueing buffer 2
>> [0:24:11.436799420] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video4[24:out]: Queueing buffer 2
>> [0:24:11.457388821] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video4[24:out]: Dequeuing buffer 1
>> [0:24:11.457948159] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video6[25:cap]: Dequeuing buffer 0
>> [0:24:11.458257692] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video5[27:out]: Dequeuing buffer 2
>> [0:24:11.458525315] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video8[28:cap]: Dequeuing buffer 1
>> [0:24:11.458968848] [10643] DEBUG IPU3Awb awb.cpp:270 Valid zones: 186
>> [0:24:11.459005634] [10643] DEBUG IPU3Awb awb.cpp:224 Grey world AWB
>> [0:24:11.459082715] [10643] DEBUG IPU3Awb awb.cpp:273 Gain found for red: 1.16864 and for blue: 1.14863
>> [0:24:11.459325998] [10629] DEBUG DelayedControls delayed_controls.cpp:179 Queuing Exposure to 269 at index 194
>> [0:24:11.459562485] [10629] DEBUG DelayedControls delayed_controls.cpp:179 Queuing Analogue Gain to 477 at index 194
>> [0:24:11.459815347] [10629] DEBUG Request request.cpp:291 Request(129:C:0/1:140736750229136)
>> [0:24:11.460390126] [10629] DEBUG HAL camera_device.cpp:1117 '\_SB_.PCI0.I2C2.CAM0': Request 140736750229136 completed with 1 streams
>> [0:24:11.461855854] [10635] DEBUG Request request.cpp:94 Created request - cookie: 140736750230064
>> [0:24:11.461939858] [10635] DEBUG HAL camera_device.cpp:941 '\_SB_.PCI0.I2C2.CAM0': Queueing request 140736750230064 with 1 streams
>> [0:24:11.462057435] [10635] DEBUG Buffer framebuffer.cpp:249 Buffer is not contiguous
>> [0:24:11.462098906] [10635] DEBUG HAL camera_device.cpp:980 '\_SB_.PCI0.I2C2.CAM0': 0 - (4160x3104)[0x00000023] -> (4160x3104)[NV12] (direct)
>> [0:24:11.463164722] [10629] DEBUG DelayedControls delayed_controls.cpp:237 frame 194 started
>> [0:24:11.463730946] [10629] DEBUG DelayedControls delayed_controls.cpp:272 Setting Analogue Gain to 477 at index 193
>> [0:24:11.464033686] [10629] DEBUG DelayedControls delayed_controls.cpp:272 Setting Exposure to 269 at index 194
>> [0:24:11.464329869] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video0[37:cap]: Queueing buffer 0
>> [0:24:11.495739092] [10629] DEBUG DelayedControls delayed_controls.cpp:237 frame 195 started
>> [0:24:11.496489311] [10629] DEBUG DelayedControls delayed_controls.cpp:272 Setting Analogue Gain to 477 at index 194
>> [0:24:11.496737385] [10629] DEBUG DelayedControls delayed_controls.cpp:285 Queue is empty, auto queue no-op.
>> [0:24:11.497044311] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video0[37:cap]: Dequeuing buffer 3
>> [0:24:11.497335155] [10643] DEBUG IPU3Awb awb.cpp:331 Color temperature estimated: 6658
>> [0:24:11.497528845] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video6[25:cap]: Queueing buffer 2
>> [0:24:11.503589322] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video5[27:out]: Queueing buffer 3
>> [0:24:11.504027344] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video8[28:cap]: Queueing buffer 3
>> [0:24:11.504287330] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video4[24:out]: Queueing buffer 3
>> [0:24:11.504712501] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video4[24:out]: Dequeuing buffer 2
>> [0:24:11.505005096] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video6[25:cap]: Dequeuing buffer 1
>> [0:24:11.505260331] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video5[27:out]: Dequeuing buffer 3
>> [0:24:11.505506837] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video8[28:cap]: Dequeuing buffer 2
>> [0:24:11.505940926] [10643] DEBUG IPU3Awb awb.cpp:270 Valid zones: 186
>> [0:24:11.505976974] [10643] DEBUG IPU3Awb awb.cpp:224 Grey world AWB
>> [0:24:11.506057427] [10643] DEBUG IPU3Awb awb.cpp:273 Gain found for red: 1.16797 and for blue: 1.14978
>> [0:24:11.506297848] [10629] DEBUG DelayedControls delayed_controls.cpp:179 Queuing Exposure to 269 at index 196
>> [0:24:11.506547021] [10629] DEBUG DelayedControls delayed_controls.cpp:179 Queuing Analogue Gain to 477 at index 196
>> [0:24:11.506796456] [10629] DEBUG Request request.cpp:291 Request(130:C:0/1:140736750225984)
>> [0:24:11.507374522] [10629] DEBUG HAL camera_device.cpp:1117 '\_SB_.PCI0.I2C2.CAM0': Request 140736750225984 completed with 1 streams
>> [0:24:11.509426987] [10635] DEBUG Request request.cpp:94 Created request - cookie: 140736750227664
>> [0:24:11.509507039] [10635] DEBUG HAL camera_device.cpp:941 '\_SB_.PCI0.I2C2.CAM0': Queueing request 140736750227664 with 1 streams
>> [0:24:11.509630706] [10635] DEBUG Buffer framebuffer.cpp:249 Buffer is not contiguous
>> [0:24:11.509675200] [10635] DEBUG HAL camera_device.cpp:980 '\_SB_.PCI0.I2C2.CAM0': 0 - (4160x3104)[0x00000023] -> (4160x3104)[NV12] (direct)
>> [0:24:11.511309605] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1582 /dev/video0[37:cap]: Queueing buffer 1
>> [0:24:11.528984601] [10629] DEBUG DelayedControls delayed_controls.cpp:237 frame 196 started
>> [0:24:11.529446048] [10629] DEBUG DelayedControls delayed_controls.cpp:272 Setting Exposure to 269 at index 196
>> [0:24:11.548431632] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video4[24:out]: Dequeuing buffer 3
>> [0:24:11.548993830] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video6[25:cap]: Dequeuing buffer 2
>> [0:24:11.549320802] [10629] DEBUG V4L2 v4l2_videodevice.cpp:1650 /dev/video8[28:cap]: Dequeuing buffer 3
>> [0:24:11.549807736] [10643] DEBUG IPU3Awb awb.cpp:270 Valid zones: 186
>> [0:24:11.549846777] [10643] DEBUG IPU3Awb awb.cpp:224 Grey world AWB
>> [0:24:11.549927350] [10643] DEBUG IPU3Awb awb.cpp:273 Gain found for red: 1.16841 and for blue: 1.14912
>> [0:24:11.550170924] [10629] DEBUG DelayedControls delayed_controls.cpp:179 Queuing Exposure to 269 at index 197
>> [0:24:11.550405629] [10629] DEBUG DelayedControls delayed_controls.cpp:179 Queuing Analogue Gain to 477 at index 197
>> [0:24:11.550646460] [10629] DEBUG Request request.cpp:291 Request(131:C:0/1:140736750235824)
>> [0:24:11.551245104] [10629] DEBUG HAL camera_device.cpp:1117 '\_SB_.PCI0.I2C2.CAM0': Request 140736750235824 completed with 2 streams
>> [0:24:11.562307680] [10629] DEBUG EXIF exif.cpp:522 Created EXIF instance (536 bytes)
>> [0:24:11.634800113] [10629] DEBUG JPEG encoder_libjpeg.cpp:220 JPEG Encode Starting:4160x3104
>>
>> Thread 6 "CameraModuleThr" received signal SIGSEGV, Segmentation fault.
>> [Switching to Thread 0x7fffe7fff640 (LWP 10629)]
>> 0x00007ffff597ead5 in EncoderLibJpeg::compressNV(libcamera::Span<unsigned char const, 18446744073709551615ul>) () from /usr/lib64/camera_hal/libcamera-hal.so
> Not nice :-S Could you please print frame.size() here ?
[3:11:40.303057525] [25328] DEBUG EXIF exif.cpp:522 Created EXIF
instance (536 bytes)
[3:11:40.368109082] [25328] DEBUG JPEG encoder_libjpeg.cpp:222 JPEG
Encode Starting:4160x3104
[3:11:40.368158138] [25328] DEBUG JPEG encoder_libjpeg.cpp:128 Frame
size : 12912640
Segmentation fault (core dumped)
>
>> (gdb) bt
>> #0 0x00007ffff597ead5 in EncoderLibJpeg::compressNV(libcamera::Span<unsigned char const, 18446744073709551615ul>) () from /usr/lib64/camera_hal/libcamera-hal.so
>> #1 0x00007ffff597eda0 in EncoderLibJpeg::encode(libcamera::Span<unsigned char const, 18446744073709551615ul>, libcamera::Span<unsigned char, 18446744073709551615ul>, libcamera::Span<unsigned char const, 18446744073709551615ul>, unsigned int) () from /usr/lib64/camera_hal/libcamera-hal.so
>> #2 0x00007ffff597ec53 in EncoderLibJpeg::encode(libcamera::FrameBuffer const&, libcamera::Span<unsigned char, 18446744073709551615ul>, libcamera::Span<unsigned char const, 18446744073709551615ul>, unsigned int) () from /usr/lib64/camera_hal/libcamera-hal.so
>> #3 0x00007ffff5981871 in PostProcessorJpeg::process(libcamera::FrameBuffer const&, CameraBuffer*, CameraMetadata const&, CameraMetadata*) () from /usr/lib64/camera_hal/libcamera-hal.so
>> #4 0x00007ffff597dd8d in CameraStream::process(libcamera::FrameBuffer const&, native_handle const*, CameraMetadata const&, CameraMetadata*) () from /usr/lib64/camera_hal/libcamera-hal.so
>> #5 0x00007ffff59717b9 in CameraDevice::requestComplete(libcamera::Request*) () from /usr/lib64/camera_hal/libcamera-hal.so
>> #6 0x00007ffff5979a7c in libcamera::BoundMethodMember<CameraDevice, void, libcamera::Request*>::activate(libcamera::Request*, bool) () from /usr/lib64/camera_hal/libcamera-hal.so
>> #7 0x00007ffff58f9051 in libcamera::Signal<libcamera::IPCMessage const&>::emit(libcamera::IPCMessage const&) () from /usr/lib64/libcamera.so
>> #8 0x00007ffff58f8fe8 in libcamera::Camera::requestComplete(libcamera::Request*) () from /usr/lib64/libcamera.so
>> #9 0x00007ffff591f28a in libcamera::PipelineHandler::completeRequest(libcamera::Request*) () from /usr/lib64/libcamera.so
>> #10 0x00007ffff5935cef in libcamera::IPU3CameraData::queueFrameAction(unsigned int, libcamera::ipa::ipu3::IPU3Action const&) () from /usr/lib64/libcamera.so
>> #11 0x00007ffff58f0b02 in libcamera::BoundMethodMember<libcamera::ipa::ipu3::IPAProxyIPU3, void, unsigned int, libcamera::ipa::ipu3::IPU3Action const&>::activate(unsigned int, libcamera::ipa::ipu3::IPU3Action const&, bool) () from /usr/lib64/libcamera.so
>> #12 0x00007ffff58edf81 in libcamera::Signal<unsigned int, libcamera::ipa::ipu3::IPU3Action const&>::emit(unsigned int, libcamera::ipa::ipu3::IPU3Action const&) () from /usr/lib64/libcamera.so
>> #13 0x00007ffff58eb5b4 in libcamera::ipa::ipu3::IPAProxyIPU3::queueFrameActionThread(unsigned int, libcamera::ipa::ipu3::IPU3Action const&) () from /usr/lib64/libcamera.so
>> #14 0x00007ffff587caed in libcamera::Object::message(libcamera::Message*) () from /usr/lib64/libcamera-base.so
>> #15 0x00007ffff587e00b in libcamera::Thread::dispatchMessages(libcamera::Message::Type) () from /usr/lib64/libcamera-base.so
>> #16 0x00007ffff5876925 in libcamera::EventDispatcherPoll::processEvents() () from /usr/lib64/libcamera-base.so
>> #17 0x00007ffff587d6e2 in libcamera::Thread::exec() () from /usr/lib64/libcamera-base.so
>> #18 0x00007ffff58fb582 in libcamera::CameraManager::Private::run() () from /usr/lib64/libcamera.so
>> #19 0x00007ffff587e43c in ?? () from /usr/lib64/libcamera-base.so
>> #20 0x0000000000000000 in ?? ()
>>
>>> The most important change is in patches 13/27 and 14/27, which translate
>>> between V4L2 buffers and libcamera FrameBuffer to handle the case where
>>> a multi-planar frame buffer is used with the V4L2 single-planar API.
>>> It's working more or less by chance at the moment (except in qcam where
>>> it's broken, and possibly in other places I haven't tested). Patches
>>> 01/27 to 12/27 are cleanups and additions to prepare for the work in
>>> V4L2VideoDevice, and patch 15/27 is a small cleanup on top. Patches
>>> 16/27 and 17/27 then improve the FrameBuffer class API as a cleanup.
>>>
>>> Patches 18/27 to 27/27 fix the cam and qcam applications, as well as an
>>> issue in the Android HAL. Worth being noted is patch 19/27 that
>>> introduces an Image class shared by cam and qcam. The class duplicates
>>> the MappedFrameBuffer implementation private to libcamera. I've tried to
>>> rework MappedFrameBuffer into something I would be happy to see in the
>>> public API, but failed to do so in a reasonable amount of time, and I
>>> didn't want to delay this important regression fix.
>>>
>>> This series doesn't break any unit test, as vimc doesn't support NV12.
>>> Addition of NV12 support to the vimc kernel driver would be very nice,
>>> in order to test multi-planar support in our unit tests. Volunteers are
>>> welcome ;-)
>>>
>>> Laurent Pinchart (27):
>>> libcamera: base: utils: Use size_t for index in utils::enumerate()
>>> libcamera: file_descriptor: Add a function to retrieve the inode
>>> libcamera: v4l2_videodevice: Drop toV4L2PixelFormat()
>>> libcamera: Use V4L2PixelFormat::fromPixelFormat()
>>> libcamera: formats: Move plane info structure to PixelFormatInfo
>>> libcamera: formats: Add planeSize() helpers to PixelFormatInfo
>>> libcamera: formats: Support V4L2 non-contiguous formats
>>> libcamera: framebuffer: Move planes check to constructor
>>> libcamera: framebuffer: Add a function to check if planes are
>>> contiguous
>>> libcamera: v4l2_videodevice: Cache PixelFormatInfo
>>> libcamera: v4l2_videodevice: Document plane handling in createBuffer()
>>> libcamera: v4l2_videodevice: Take stride into account to compute
>>> offsets
>>> libcamera: v4l2_videodevice: Coalesce planes when queuing buffer
>>> libcamera: v4l2_videodevice: Split planes when dequeuing buffer
>>> libcamera: v4l2_videodevice: Use utils::enumerate()
>>> libcamera: framebuffer: Allocate metadata planes at construction time
>>> libcamera: framebuffer: Prevent modifying the number of metadata
>>> planes
>>> android: camera_device: Don't assume all planes use the same fd
>>> cam: Add Image class
>>> cam: file_sink: Use Image class to access pixel data
>>> cam: drm: Support per-plane stride values
>>> cam: drm: Set per-plane offsets when creating DRM frame buffer
>>> cam: drm: Avoid importing the same dmabuf multiple times
>>> qcam: Print bytesused for all planes
>>> qcam: Use Image class to access pixel data
>>> qcam: viewfinder_gl: Support multi-planar buffers
>>> qcam: viewfinder_qt: Support multi-planar buffers
>>>
>>> include/libcamera/base/utils.h | 4 +-
>>> include/libcamera/file_descriptor.h | 3 +
>>> include/libcamera/framebuffer.h | 19 +-
>>> include/libcamera/internal/formats.h | 22 +-
>>> include/libcamera/internal/framebuffer.h | 2 +
>>> include/libcamera/internal/v4l2_pixelformat.h | 2 +-
>>> include/libcamera/internal/v4l2_videodevice.h | 3 +-
>>> src/android/camera_device.cpp | 25 +-
>>> src/android/mm/generic_camera_buffer.cpp | 11 +-
>>> src/android/yuv/post_processor_yuv.cpp | 10 +-
>>> src/cam/camera_session.cpp | 4 +-
>>> src/cam/drm.cpp | 38 +-
>>> src/cam/drm.h | 7 +-
>>> src/cam/file_sink.cpp | 44 +--
>>> src/cam/file_sink.h | 6 +-
>>> src/cam/image.cpp | 107 +++++
>>> src/cam/image.h | 52 +++
>>> src/cam/kms_sink.cpp | 28 +-
>>> src/cam/meson.build | 1 +
>>> src/libcamera/file_descriptor.cpp | 26 ++
>>> src/libcamera/formats.cpp | 373 ++++++++++++++----
>>> src/libcamera/framebuffer.cpp | 57 ++-
>>> src/libcamera/pipeline/ipu3/cio2.cpp | 4 +-
>>> src/libcamera/pipeline/ipu3/imgu.cpp | 2 +-
>>> .../pipeline/raspberrypi/raspberrypi.cpp | 8 +-
>>> src/libcamera/pipeline/rkisp1/rkisp1_path.cpp | 6 +-
>>> src/libcamera/pipeline/simple/converter.cpp | 8 +-
>>> src/libcamera/pipeline/simple/simple.cpp | 4 +-
>>> src/libcamera/pipeline/uvcvideo/uvcvideo.cpp | 6 +-
>>> src/libcamera/pipeline/vimc/vimc.cpp | 8 +-
>>> src/libcamera/v4l2_pixelformat.cpp | 11 +-
>>> src/libcamera/v4l2_videodevice.cpp | 196 ++++++---
>>> src/qcam/format_converter.cpp | 18 +-
>>> src/qcam/format_converter.h | 9 +-
>>> src/qcam/main_window.cpp | 38 +-
>>> src/qcam/main_window.h | 4 +-
>>> src/qcam/meson.build | 1 +
>>> src/qcam/viewfinder.h | 6 +-
>>> src/qcam/viewfinder_gl.cpp | 45 +--
>>> src/qcam/viewfinder_gl.h | 4 +-
>>> src/qcam/viewfinder_qt.cpp | 20 +-
>>> src/qcam/viewfinder_qt.h | 2 +-
>>> src/v4l2/v4l2_camera_proxy.cpp | 11 +-
>>> test/libtest/buffer_source.cpp | 3 +-
>>> test/utils.cpp | 10 +-
>>> 45 files changed, 911 insertions(+), 357 deletions(-)
>>> create mode 100644 src/cam/image.cpp
>>> create mode 100644 src/cam/image.h
More information about the libcamera-devel
mailing list