[libcamera-devel] raspberrypi pipeline gstreamer bug

[Naushir Patuck]

Hi,

On Wed, 10 Jun 2020 at 12:22, Naushir Patuck <naush at raspberrypi.com> wrote:
>
> Hi Xishan and Kieran,
>
> On Wed, 10 Jun 2020 at 11:26, Kieran Bingham
> <kieran.bingham at ideasonboard.com> wrote:
> >
> > Hi Xishan, (Adding Naush)
> >
> > On 10/06/2020 10:57, Xishan Sun wrote:
> > >
> > > Trying to use gstreamer plugin on raspberrypi platform.
> > >
> > >
> > > For very simple test started with
> > >
> > > gst-launch-1.0 libcamerasrc ! fakesink
> > >
> > > will get error:
> > >
> > > Setting pipeline to PAUSED ...
> > > [5:14:54.902707301] [6325]  INFO Camera camera_manager.cpp:277 libcamera
> > > v0.0.0+1423-78150899-dirty
> > > Pipeline is live and does not need PREROLL ...
> > > Setting pipeline to PLAYING ...
> > > ERROR: from element
> > > /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0: Internal data
> > > stream error.
> > > Additional debug info:
> > > ../src/gstreamer/gstlibcamerasrc.cpp(461): gst_libcamera_src_task_enter
> > > (): /GstPipeline:pipeline0/GstLibcameraSrc:libcamerasrc0:
> > > streaming stopped, reason not-negotiated (-4)
> > > Execution ended after 0:00:00.000989404
> > > Setting pipeline to PAUSED ...
> > > Setting pipeline to READY ...
> > > Setting pipeline to NULL ...
> > > Freeing pipeline ...
> > >
> > > debug tracking all the way down to
> > > file src/gstreamer/gstlibcamerasrc.cpp line 377, the function
> > >
> > > g_autoptr(GstCaps) filter =
> > > gst_libcamera_stream_formats_to_caps(stream_cfg.formats());
> > >
> > > will always return empty filter list, since stream_cfg.formats() return
> > > empty.
> > >
> > > further down to the source:
> > > src/libcamera/pipeline/raspberrypi/raspberrypi.cpp line 515, in the
> > > function
> > >
> > > CameraConfiguration *PipelineHandlerRPi::generateConfiguration(Camera
> > > *camera, const StreamRoles &roles)
> > >
> > > there is no configuration for the output format sets. In order to
> > > generate the format sets, I copied the code from uvc pipeline. add it
> > > into line 522
> > >
> > > ....
> > >         if (roles.empty())
> > >                  return config;
> > >
> > >         std::map<V4L2PixelFormat, std::vector<SizeRange>> v4l2Formats =
> > >                  data->isp_[Isp::Output0].dev()->formats();
> > >         std::map<PixelFormat, std::vector<SizeRange>> deviceFormats;
> > >         std::transform(v4l2Formats.begin(), v4l2Formats.end(),
> > >                         std::inserter(deviceFormats, deviceFormats.begin()),
> > >                         [&](const decltype(v4l2Formats)::value_type
> > > &format) {
> > >                                 return decltype(deviceFormats)::value_type{
> > >                                         format.first.toPixelFormat(),
> > >                                         format.second
> > >                                 };
> > >                          });
> >
> > I wish all that code was friendlier to read (from the original) ... But
> > I think that's going to be cleaned up by Jacopo's ImageFormats anyway?
> >
> > >          StreamFormats formats(deviceFormats);
> > >
> > >          for (const StreamRole role : roles) {
> > >                  //StreamConfiguration cfg{};
> > >                  StreamConfiguration cfg(formats);
> > > ....
> > >
> > > it can solve the issue but obviously it is not the best way to do it.
> > > Could you please revise the code here and advice me a better solution?
> >
> > Actually it might be quite close to being the right solution,
> >
> > The raspberrypi StreamConfiguration isn't currently being populated with
> > the 'capabilities' of the stream, - only a single option based on the
> > role given, so applications have no way to know what they 'could' ask
> > for other than the specific choice given by the requested role.
> >
>
> Sorry, was not aware this was expected :)
>
> >
> > Preloading the StreamConfiguration with the formats supported by the ISP
> > is probably the right thing to do indeed, as long as it only exposes
> > things which are supported by the stream.
> >
> > There may be cases (depending on capabilities of the hardware) where
> > this list might have to be filtered depending on the number (or type) of
> > streams requested, but at least initially - populating the
> > StreamConfiguration with the output capabilities of the ISP should get
> > things going again.
> >
> > Naush - Is this something you can look at please?
>
> I think for now the above change by Xishan would be suitable as-is.
> Things do get a bit more restrictive when we are processing two
> streams simultaneously, but the pipeline::validate() will account for
> this and adjust the stream format appropriately.
>
> I will prepare and test a patch.
>
> Regards,
> Naush
>

Sorry forgot to mention, it does not seem that rkisp1 and ipu3
pipeline handlers do this either, so somebody may have to do something
similar there as well.

Naush