[libcamera-devel] How to detect camera after power on the raspberryPi.

Tue Oct 20 17:40:45 CEST 2020

Hi

On Tue, 20 Oct 2020 at 13:59, <tetsuya.nomura at soho-enterprise.com> wrote:
>
> Dear Dave-san
>
> Thank you for your explanation.
> I understand what you are discussing.
>
> I'm now using DS90UB953-Q1 and DS90UB954-Q1 from TI.

There is an RFC (Request For Comments) patchset from last July for
that pair of chips at
https://patchwork.linuxtv.org/project/linux-media/cover/20190723203723.11730-1-luca@lucaceresoli.net/
I haven't looked through the patches to see what is and isn't still
valid, but I seem to recall that the similar GMSL patches have now
been merged which worked on very similar areas for I2C address
mapping.

> I just set several register parameters on them, set the I2C configuration, set the serial speed, number of lanes and speed for MIPI.
> I just send the command by i2cset without using the driver.
> If I can configure the FPD Link III transceiver at the beginning of the boot, Maybe after configure the I2C port on MIPI port then send the I2C commands to transceiver, it will behave as if the sensor board were directly connected to Raspi CSI-2 port.
> I wonder if I can reconfigure the device tree after setting the transceiver registers to run qcam.

You could, but it's always going to be a race condition that needs to
be managed somehow. Blacklisting one or both of the imx219 and
bcm2835-unicam modules, and then manually modprobe'ing them once
you've sent your I2C commands to the FPD chips may work.

Incorporating the serialiser and deserialiser into the pipeline
removes that and is really the correct way of doing things, and the
only one you're likely to get any degree of support implementing (even
though it's the more complex solution). It should also allow for
things like power control to be utilised should the chips have lower
power modes whilst not streaming.

  Dave

> Best Regards,
>
> NOMURA
>
> -----Original Message-----
> From: Dave Stevenson <dave.stevenson at raspberrypi.com>
> Sent: Tuesday, October 20, 2020 9:39 PM
> To: tetsuya.nomura at soho-enterprise.com
> Cc: cc: Kieran Bingham <kieran.bingham at ideasonboard.com>; Laurent Pinchart <laurent.pinchart at ideasonboard.com>; libcamera-devel at lists.libcamera.org; Niklas Söderlund <niklas.soderlund at ragnatech.se>
> Subject: Re: [libcamera-devel] How to detect camera after power on the raspberryPi.
>
> On Tue, 20 Oct 2020 at 12:48, <tetsuya.nomura at soho-enterprise.com> wrote:
> >
> > Dear Sirs
> >
> > Maybe I understand less than 50% of your conversation below, but if
> > you need MIPI 4 lane camera board which can be connected to Raspi CM3+, I can provide you the IMX219 4lane camera board.
>
> Apologies, we've gone off on a slight tangent discussing the logistics of the correct way to implement a multi element chain such as sensor -> FPD Link transceivers -> CSI2 receiver (Unicam).
>
> At present the drivers are only set up for 2 drivers being involved (sensor and receiver), and there are significant issues to resolve to involve three or more.
>
> We already have 2 use cases being discussed for which this Media Controller (MC) API is required, and I have hardware to test out one of them. If we get it right there, and if there is a driver available for your FPD Link III devices, then switching to that at a kernel level should be trivial.
> However it isn't so trivial to integrate that change of architecture into libcamera.
>
> Could you tell us which FPD Link devices it is that you are using?
> Several of those who are directly involved in libcamera have also been involved with writing drivers for GMSL and other CSI2 transceiver chips, so they may well be able to advise on the best route forward on supporting the devices you've chosen.
>
>   Dave
>
> > And if you are interested in controlling the focus, I can also send you the AF type module of IMX219.
> > The other option, IMX378 which is MX477 compatible sensor, with focus driver is also available.
> > https://soho-enterprise.com/2020/08/31/raspi-libcameraqcam-for-imx378/
> >
> > I already tested them on Raspi 3 and 4 with qcam and they works well.
> > I tested the 4 lane mode of IMX219 module with Tinker Edge R, this means you can get full pixel 30fps image.
> > https://soho-enterprise.com/2020/02/27/post-681/
> > I captured high speed RAW image in 4 lane mode as below article.
> > https://soho-enterprise.com/2020/04/10/high-frame-rate-high-speed-came
> > ra-on-tinker-edge-r-testing-mipi-4-lane-feature/
> >
> > And if you could help us to operate our SE-CXB01 Tx/Rx, the FPD LINK
> > III camera extension board, I also can provide the one.
> > (Let me try the method using "dtoverlay" first.)
> > https://soho-enterprise.com/2020/10/19/dual-camera-system-under-develo
> > pment-se-cxb01-tx-rx/
> >
> > I hope my offer will be your help.
> >
> > Best Regards,
> >
> > NOMURA
> >
> > -----Original Message-----
> > From: Kieran Bingham <kieran.bingham at ideasonboard.com>
> > Sent: Tuesday, October 20, 2020 7:50 PM
> > To: Dave Stevenson <dave.stevenson at raspberrypi.com>
> > Cc: Laurent Pinchart <laurent.pinchart at ideasonboard.com>;
> > libcamera-devel at lists.libcamera.org;
> > tetsuya.nomura at soho-enterprise.com; Niklas Söderlund
> > <niklas.soderlund at ragnatech.se>
> > Subject: Re: [libcamera-devel] How to detect camera after power on the raspberryPi.
> >
> > Hi Dave,
> >
> > On 20/10/2020 11:32, Dave Stevenson wrote:
> > > Hi Kieran
> > >
> > > On Tue, 20 Oct 2020 at 10:36, Kieran Bingham
> > > <kieran.bingham at ideasonboard.com> wrote:
> > >>
> > >> Hi Dave,
> > >>
> > >> On 19/10/2020 16:53, Dave Stevenson wrote:
> > >>> Hi Laurent
> > >>>
> > >>> On Mon, 19 Oct 2020 at 13:50, Laurent Pinchart
> > >>> <laurent.pinchart at ideasonboard.com> wrote:
> > >>>>
> > >>>> Hi Dave,
> > >>>>
> > >>>> On Mon, Oct 19, 2020 at 11:10:55AM +0100, Dave Stevenson wrote:
> > >>>>> On Mon, 19 Oct 2020 at 10:38, <tetsuya.nomura at soho-enterprise.com> wrote:
> > >>>>>>
> > >>>>>> Dear Sirs.
> > >>>>>>
> > >>>>>> May I have your favor about initialization of the RaspberryPi and the camera?
> > >>>>>>
> > >>>>>> I'm now evaluating the camera extension board of FPD LINK III technology to extend the camera cable for 2~3m from the RaspberryPi.
> > >>>>>> https://soho-enterprise.com/2020/10/11/post-866/
> > >>>>>>
> > >>>>>> This board can pass the I2C communication to IMX219 after
> > >>>>>> configuring the transceiver chip, However, after configuration
> > >>>>>> and run the qcam, it seems the qcam program does not detect the IMX219 And I cannot start the camera even I designate the image sensor name.
> > >>>>>
> > >>>>> The Pi kernel driver for imx219 probes the sensor driver as the
> > >>>>> module loads, which is only a few seconds into boot if you've
> > >>>>> added "dtoverlay=imx219" into config.txt. No or incorrect
> > >>>>> response from the sensor and it won't create the /dev/video0 device nodes.
> > >>>>> If your transceivers need some form of configuration then I
> > >>>>> suspect this has not happened by this point.
> > >>>>>
> > >>>>> The correct approach is probably to have a driver for your
> > >>>>> transceiver in the kernel driver chain, ie imx219 -> transceiver
> > >>>>> -> Unicam, however that opens a big can of worms over
> > >>>>> configuration as it forces the use of the Media Controller API
> > >>>>> to configure each link independently.
> > >>>>
> > >>>> A very open question: assuming we can retain control through the
> > >>>> video node as implemented today for the existing use cases, and
> > >>>> implement MC-based pipeline control in parallel in the driver
> > >>>> (possibly selecting one of the options through a module
> > >>>> parameter) for use cases that would be tricky to support otherwise, would you be OK with that ?
> > >>>
> > >>> I'm totally OK with it!
> > >>> I've had a couple of other threads in the last few weeks that
> > >>> kicked me to look into MC and what is required.
> > >>>
> > >>> The first was trying to support Analog Devices ADV7482 which
> > >>> Kieran will know and "love". That showed up a couple of oddities
> > >>> around
> > >>
> > >> Ahem, cough cough. Indeed :-)
> > >>
> > >> Is there a board available that can connect this to the RPi?
> > >> If so I'd be interested in buying one.
> > >
> > > Not that I'm aware of.
> > > The forum contributor's company had brought up a board with
> > > ADV7282-M (analogue video to CSI2) on. Based on that they ploughed
> > > ahead in building a prototype Compute Module carrier board for
> > > ADV7482 (and DPI to VGA as done on the VGA666 board). That's the
> > > point that he posted on the forums as things didn't connect up nice and easily.
> > > He has been kind enough to ship one to me from Chile(!), but I
> > > haven't had time to power it up as yet. Looking at the PCB it
> > > appears to only route 2 CSI-2 data lanes which is a bit of a shame - oh well.
> >
> > Oh indeed, would have been nice to route all 4 lanes for a compute-module. Perhaps it's not too late for their next board revision?
> >
> > Well if you hit any blocker, or need anything let us know.
> >
> >
> > >
> > >>> linking to async_notifier and incorrect behaviour with our use of
> > >>> pad indexes. Those two bits are easy to rectify, although having
> > >>> made the async_notifier stuff match other platforms it no longer
> > >>> probed the simple modules correctly :-( There must be a solution
> > >>> there somewhere, it's only finding the correct runes.
> > >>
> > >> The ADV748x requires endpoint matching, and it complicates things indeed.
> > >>
> > >> For a long time, I think the only receiver that supported this
> > >> driver was the Renesas RCar-CSI2/VIN.
> > >>
> > >> There is a series here:
> > >>
> > >> https://lore.kernel.org/linux-media/20200701062140.12953-1-laurent.
> > >> pi
> > >> nchart+renesas at ideasonboard.com/
> > >>
> > >> Which should finally ease things and make endpoint matching
> > >> compatible with all receivers, which I believe has landed in v5.9.
> > >
> > > I thought I'd been running on our 5.9 branch, but looking at the
> > > branch I'd pushed for their reference[1] it looks to be 5.4.
> > > I'll try rebasing and see what I get.
> > >
> > > [1] https://github.com/6by9/linux/tree/adv748x
> > >
> > >>> The second use case was Toshiba's TC358746/TC358748 parallel to
> > >>> CSI bridges, where similar to this case it's chaining sensor ->
> > >>> TC35874[6|8] -> Unicam.
> > >>>
> > >>> With regard switching behaviours, it's relatively simple to look
> > >>> at the immediately upstream node and see if it has any sink pads.
> > >>> If yes we go MC, otherwise video-device. Doing so dynamically at
> > >>> runtime is nicer than a module parameter.
> > >>
> > >> +Niklas,
> > >>
> > >> I sort of agree here, as it should be easier for end users. Niklas
> > >> may have different opinions as he has felt pain of supporting both
> > >> MC and non-MC interfaces in the RCar-VIN.
> > >
> > > Felt as in past tense and you've dropped non-MC?
> > > I'll have a look at the RCar-VIN driver to see if I can work through
> > > the differences.
> > > ...
> > > It appears MC is enabled dependent on the platform - that gives me
> > > something to follow.
> > > I'm a little confused though as it still seems to call
> > > v4l2_subdev_call(sd, pad, set_fmt, pad_cfg, &format); from
> > > rvin_s_fmt_vid_cap to set the format on the upstream subdevice sink
> > > pad. Unless I'm missing something then I thought it wasn't allowed
> > > in MC to propagate formats across links.
> > >
> > >>> That's the easy bit. The harder bit is working out which bits of
> > >>> functionality that then requires dropping when in MC mode. At the
> > >>> moment I'm not totally clear over the correct approach for
> > >>> configuring the actual Unicam hardware - S_FMT on /dev/videoN, or
> > >>> MC API setting the sink pad format of Unicam. The mismash of
> > >>> MEDIA_BUS_FMT_xxx vs V4L2_PIX_FMT_xxx formats surfaces again, how
> > >>> do you avoid format mismatches (and unpacking to 16bit), and which
> > >>> do you use to configure CSI data type filtering.
> > >>> I'm assuming all the subdev API calls for EDIDs, timings (DV and
> > >>> standards), parm, and enumerations disappear with MC too, as you
> > >>> open the subdev node to configure that lot.
> > >>
> > >> Indeed, those can then be operated on the subdev node.
> > >
> > > OK, so lots of disable_ioctl calls but those are easy enough and
> > > already half in place for when the subdev doesn't support them.
> > >
> > > So just the step of understanding what configures the hardware
> > > format/width/height/stride/buffer size.
> >
> > Userspace ? I think that's where all the format propagation comes in.
> >
> > I'm very much aware that the TC358746/TC358748 is not a camera sensor,
> > but libcamera has so much of this 'done' that I do wonder how
> > non-camera
> > (media-controller) type devices should be used in the future. Perhaps there is some core parts of libcamera that should be factored out to facilitate non-camera type devices that use the same pipelines, or perhaps libcamera would grow to support them. It feels like a bit of a shame that they are out of scope for us at the moment.
> >
> > Same with the ADV748x HDMI input ... that's not a camera, but
> > libcamera would make it much easier to configure/capture from - though
> > technically, the 8 analogue inputs could be ... ;-)
> >
> > --
> > Kieran
> >
> >
> > >   Dave
> > >
> > >>>
> > >>> Thanks
> > >>>   Dave
> > >>>
> > >>>>> One workaround would be to remove "dtoverlay=imx219" from
> > >>>>> config.txt and use dynamic device tree overlay loading to load
> > >>>>> the relevant modules later in boot. "sudo dtoverlay imx219" should do that for you.
> > >>>>> The niggle is that when done from config.txt the firmware fixes
> > >>>>> up the camera shutdown GPIO to match the platform (it moves
> > >>>>> GPIOs between different variants of the Pi). If loading
> > >>>>> dynamically then this can't happen, and you'll need to fix up
> > >>>>> the regulator shutdown line manually [1] (assuming that it is
> > >>>>> actually controllable at the other end of the FPD Link)
> > >>>>>
> > >>>>> [1]
> > >>>>> (https://github.com/raspberrypi/linux/blob/rpi-5.4.y/arch/arm/bo
> > >>>>> ot
> > >>>>> /dts/overlays/imx219-overlay.dts#L77)
> > >>>>>
> > >>>>>> I can start the IMX219, get the RAW image by our own program and see the image.
> > >>>>>> Then if qcam can send the necessary command to IMX219 when
> > >>>>>> start running the qcam, I think I can see the image through qcam program.
> > >>>>>>
> > >>>>>> It would be great if you could give us an advice to solve it.
> > >>>>>> I appreciate your kind help in advance.
> > >>>>>>
> > >>>>>> Best Regards,
> > >>>>>>
> > >>>>>> NOMURA
> > >>>>
> > >>>> --
> > >>>> Regards,
> > >>>>
> > >>>> Laurent Pinchart
> > >>> _______________________________________________
> > >>> libcamera-devel mailing list
> > >>> libcamera-devel at lists.libcamera.org
> > >>> https://lists.libcamera.org/listinfo/libcamera-devel
> > >>>
> > >>
> > >> --
> > >> Regards
> > >> --
> > >> Kieran
> >
> > --
> > Regards
> > --
> > Kieran
> >
>