[libcamera-devel] How to detect camera after power on the raspberryPi.

Fri Nov 20 08:09:30 CET 2020

Dear Dave-san

Now we are taking the below way to identify the crypto chip.

Power supply from GPIO 5V port to FPD Link III  board.
Power up and check the status of the Rx and if no i2c configuration for sensor and the crypto chip,
configure the register and reboot by rc.local to let Raspi detect the sensor and the crypto chip.

Maybe this is the easiest way without any configuration change of the RaspberryPi.

Now I can run the qcam and raspivid as well as before.

Thank you for your advice again.

Best Regards,

NOMURA 

-----Original Message-----
From: Dave Stevenson <dave.stevenson at raspberrypi.com> 
Sent: Wednesday, October 21, 2020 12:41 AM
To: tetsuya.nomura at soho-enterprise.com
Cc: cc: Kieran Bingham <kieran.bingham at ideasonboard.com>; Laurent Pinchart <laurent.pinchart at ideasonboard.com>; libcamera-devel at lists.libcamera.org; Niklas Söderlund <niklas.soderlund at ragnatech.se>
Subject: Re: [libcamera-devel] How to detect camera after power on the raspberryPi.

Hi

On Tue, 20 Oct 2020 at 13:59, <tetsuya.nomura at soho-enterprise.com> wrote:
>
> Dear Dave-san
>
> Thank you for your explanation.
> I understand what you are discussing.
>
> I'm now using DS90UB953-Q1 and DS90UB954-Q1 from TI.

There is an RFC (Request For Comments) patchset from last July for that pair of chips at https://patchwork.linuxtv.org/project/linux-media/cover/20190723203723.11730-1-luca@lucaceresoli.net/
I haven't looked through the patches to see what is and isn't still valid, but I seem to recall that the similar GMSL patches have now been merged which worked on very similar areas for I2C address mapping.

> I just set several register parameters on them, set the I2C configuration, set the serial speed, number of lanes and speed for MIPI.
> I just send the command by i2cset without using the driver.
> If I can configure the FPD Link III transceiver at the beginning of the boot, Maybe after configure the I2C port on MIPI port then send the I2C commands to transceiver, it will behave as if the sensor board were directly connected to Raspi CSI-2 port.
> I wonder if I can reconfigure the device tree after setting the transceiver registers to run qcam.

You could, but it's always going to be a race condition that needs to be managed somehow. Blacklisting one or both of the imx219 and bcm2835-unicam modules, and then manually modprobe'ing them once you've sent your I2C commands to the FPD chips may work.

Incorporating the serialiser and deserialiser into the pipeline removes that and is really the correct way of doing things, and the only one you're likely to get any degree of support implementing (even though it's the more complex solution). It should also allow for things like power control to be utilised should the chips have lower power modes whilst not streaming.

  Dave

> Best Regards,
>
> NOMURA
>
> -----Original Message-----
> From: Dave Stevenson <dave.stevenson at raspberrypi.com>
> Sent: Tuesday, October 20, 2020 9:39 PM
> To: tetsuya.nomura at soho-enterprise.com
> Cc: cc: Kieran Bingham <kieran.bingham at ideasonboard.com>; Laurent 
> Pinchart <laurent.pinchart at ideasonboard.com>; 
> libcamera-devel at lists.libcamera.org; Niklas Söderlund 
> <niklas.soderlund at ragnatech.se>
> Subject: Re: [libcamera-devel] How to detect camera after power on the raspberryPi.
>
> On Tue, 20 Oct 2020 at 12:48, <tetsuya.nomura at soho-enterprise.com> wrote:
> >
> > Dear Sirs
> >
> > Maybe I understand less than 50% of your conversation below, but if 
> > you need MIPI 4 lane camera board which can be connected to Raspi CM3+, I can provide you the IMX219 4lane camera board.
>
> Apologies, we've gone off on a slight tangent discussing the logistics of the correct way to implement a multi element chain such as sensor -> FPD Link transceivers -> CSI2 receiver (Unicam).
>
> At present the drivers are only set up for 2 drivers being involved (sensor and receiver), and there are significant issues to resolve to involve three or more.
>
> We already have 2 use cases being discussed for which this Media Controller (MC) API is required, and I have hardware to test out one of them. If we get it right there, and if there is a driver available for your FPD Link III devices, then switching to that at a kernel level should be trivial.
> However it isn't so trivial to integrate that change of architecture into libcamera.
>
> Could you tell us which FPD Link devices it is that you are using?
> Several of those who are directly involved in libcamera have also been involved with writing drivers for GMSL and other CSI2 transceiver chips, so they may well be able to advise on the best route forward on supporting the devices you've chosen.
>
>   Dave
>
> > And if you are interested in controlling the focus, I can also send you the AF type module of IMX219.
> > The other option, IMX378 which is MX477 compatible sensor, with focus driver is also available.
> > https://soho-enterprise.com/2020/08/31/raspi-libcameraqcam-for-imx37
> > 8/
> >
> > I already tested them on Raspi 3 and 4 with qcam and they works well.
> > I tested the 4 lane mode of IMX219 module with Tinker Edge R, this means you can get full pixel 30fps image.
> > https://soho-enterprise.com/2020/02/27/post-681/
> > I captured high speed RAW image in 4 lane mode as below article.
> > https://soho-enterprise.com/2020/04/10/high-frame-rate-high-speed-ca
> > me ra-on-tinker-edge-r-testing-mipi-4-lane-feature/
> >
> > And if you could help us to operate our SE-CXB01 Tx/Rx, the FPD LINK 
> > III camera extension board, I also can provide the one.
> > (Let me try the method using "dtoverlay" first.) 
> > https://soho-enterprise.com/2020/10/19/dual-camera-system-under-deve
> > lo
> > pment-se-cxb01-tx-rx/
> >
> > I hope my offer will be your help.
> >
> > Best Regards,
> >
> > NOMURA
> >
> > -----Original Message-----
> > From: Kieran Bingham <kieran.bingham at ideasonboard.com>
> > Sent: Tuesday, October 20, 2020 7:50 PM
> > To: Dave Stevenson <dave.stevenson at raspberrypi.com>
> > Cc: Laurent Pinchart <laurent.pinchart at ideasonboard.com>;
> > libcamera-devel at lists.libcamera.org;
> > tetsuya.nomura at soho-enterprise.com; Niklas Söderlund 
> > <niklas.soderlund at ragnatech.se>
> > Subject: Re: [libcamera-devel] How to detect camera after power on the raspberryPi.
> >
> > Hi Dave,
> >
> > On 20/10/2020 11:32, Dave Stevenson wrote:
> > > Hi Kieran
> > >
> > > On Tue, 20 Oct 2020 at 10:36, Kieran Bingham 
> > > <kieran.bingham at ideasonboard.com> wrote:
> > >>
> > >> Hi Dave,
> > >>
> > >> On 19/10/2020 16:53, Dave Stevenson wrote:
> > >>> Hi Laurent
> > >>>
> > >>> On Mon, 19 Oct 2020 at 13:50, Laurent Pinchart 
> > >>> <laurent.pinchart at ideasonboard.com> wrote:
> > >>>>
> > >>>> Hi Dave,
> > >>>>
> > >>>> On Mon, Oct 19, 2020 at 11:10:55AM +0100, Dave Stevenson wrote:
> > >>>>> On Mon, 19 Oct 2020 at 10:38, <tetsuya.nomura at soho-enterprise.com> wrote:
> > >>>>>>
> > >>>>>> Dear Sirs.
> > >>>>>>
> > >>>>>> May I have your favor about initialization of the RaspberryPi and the camera?
> > >>>>>>
> > >>>>>> I'm now evaluating the camera extension board of FPD LINK III technology to extend the camera cable for 2~3m from the RaspberryPi.
> > >>>>>> https://soho-enterprise.com/2020/10/11/post-866/
> > >>>>>>
> > >>>>>> This board can pass the I2C communication to IMX219 after 
> > >>>>>> configuring the transceiver chip, However, after 
> > >>>>>> configuration and run the qcam, it seems the qcam program does not detect the IMX219 And I cannot start the camera even I designate the image sensor name.
> > >>>>>
> > >>>>> The Pi kernel driver for imx219 probes the sensor driver as 
> > >>>>> the module loads, which is only a few seconds into boot if 
> > >>>>> you've added "dtoverlay=imx219" into config.txt. No or 
> > >>>>> incorrect response from the sensor and it won't create the /dev/video0 device nodes.
> > >>>>> If your transceivers need some form of configuration then I 
> > >>>>> suspect this has not happened by this point.
> > >>>>>
> > >>>>> The correct approach is probably to have a driver for your 
> > >>>>> transceiver in the kernel driver chain, ie imx219 -> 
> > >>>>> transceiver
> > >>>>> -> Unicam, however that opens a big can of worms over
> > >>>>> configuration as it forces the use of the Media Controller API 
> > >>>>> to configure each link independently.
> > >>>>
> > >>>> A very open question: assuming we can retain control through 
> > >>>> the video node as implemented today for the existing use cases, 
> > >>>> and implement MC-based pipeline control in parallel in the 
> > >>>> driver (possibly selecting one of the options through a module
> > >>>> parameter) for use cases that would be tricky to support otherwise, would you be OK with that ?
> > >>>
> > >>> I'm totally OK with it!
> > >>> I've had a couple of other threads in the last few weeks that 
> > >>> kicked me to look into MC and what is required.
> > >>>
> > >>> The first was trying to support Analog Devices ADV7482 which 
> > >>> Kieran will know and "love". That showed up a couple of oddities 
> > >>> around
> > >>
> > >> Ahem, cough cough. Indeed :-)
> > >>
> > >> Is there a board available that can connect this to the RPi?
> > >> If so I'd be interested in buying one.
> > >
> > > Not that I'm aware of.
> > > The forum contributor's company had brought up a board with 
> > > ADV7282-M (analogue video to CSI2) on. Based on that they ploughed 
> > > ahead in building a prototype Compute Module carrier board for
> > > ADV7482 (and DPI to VGA as done on the VGA666 board). That's the 
> > > point that he posted on the forums as things didn't connect up nice and easily.
> > > He has been kind enough to ship one to me from Chile(!), but I 
> > > haven't had time to power it up as yet. Looking at the PCB it 
> > > appears to only route 2 CSI-2 data lanes which is a bit of a shame - oh well.
> >
> > Oh indeed, would have been nice to route all 4 lanes for a compute-module. Perhaps it's not too late for their next board revision?
> >
> > Well if you hit any blocker, or need anything let us know.
> >
> >
> > >
> > >>> linking to async_notifier and incorrect behaviour with our use 
> > >>> of pad indexes. Those two bits are easy to rectify, although 
> > >>> having made the async_notifier stuff match other platforms it no 
> > >>> longer probed the simple modules correctly :-( There must be a 
> > >>> solution there somewhere, it's only finding the correct runes.
> > >>
> > >> The ADV748x requires endpoint matching, and it complicates things indeed.
> > >>
> > >> For a long time, I think the only receiver that supported this 
> > >> driver was the Renesas RCar-CSI2/VIN.
> > >>
> > >> There is a series here:
> > >>
> > >> https://lore.kernel.org/linux-media/20200701062140.12953-1-laurent.
> > >> pi
> > >> nchart+renesas at ideasonboard.com/
> > >>
> > >> Which should finally ease things and make endpoint matching 
> > >> compatible with all receivers, which I believe has landed in v5.9.
> > >
> > > I thought I'd been running on our 5.9 branch, but looking at the 
> > > branch I'd pushed for their reference[1] it looks to be 5.4.
> > > I'll try rebasing and see what I get.
> > >
> > > [1] https://github.com/6by9/linux/tree/adv748x
> > >
> > >>> The second use case was Toshiba's TC358746/TC358748 parallel to 
> > >>> CSI bridges, where similar to this case it's chaining sensor -> 
> > >>> TC35874[6|8] -> Unicam.
> > >>>
> > >>> With regard switching behaviours, it's relatively simple to look 
> > >>> at the immediately upstream node and see if it has any sink pads.
> > >>> If yes we go MC, otherwise video-device. Doing so dynamically at 
> > >>> runtime is nicer than a module parameter.
> > >>
> > >> +Niklas,
> > >>
> > >> I sort of agree here, as it should be easier for end users. 
> > >> Niklas may have different opinions as he has felt pain of 
> > >> supporting both MC and non-MC interfaces in the RCar-VIN.
> > >
> > > Felt as in past tense and you've dropped non-MC?
> > > I'll have a look at the RCar-VIN driver to see if I can work 
> > > through the differences.
> > > ...
> > > It appears MC is enabled dependent on the platform - that gives me 
> > > something to follow.
> > > I'm a little confused though as it still seems to call 
> > > v4l2_subdev_call(sd, pad, set_fmt, pad_cfg, &format); from 
> > > rvin_s_fmt_vid_cap to set the format on the upstream subdevice 
> > > sink pad. Unless I'm missing something then I thought it wasn't 
> > > allowed in MC to propagate formats across links.
> > >
> > >>> That's the easy bit. The harder bit is working out which bits of 
> > >>> functionality that then requires dropping when in MC mode. At 
> > >>> the moment I'm not totally clear over the correct approach for 
> > >>> configuring the actual Unicam hardware - S_FMT on /dev/videoN, 
> > >>> or MC API setting the sink pad format of Unicam. The mismash of 
> > >>> MEDIA_BUS_FMT_xxx vs V4L2_PIX_FMT_xxx formats surfaces again, 
> > >>> how do you avoid format mismatches (and unpacking to 16bit), and 
> > >>> which do you use to configure CSI data type filtering.
> > >>> I'm assuming all the subdev API calls for EDIDs, timings (DV and 
> > >>> standards), parm, and enumerations disappear with MC too, as you 
> > >>> open the subdev node to configure that lot.
> > >>
> > >> Indeed, those can then be operated on the subdev node.
> > >
> > > OK, so lots of disable_ioctl calls but those are easy enough and 
> > > already half in place for when the subdev doesn't support them.
> > >
> > > So just the step of understanding what configures the hardware 
> > > format/width/height/stride/buffer size.
> >
> > Userspace ? I think that's where all the format propagation comes in.
> >
> > I'm very much aware that the TC358746/TC358748 is not a camera 
> > sensor, but libcamera has so much of this 'done' that I do wonder 
> > how non-camera
> > (media-controller) type devices should be used in the future. Perhaps there is some core parts of libcamera that should be factored out to facilitate non-camera type devices that use the same pipelines, or perhaps libcamera would grow to support them. It feels like a bit of a shame that they are out of scope for us at the moment.
> >
> > Same with the ADV748x HDMI input ... that's not a camera, but 
> > libcamera would make it much easier to configure/capture from - 
> > though technically, the 8 analogue inputs could be ... ;-)
> >
> > --
> > Kieran
> >
> >
> > >   Dave
> > >
> > >>>
> > >>> Thanks
> > >>>   Dave
> > >>>
> > >>>>> One workaround would be to remove "dtoverlay=imx219" from 
> > >>>>> config.txt and use dynamic device tree overlay loading to load 
> > >>>>> the relevant modules later in boot. "sudo dtoverlay imx219" should do that for you.
> > >>>>> The niggle is that when done from config.txt the firmware 
> > >>>>> fixes up the camera shutdown GPIO to match the platform (it 
> > >>>>> moves GPIOs between different variants of the Pi). If loading 
> > >>>>> dynamically then this can't happen, and you'll need to fix up 
> > >>>>> the regulator shutdown line manually [1] (assuming that it is 
> > >>>>> actually controllable at the other end of the FPD Link)
> > >>>>>
> > >>>>> [1]
> > >>>>> (https://github.com/raspberrypi/linux/blob/rpi-5.4.y/arch/arm/
> > >>>>> bo
> > >>>>> ot
> > >>>>> /dts/overlays/imx219-overlay.dts#L77)
> > >>>>>
> > >>>>>> I can start the IMX219, get the RAW image by our own program and see the image.
> > >>>>>> Then if qcam can send the necessary command to IMX219 when 
> > >>>>>> start running the qcam, I think I can see the image through qcam program.
> > >>>>>>
> > >>>>>> It would be great if you could give us an advice to solve it.
> > >>>>>> I appreciate your kind help in advance.
> > >>>>>>
> > >>>>>> Best Regards,
> > >>>>>>
> > >>>>>> NOMURA
> > >>>>
> > >>>> --
> > >>>> Regards,
> > >>>>
> > >>>> Laurent Pinchart
> > >>> _______________________________________________
> > >>> libcamera-devel mailing list
> > >>> libcamera-devel at lists.libcamera.org
> > >>> https://lists.libcamera.org/listinfo/libcamera-devel
> > >>>
> > >>
> > >> --
> > >> Regards
> > >> --
> > >> Kieran
> >
> > --
> > Regards
> > --
> > Kieran
> >
>