[libcamera-devel] Fwd: Surface Go VCM type (was: Need to pass acpi_enforce_resources=lax on the Surface Go (version1))

[Hans de Goede]

Hi,

On 11/9/21 13:09, Daniel Scally wrote:
> Hi Hans
> 
> On 09/11/2021 00:43, Daniel Scally wrote:
>> Hi Hans
>>
>> On 08/11/2021 13:12, Hans de Goede wrote:
>>> Hi,
>>>
>>> On 11/2/21 00:43, Daniel Scally wrote:
>>>> Hi Hans
>>> <snip>
>>>  
>>>
>>>>> 2. I need some help with all the fwnode link stuff (I'm not very familiar
>>>>> with this). There seems to be a chicken and egg problem here though,
>>>>> because the v4l2subdev for the VCM does not register because of async stuff
>>>>> and if we add it to the "graph" then my idea to enumerate the VCMs
>>>>> from the SSDB on the complete() callback won't work. But we can do this
>>>>> on a per sensor basis instead from the cio2_notifier_bound() callback
>>>>> instead I guess ?
>>>> I think on top of your work in the cio2-bridge for patch 3 you can do this:
>>>>
>>>>
>>>> 1. Create another software node against the cio2_sensor struct, with the
>>>> name coming from the vcm_types array
>>>>
>>>> 2. Assign that software node to board_info.swnode in
>>>> cio2_bridge_instantiate_vcm_i2c_client()
>>>>
>>>> 3. Add another entry to dev_properties for the sensor, that is named
>>>> "lens-focus" and contains a reference to the software_node created in #2
>>>> just like the references to the sensor/cio2 nodes.
>>>>
>>>>
>>>> This way when the sensor driver calls
>>>> v4l2_async_register_subdev_sensor() it should create a notifier that
>>>> looks for that VCM client to bind. I think then rather than putting
>>>> anything in the .bound() / .complete() callbacks, we should modify core
>>>> to do _something_ when async matching some subdevs. The something would
>>>> depend on the kind of devices that match, for example with the sensor
>>>> driver and the ipu3-cio2 driver, there's an entity whos function is
>>>> MEDIA_ENT_F_VID_IF_BRIDGE matching to an entity whos function is
>>>> MEDIA_ENT_F_CAM_SENSOR, and it seems to me that every scenario like that
>>>> is going to result in media pad links being created. Similarly for our
>>>> sensor that's a device with entity function MEDIA_ENT_F_LENS matching to
>>>> MEDIA_ENT_F_CAM_SENSOR, and I think that in those cases we can create
>>>> either an interface link or a new kind of link (maybe
>>>> "MEDIA_LNK_FL_ANCILLARY_LINK" or something...) between the two to show
>>>> that they form a single logical unit, which we can then report to libcamera.
>>>>
>>>>
>>>> Hope that makes sense...
>>> Ok, so I gave this a try, see the attached patches, but the v4l2-subdev for
>>> the VCM still does not show up.
>>
>> This is exactly where I got to over the weekend too
>>
>>> I think that instead I need to build a full link between the sensor
>>> and the VCM similar to the cio2 <-> sensor link. Both ends of that link
>>> have:
>>>
>>> <base-swnode attached to the device>
>>> |
>>> --<port-swnode named (SWNODE_GRAPH_PORT_NAME_FMT, X), where X is 0 on the
>>>   |                           sensor side and the link nr on the cio2 side
>>>   |
>>>   --<end-point-swnode named (SWNODE_GRAPH_ENDPOINT_NAME_FMT, 0)
>>>
>>> And then the 2 endpoints contain a swref property pointing to the
>>> other endpoint swnode.
>>>
>>> I think we need a similar setup adding a swnode child named
>>> (SWNODE_GRAPH_PORT_NAME_FMT, 1), to the nodes[SWNODE_SENSOR_HID] node.
>>>
>>> Note 1, since 0 is the "port" to the cio2, this new port child then
>>> gets an endpoint "0" child itself, likewise we add a "port 0" child
>>> to the vcm swnode, with a "endpoint 0" child below that and then have
>>> the 2 endpoints contain swref properties pointing to each other.
>>>
>>> I think that this will properly make the VCm part of the graph and
>>> will make its v4l2-subdev get instantiated when the graph is
>>> complete.  Before I spend a bunch of time on implementing this,
>>> let me ask this:
>>>
>>> Does this sound reasonable / like I'm heading in the right direction?
>> I don't think that we need to add the software nodes as
>> ports/endpoints...as far as I can tell it ought to work like this:
>>
>>
>> 1. The sensor calls v4l2_async_register_subdev_sensor() which...
>>
>>     a) creates a notifier
>>
>>     b) looks for reference properties of the device's fwnode called
>> "lens-focus" and calls v4l2_async_notifier_add_fwnode_subdev() against
>> the reference, which tells the notifier it's connected to this other
>> fwnode and to expect it to bind.
>>
>> 2. When new subdevs are registered they get tested for a match against
>> the notifier registered in 1a that matches to their fwnode using
>> match_fwnode() [1]. This should work, on the grounds that we registered
>> the device using the board_info.swnode and registered a lens-focus
>> property that points to that software_node
>>
>> 3. When a match is found, the notifier's .bound() function is called.
>> When all the asds that the notifier expects are bound the notifier's
>> .complete() callback is called.
>>
>>
>> That's not working correctly for me at the moment, but I think this is a
>> surmountable problem rather than the wrong approach, so I'm just working
>> through the differences to try and get the matching working.
> 
> 
> OK, I eventually got this working - the dw9719 registers as
> /dev/v4l-subdev7 for me now ... long story short is the attached patch
> was needed to make the references work, as the internals of v4l2 aren't
> checking for fwnode->secondary. Prior to your latest series as well, an
> additional problem was that once the VCMs fwnode was linked to the
> sensor's the .complete() callback for ipu3-cio2 would never call
> (because it needs ALL the devices for the linked fwnodes to be bound to
> do that)...which meant the VCMs never got instantiated, because that was
> where that function was called. With your new set separating those
> processes it works well, so yes I like that new approach very much :D
> 
> 
> In the end we don't have to add a call creating the subdev's - it turns
> out that v4l2 knows it's part of ipu3-cio2's v4l2-device so it registers
> the nodes for the vcm when .complete() is called for that driver. I
> still think we should add a bit creating the link to expose to userspace
> in match_notify() though.
> 
> 
> Trying to list controls for the dw9719 with v4l2-ctl -d /dev/v4l-subdev7
> -L fails with an IOCTL error, so I have some remedial work on the driver
> which I'll do tonight; I'd expect to be able to control focus with
> v4l2-ctl -d /dev/v4l-subdev7 -c absolute_focus=n once this is sorted.

That is great, thank you so much. I wanted to look into this myself
today but I got distracted by other stuff.

Mainly getting Windows to work (including cameras) after a fresh
Windows install on A Dell Latitude 7285 which I just got as a second
device to test IPU3 stuff on :)

Talking about this Dell Latitude 7285, I haven't had a chance to
look into this at all. But chances are I will need to do some
I2C-register dumps under Windows, last time you mentioned you
had some small tool for this ? It is ok if it is a bit hackish,
it will still be very useful to have :)  And I believe I will
also need to override the DSDT under Windows for this, right?
I should be able to cope with that too.

Regards,

Hans

> 
>>
>>
>> For the devnodes, the ipu3-cio2 driver itself creates the devnodes for
>> the subdevices that bind to it (like the sensor) as part of its
>> .complete() callback [2] by calling v4l2_device_register_subdev_nodes(),
>> as far as I can tell there's nothing in v4l2 core that handles that
>> automatically so I think that that lack is what's preventing the
>> devnodes from showing up. I think we should tackle the problem of the
>> missing devnodes by mimicking the effects of that function somewhere
>> within core, probably v4l2_async_match_notify() (which calls the
>> notifier's .bound() callback). I think the creation of the links to
>> expose to userspace that this is a logical unit should probably happen
>> in the same place, using the entity.function field of the subdev and the
>> asd to decide exactly what kind of link to create.
>>
>>
>> [1]
>> https://elixir.bootlin.com/linux/latest/source/drivers/media/v4l2-core/v4l2-async.c#L69
>>
>> [2]
>> https://elixir.bootlin.com/linux/latest/source/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c#L1449
>>
>>> Regards,
>>>
>>> Hans
>>>
>>>
>>>
>>> p.s.
>>>
>>> I have found a new solution for the probe-ordering problem which
>>> is patch 2 of the attached patches, I personally I'm happy with
>>> this solution. I hope you like it too.