PCIe OF overlay driver - #3555
rodrigo455 wants to merge 9 commits into
Conversation
| if (ret) | ||
| return ret; | ||
|
|
||
| ret = adi_pcie_iommu_setup(apo); |
There was a problem hiding this comment.
this is likely misplaced, will move out from this function
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as resolved.
This comment was marked as resolved.
LLM reviewThis series adds a PCI driver ( Session: 35642498388 CI:
|
053bf4d to
4bc9804
Compare
|
@rodrigo455 can you cherry-pick your series on next again? i looks like the source branch rebased |
ed9c3d9 to
21220ee
Compare
4bc9804 to
f51c752
Compare
Analog Devices FPGA designs can be exposed as a PCIe endpoint whose BARs
window an AXI interconnect full of memory-mapped peripherals: DMA engines,
ADC/DAC interfaces, system ID cores and so on. Those peripherals already
have some upstream drivers, which often rely on dt/fw properties.
This series follows the strategy of drivers/misc/lan966x_pci.c: apply a
device-tree overlay under the endpoint's OF node and let the existing
platform drivers probe the devices it describes. It differs in three ways.
The overlay is requested as firmware, named by a module parameter, rather
than embedded in the module, so one driver serves designs whose contents
differ per bitstream.
The endpoint's OF node is created by the driver. lan966x_pci requires the
PCI core to have made one already (via quirks) and fails if it did not.
Patch 1 adds devm_of_pci_make_dev_node(), which creates the node on demand,
leaves an existing one alone, and removes what it created on unbind.
Interrupts revolve around a dedicated interrupt controller in the FPGA
rather than a single shared line. lan966x_pci maps one dummy_irq_chip
hwirq, which is all a single-interrupt endpoint needs. A design with
several DMA engines needs per-source masking and acknowledgement, so the
int controller behind BAR0 reports how many sources it samples and how
many vectors it implements, and the driver registers a real irq_chip over
it -- mask, unmask, eoi and set_affinity -- with MSI-X, MSI or shared INTx
as the parent. Each source is routed to one of the granted vectors, and
each vector is installed as a chained handler that claims and dispatches
what it collected.
The other half of the problem is DMA. The platform devices created from
the overlay are not PCI devices and have no IOMMU group, so on a
translated host a DMA engine behind a BAR would be programmed with
physical addresses while its transactions arrive at the IOMMU under the
endpoint's requester ID. Patch 4 joins those children to the endpoint's
IOMMU group so the DMA API allocates their IOVAs out of the domain that
will translate them; patch 2 is the iommu/dma prerequisite, since a device
that only shares a group has no per-device IOMMU state.
The driver declares no PCI device table, and binding is left to the user
through driver_override or new_id. Three things have to settle first:
upstreaming of the platform drivers for these peripherals has fallen
behind, so no endpoint yet has all of its contents served by upstream
code; the vendor ID and device ID pair a product will carry is not agreed
on; and the boards tested so far are FPGA-based SoCs, whose designs change
frequently at this initial phase. A later patch can add a table once a
first entry meets all of those requirements.
Patches 7 and 8 let the AXI-DMAC and AXI-ADC drivers build for such an
endpoint. Their architecture dependency lists soft cores and Zynq, which
excludes the host architectures that reach the core over PCIe, x86_64
among them.
To: linux-pci@vger.kernel.org
To: linux-kernel@vger.kernel.org
To: devicetree@vger.kernel.org
To: iommu@lists.linux.dev
To: dmaengine@vger.kernel.org
To: linux-iio@vger.kernel.org
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Rob Herring <robh@kernel.org>
Cc: Saravana Kannan <saravanak@kernel.org>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: "Joerg Roedel (AMD)" <joro@8bytes.org>
Cc: Will Deacon <will@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Vinod Koul <vkoul@kernel.org>
Cc: Frank Li <Frank.Li@kernel.org>
Cc: Jonathan Cameron <jic23@kernel.org>
Cc: David Lechner <dlechner@baylibre.com>
Cc: Nuno Sá <nuno.sa@analog.com>
Cc: Andy Shevchenko <andy@kernel.org>
Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
--- b4-submit-tracking ---
# This section is used internally by b4 prep for tracking purposes.
{
"series": {
"revision": 1,
"change-id": "20260921-misc-adi-pcie-of-overlay-aa1962edca69",
"prefixes": [],
"presubject": ""
}
}
of_pci_make_dev_node() creates an OF node for a PCI device, but its lifetime belongs to whoever created it: a DT-native system, or a DECLARE_PCI_FIXUP_FINAL quirk. A driver that describes an endpoint's internals with a device-tree overlay needs a node to apply that overlay under, needs it to outlive its own bind, and must not release a node someone else owns. Add a devm wrapper covering both cases: leave a pre-existing of_node alone, otherwise create one and tie its removal to the device. It stubs out to -ENOENT without CONFIG_PCI_DYNAMIC_OF_NODES, the same option that already gives the parent bridge the node creation requires. Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
A device can be a member of another device's IOMMU group without having been probed by an IOMMU driver: iommu_group_add_device() puts it in the group but leaves dev->iommu NULL. That fits a device whose transactions reach the IOMMU tagged as another's, a non-PCI device behind a PCI endpoint's BARs for instance, which has no requester ID to probe an IOMMU driver against but whose DMA must still go through the endpoint's domain. It reaches iommu_dma_alloc_iova() through the DMA API like any other device and oopses on the dev->iommu deref for the 32-bit-first address preference. Skip the preference when there is no per-device state to keep it in, as iommu_deferred_attach() already does; the device then takes whatever the allocator returns, capped as always by its DMA mask. Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
Analog Devices designs can be exposed as a PCIe endpoint whose BARs window an AXI interconnect full of memory-mapped peripherals: DMA engines, ADC/DAC interfaces, and so on. Those peripherals already have drivers, but they are device-tree based, and enumerating the endpoint gives no description of what is behind its BARs. Add a PCI driver that takes that description from firmware. On probe it requests the FDT overlay named by the "overlay" module parameter, creates an OF node for the endpoint, applies the overlay under it and populates the platform devices so the peripheral drivers probe. Everything is undone in reverse by devm actions, so an unbind leaves no node, overlay or child behind. Interrupts come from a dedicated controller behind BAR0, which reports how many sources it samples and how many vectors it implements. The driver registers a linear irqdomain over the sources so overlay nodes can reference them by number, routes each source to one of the MSI-X, MSI or shared INTx interrupts it was granted, and installs those as chained handlers that claim and dispatch what their vector collected. The driver carries no id_table, so binding is left to the user through driver_override or new_id: no endpoint whose peripherals are all served by upstream drivers is available yet, and the FPGA designs are still changing at this early stage. Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
…dren The platform devices created from the overlay are not PCI devices and have no IOMMU group of their own. On a host where the endpoint is translated they keep the platform bus default of direct DMA, so a DMA engine behind a BAR is programmed with physical addresses while its transactions arrive at the IOMMU as the endpoint's requester ID. The mapping a child driver installs is never the one its hardware needs. Join every device probed below the endpoint to the endpoint's IOMMU group, from a platform bus notifier registered when the endpoint itself is translated, and mark it as taking the IOMMU DMA path. A child has no requester ID of its own, so it gets membership only: no domain attach and no per-device IOMMU state. The DMA API then allocates its IOVAs out of the endpoint's domain, the one that will translate them, and the iommu core drops the membership when the device goes away. An "iommus" property in the overlay cannot do this instead: the stream ID comes from a requester ID that enumeration assigns, so firmware cannot name it; the same stream ID handed to an IOMMU driver twice puts the child in a second group with a second domain, which arm-smmu-v3 rejects as an unsupported alias; and on x86 there is no IOMMU node to point at. Children now size their own mappings rather than inheriting the endpoint's 32-bit mask, so the overlay has to describe what its hardware can address (dma-ranges) and whether it is coherent (dma-coherent), as for any other DT platform device. A child that fails to join keeps direct DMA and hands its hardware untranslated addresses. That cannot be prevented here: device_add() ignores what the notifier returns, and probe cannot wait for children that deferred probing and nested buses add later. It is logged. Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
Sources are routed to vectors at probe and stay there, so irqbalance and an explicit smp_affinity write on a child's interrupt had nothing to act on: the source kept arriving on whichever CPU its vector was pinned to. Move the source between vectors rather than a vector between CPUs. The parent MSI interrupts keep the CPUs the PCI core gave them, so walk the granted vectors from the one currently routed, take the first whose effective affinity intersects the requested mask, reprogram SRC_ROUTE and report the CPU the source will land on. Fail when no vector serves the mask, and under INTx where there is nothing to route. Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
… msi An MSI-capable controller delivers a vector by writing the address and data the host assigned to it, which it takes from an MSI-X style table in its register space. Under MSI-X the PCI core owns that table and programs it directly. Under plain MSI the core writes one address and data pair into config space, the table is left holding whatever the previous host put there, and the endpoint writes to memory that is no longer anyone's. Install desc->write_msi_msg on the associated descriptors so the core's message writes reach the table as well, and fill it from the descriptor the core has already composed. The hook covers a move, a CPU going offline and a resume, not just activation, since it runs on every write. Entries are programmed masked and unmasked once the address is non-zero, so a teardown write leaves the entry masked rather than live on a stale address. A write arriving while the device is not in D0 is dropped, as the core drops its own, and replayed at resume. Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
The AXI-DMAC can also be reached through a PCIe endpoint, as a platform device created by adi-pcie-of-overlay from a firmware device-tree overlay. The host architecture is then unrelated to the fabric the core runs in, so the existing soft-core and Zynq architecture list excludes the very configurations that need it, x86_64 among them. Add ADI_PCIE_OF_OVERLAY to the dependency list. Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
The AXI ADC core can also be reached through a PCIe endpoint, as a platform device created by adi-pcie-of-overlay from a firmware device-tree overlay. The host architecture is then unrelated to the fabric the core runs in, so the existing soft-core and Zynq architecture list excludes the very configurations that need it, x86_64 among them. Add ADI_PCIE_OF_OVERLAY to the dependency list. Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
21220ee to
c9bedaf
Compare
Support ADI PCIe endpoints described by a device-tree overlay
The cover letter:
PR Type
PR Checklist