Skip to content

PCIe OF overlay driver - #3555

Open
rodrigo455 wants to merge 9 commits into
mirror_ci/next/linux-next/masterfrom
staging/misc-adi-pcie-of-overlay
Open

rodrigo455 wants to merge 9 commits into
mirror_ci/next/linux-next/masterfrom
staging/misc-adi-pcie-of-overlay

Conversation

@rodrigo455

@rodrigo455 rodrigo455 commented Sep 21, 2026

Copy link
Copy Markdown
Collaborator

Support ADI PCIe endpoints described by a device-tree overlay

The cover letter:

Analog Devices FPGA designs can be exposed as a PCIe endpoint whose BARs
window an AXI interconnect full of memory-mapped peripherals: DMA engines,
ADC/DAC interfaces, system ID cores and so on. Those peripherals already
have some upstream drivers, which often rely on dt/fw properties.

This series follows the strategy of drivers/misc/lan966x_pci.c: apply a
device-tree overlay under the endpoint's OF node and let the existing
platform drivers probe the devices it describes. It differs in three ways.

The overlay is requested as firmware, named by a module parameter, rather
than embedded in the module, so one driver serves designs whose contents
differ per bitstream.

The endpoint's OF node is created by the driver. lan966x_pci requires the
PCI core to have made one already (via quirks) and fails if it did not.
Patch 1 adds devm_of_pci_make_dev_node(), which creates the node on demand,
leaves an existing one alone, and removes what it created on unbind.

Interrupts revolve around a dedicated interrupt controller in the FPGA
rather than a single shared line. lan966x_pci maps one dummy_irq_chip
hwirq, which is all a single-interrupt endpoint needs. A design with
several DMA engines needs per-source masking and acknowledgement, so the
int controller behind BAR0 reports how many sources it samples and how
many vectors it implements, and the driver registers a real irq_chip over
it -- mask, unmask, eoi and set_affinity -- with MSI-X, MSI or shared INTx
as the parent. Each source is routed to one of the granted vectors, and
each vector is installed as a chained handler that claims and dispatches
what it collected.

The other half of the problem is DMA. The platform devices created from
the overlay are not PCI devices and have no IOMMU group, so on a
translated host a DMA engine behind a BAR would be programmed with
physical addresses while its transactions arrive at the IOMMU under the
endpoint's requester ID. Patch 4 joins those children to the endpoint's
IOMMU group so the DMA API allocates their IOVAs out of the domain that
will translate them; patch 2 is the iommu/dma prerequisite, since a device
that only shares a group has no per-device IOMMU state.

The driver declares no PCI device table, and binding is left to the user
through driver_override or new_id. Three things have to settle first:
upstreaming of the platform drivers for these peripherals has fallen
behind, so no endpoint yet has all of its contents served by upstream
code; the vendor ID and device ID pair a product will carry is not agreed
on; and the boards tested so far are FPGA-based SoCs, whose designs change
frequently at this initial phase. A later patch can add a table once a
first entry meets all of those requirements.

Patches 7 and 8 let the AXI-DMAC and AXI-ADC drivers build for such an
endpoint. Their architecture dependency lists soft cores and Zynq, which
excludes the host architectures that reach the core over PCIe, x86_64
among them.

PR Type

  • Bug fix (a change that fixes an issue)
  • New feature (a change that adds new functionality)
  • Breaking change (a change that affects other repos or cause CIs to fail)

PR Checklist

  • I have conducted a self-review of my own code changes
  • I have compiled my changes, including the documentation
  • I have tested the changes on the relevant hardware
  • I have updated the documentation outside this repo accordingly
  • I have provided links for the relevant upstream lore

Comment thread drivers/misc/adi-pcie-of-overlay.c Outdated
if (ret)
return ret;

ret = adi_pcie_iommu_setup(apo);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is likely misplaced, will move out from this function

@gastmaier

This comment was marked as outdated.

@gastmaier gastmaier added the llm review Request a review from a LLM Reviewer label Sep 21, 2026
@github-actions

This comment was marked as resolved.

@github-actions

github-actions Bot commented Sep 21, 2026

Copy link
Copy Markdown
Contributor

LLM review

This series adds a PCI driver (adi-pcie-of-overlay) that applies a firmware device-tree
overlay under an ADI PCIe endpoint's OF node so upstream platform drivers can bind to what's
behind its BARs, together with the prerequisites for that (devm_of_pci_make_dev_node(), an
iommu/dma fix for devices that only share another device's IOMMU group), an IRQ-affinity
callback, MSI-X table programming under plain MSI, and Kconfig changes so AXI_DMAC and
ADI_AXI_ADC can build for such PCIe-attached endpoints.

Session: 35642498388

CI: drivers/pci/of.c:505 -fanalyzer NULL deref

Reproduced with KCFLAGS=-fanalyzer; git blame shows the line predates this series
(unrelated merge 5b13d271879e). Pre-existing false positive, resurfaces only because of.c
gains devm_of_pci_make_dev_node() at EOF (0f8cdda2df5c).

8d7a34d3a133 - program the controller's table under plain msi

Self-flagged, still open: the review comment on drivers/misc/adi-pcie-of-overlay.c:456 ("this
is likely misplaced, will move out from this function") is unaddressed through ed9c3d9e.
adi_pcie_msi_hook_install() is called from inside adi_pcie_irq_domain_setup()
(irqdomain/chained-handler wiring) rather than with the rest of one-time controller setup in
adi_pcie_intc_probe(). Not a functional bug — traced that no child can reach an unmasked
interrupt before the hook and its explicit resync run — just the outstanding cleanup.

ed9c3d9eed93 - Kconfig continuation indentation

drivers/iio/adc/Kconfig's new depends on ... \ continuation line is indented one tab deeper
than the near-identical continuation 4cb41af5e0a9 adds to drivers/dma/Kconfig right before it
in the series; checkpatch doesn't check Kconfig continuation alignment so this slipped
through.

Verification data

  • Built adi-pcie-of-overlay.o, dma-axi-dmac.o, adi-axi-adc.o, dma-iommu.o, pci/of.o for
    x86_64, arm and arm64 via set_arch/auto_set_kconfig at this range's base/head SHAs, W=1:
    clean on all three.
  • sparse (make C=2) and checkpatch.pl --strict on every real commit: clean beyond generic,
    pre-existing tool false positives (reproduced on unmodified files).
  • Cross-checked axi_pcie_intc's BAR0 offset against
    analogdevicesinc/hdl@7ad7ac04a47c3c66845d8278e20262176cf6b29c (fetched via the GitHub API):
    BAR0+0x10000 for the interrupt controller matches ADI_PCIE_INTC_OFFSET exactly. The PCIe
    hard IP's own MSI-X table (BAR0+0x8000) is a disjoint 64 KB block from pcie_intc's window, so
    the driver's ADI_PCIE_INTC_MSIX_TABLE (relative to apo->intc, landing at BAR0+0x18000) must
    be a second, IP-internal shadow table — consistent with the commit message, but the
    axi_pcie_intc RTL isn't published yet, so its internal field layout can't be independently
    confirmed beyond that.
  • Traced BUS_NOTIFY_ADD_DEVICE (e35eca13a55e) vs. of_dma_configure() ordering: the notifier
    fires from device_add(), strictly before DMA configuration at probe time — no race between
    IOMMU-group joining and a child's dma-ranges/dma-coherent parsing.

Suggested patches

  • 0001-fixup-misc-adi-pcie-of-overlay-program-the-controlle.patch (8d7a34d3a133) — moves the
    MSI-X table hook install out of adi_pcie_irq_domain_setup() into adi_pcie_intc_probe(),
    resolving the author's own review comment. No functional change; rebuilt clean under W=1.
  • 0002-fixup-iio-adc-adi-axi-adc-allow-build-for-pcie-attac.patch (ed9c3d9eed93) — fixes the
    stray extra tab in the depends on continuation line.

Apply with:

cd path/to/repository
export GITHUB_TOKEN=ghp_***
apply-patches --repo=analogdevicesinc/linux 35642498388
Install instructions

The following one-liner installs the script if not present already:

curl -fSsL "https://raw.githubusercontent.com/analogdevicesinc/doctools/refs/heads/main/ci/scripts/apply-patches.sh"      -o ~/.local/bin/apply-patches.sh &&   grep -q "/apply-patches.sh" ~/.bashrc || echo "source ~/.local/bin/apply-patches.sh" >> $_ ; . $_

More information at AI Usage.

@gastmaier
gastmaier force-pushed the mirror_ci/next/linux-next/master branch from 053bf4d to 4bc9804 Compare September 22, 2026 00:02
@gastmaier

Copy link
Copy Markdown
Collaborator

@rodrigo455 can you cherry-pick your series on next again? i looks like the source branch rebased

@rodrigo455
rodrigo455 force-pushed the staging/misc-adi-pcie-of-overlay branch from ed9c3d9 to 21220ee Compare September 22, 2026 08:49
@gastmaier
gastmaier force-pushed the mirror_ci/next/linux-next/master branch from 4bc9804 to f51c752 Compare September 23, 2026 00:12
Analog Devices FPGA designs can be exposed as a PCIe endpoint whose BARs
window an AXI interconnect full of memory-mapped peripherals: DMA engines,
ADC/DAC interfaces, system ID cores and so on. Those peripherals already
have some upstream drivers, which often rely on dt/fw properties.

This series follows the strategy of drivers/misc/lan966x_pci.c: apply a
device-tree overlay under the endpoint's OF node and let the existing
platform drivers probe the devices it describes. It differs in three ways.

The overlay is requested as firmware, named by a module parameter, rather
than embedded in the module, so one driver serves designs whose contents
differ per bitstream.

The endpoint's OF node is created by the driver. lan966x_pci requires the
PCI core to have made one already (via quirks) and fails if it did not.
Patch 1 adds devm_of_pci_make_dev_node(), which creates the node on demand,
leaves an existing one alone, and removes what it created on unbind.

Interrupts revolve around a dedicated interrupt controller in the FPGA
rather than a single shared line. lan966x_pci maps one dummy_irq_chip
hwirq, which is all a single-interrupt endpoint needs. A design with
several DMA engines needs per-source masking and acknowledgement, so the
int controller behind BAR0 reports how many sources it samples and how
many vectors it implements, and the driver registers a real irq_chip over
it -- mask, unmask, eoi and set_affinity -- with MSI-X, MSI or shared INTx
as the parent. Each source is routed to one of the granted vectors, and
each vector is installed as a chained handler that claims and dispatches
what it collected.

The other half of the problem is DMA. The platform devices created from
the overlay are not PCI devices and have no IOMMU group, so on a
translated host a DMA engine behind a BAR would be programmed with
physical addresses while its transactions arrive at the IOMMU under the
endpoint's requester ID. Patch 4 joins those children to the endpoint's
IOMMU group so the DMA API allocates their IOVAs out of the domain that
will translate them; patch 2 is the iommu/dma prerequisite, since a device
that only shares a group has no per-device IOMMU state.

The driver declares no PCI device table, and binding is left to the user
through driver_override or new_id. Three things have to settle first:
upstreaming of the platform drivers for these peripherals has fallen
behind, so no endpoint yet has all of its contents served by upstream
code; the vendor ID and device ID pair a product will carry is not agreed
on; and the boards tested so far are FPGA-based SoCs, whose designs change
frequently at this initial phase. A later patch can add a table once a
first entry meets all of those requirements.

Patches 7 and 8 let the AXI-DMAC and AXI-ADC drivers build for such an
endpoint. Their architecture dependency lists soft cores and Zynq, which
excludes the host architectures that reach the core over PCIe, x86_64
among them.

To: linux-pci@vger.kernel.org
To: linux-kernel@vger.kernel.org
To: devicetree@vger.kernel.org
To: iommu@lists.linux.dev
To: dmaengine@vger.kernel.org
To: linux-iio@vger.kernel.org
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Rob Herring <robh@kernel.org>
Cc: Saravana Kannan <saravanak@kernel.org>
Cc: Robin Murphy <robin.murphy@arm.com>
Cc: "Joerg Roedel (AMD)" <joro@8bytes.org>
Cc: Will Deacon <will@kernel.org>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Cc: Vinod Koul <vkoul@kernel.org>
Cc: Frank Li <Frank.Li@kernel.org>
Cc: Jonathan Cameron <jic23@kernel.org>
Cc: David Lechner <dlechner@baylibre.com>
Cc: Nuno Sá <nuno.sa@analog.com>
Cc: Andy Shevchenko <andy@kernel.org>
Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>

--- b4-submit-tracking ---
# This section is used internally by b4 prep for tracking purposes.
{
  "series": {
    "revision": 1,
    "change-id": "20260921-misc-adi-pcie-of-overlay-aa1962edca69",
    "prefixes": [],
    "presubject": ""
  }
}
of_pci_make_dev_node() creates an OF node for a PCI device, but its
lifetime belongs to whoever created it: a DT-native system, or a
DECLARE_PCI_FIXUP_FINAL quirk. A driver that describes an endpoint's
internals with a device-tree overlay needs a node to apply that overlay
under, needs it to outlive its own bind, and must not release a node
someone else owns.

Add a devm wrapper covering both cases: leave a pre-existing of_node
alone, otherwise create one and tie its removal to the device. It stubs
out to -ENOENT without CONFIG_PCI_DYNAMIC_OF_NODES, the same option that
already gives the parent bridge the node creation requires.

Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
A device can be a member of another device's IOMMU group without having
been probed by an IOMMU driver: iommu_group_add_device() puts it in the
group but leaves dev->iommu NULL. That fits a device whose transactions
reach the IOMMU tagged as another's, a non-PCI device behind a PCI
endpoint's BARs for instance, which has no requester ID to probe an IOMMU
driver against but whose DMA must still go through the endpoint's domain.

It reaches iommu_dma_alloc_iova() through the DMA API like any other
device and oopses on the dev->iommu deref for the 32-bit-first address
preference. Skip the preference when there is no per-device state to keep
it in, as iommu_deferred_attach() already does; the device then takes
whatever the allocator returns, capped as always by its DMA mask.

Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
Analog Devices designs can be exposed as a PCIe endpoint whose BARs window
an AXI interconnect full of memory-mapped peripherals: DMA engines,
ADC/DAC interfaces, and so on. Those peripherals already have drivers, but
they are device-tree based, and enumerating the endpoint gives no
description of what is behind its BARs.

Add a PCI driver that takes that description from firmware. On probe it
requests the FDT overlay named by the "overlay" module parameter, creates
an OF node for the endpoint, applies the overlay under it and populates
the platform devices so the peripheral drivers probe. Everything is undone
in reverse by devm actions, so an unbind leaves no node, overlay or child
behind.

Interrupts come from a dedicated controller behind BAR0, which reports how
many sources it samples and how many vectors it implements. The driver
registers a linear irqdomain over the sources so overlay nodes can
reference them by number, routes each source to one of the MSI-X, MSI or
shared INTx interrupts it was granted, and installs those as chained
handlers that claim and dispatch what their vector collected.

The driver carries no id_table, so binding is left to the user through
driver_override or new_id: no endpoint whose peripherals are all served by
upstream drivers is available yet, and the FPGA designs are still changing
at this early stage.

Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
…dren

The platform devices created from the overlay are not PCI devices and have
no IOMMU group of their own. On a host where the endpoint is translated
they keep the platform bus default of direct DMA, so a DMA engine behind a
BAR is programmed with physical addresses while its transactions arrive at
the IOMMU as the endpoint's requester ID. The mapping a child driver
installs is never the one its hardware needs.

Join every device probed below the endpoint to the endpoint's IOMMU group,
from a platform bus notifier registered when the endpoint itself is
translated, and mark it as taking the IOMMU DMA path. A child has no
requester ID of its own, so it gets membership only: no domain attach and
no per-device IOMMU state. The DMA API then allocates its IOVAs out of the
endpoint's domain, the one that will translate them, and the iommu core
drops the membership when the device goes away.

An "iommus" property in the overlay cannot do this instead: the stream ID
comes from a requester ID that enumeration assigns, so firmware cannot
name it; the same stream ID handed to an IOMMU driver twice puts the child
in a second group with a second domain, which arm-smmu-v3 rejects as an
unsupported alias; and on x86 there is no IOMMU node to point at.

Children now size their own mappings rather than inheriting the endpoint's
32-bit mask, so the overlay has to describe what its hardware can address
(dma-ranges) and whether it is coherent (dma-coherent), as for any other
DT platform device.

A child that fails to join keeps direct DMA and hands its hardware
untranslated addresses. That cannot be prevented here: device_add()
ignores what the notifier returns, and probe cannot wait for children that
deferred probing and nested buses add later. It is logged.

Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
Sources are routed to vectors at probe and stay there, so irqbalance and
an explicit smp_affinity write on a child's interrupt had nothing to act
on: the source kept arriving on whichever CPU its vector was pinned to.

Move the source between vectors rather than a vector between CPUs. The
parent MSI interrupts keep the CPUs the PCI core gave them, so walk the
granted vectors from the one currently routed, take the first whose
effective affinity intersects the requested mask, reprogram SRC_ROUTE and
report the CPU the source will land on. Fail when no vector serves the
mask, and under INTx where there is nothing to route.

Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
… msi

An MSI-capable controller delivers a vector by writing the address and
data the host assigned to it, which it takes from an MSI-X style table in
its register space. Under MSI-X the PCI core owns that table and programs
it directly. Under plain MSI the core writes one address and data pair
into config space, the table is left holding whatever the previous host
put there, and the endpoint writes to memory that is no longer anyone's.

Install desc->write_msi_msg on the associated descriptors so the core's
message writes reach the table as well, and fill it from the descriptor
the core has already composed. The hook covers a move, a CPU going offline
and a resume, not just activation, since it runs on every write. Entries
are programmed masked and unmasked once the address is non-zero, so a
teardown write leaves the entry masked rather than live on a stale
address. A write arriving while the device is not in D0 is dropped, as the
core drops its own, and replayed at resume.

Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
The AXI-DMAC can also be reached through a PCIe endpoint, as a platform
device created by adi-pcie-of-overlay from a firmware device-tree overlay.
The host architecture is then unrelated to the fabric the core runs in, so
the existing soft-core and Zynq architecture list excludes the very
configurations that need it, x86_64 among them.

Add ADI_PCIE_OF_OVERLAY to the dependency list.

Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
The AXI ADC core can also be reached through a PCIe endpoint, as a
platform device created by adi-pcie-of-overlay from a firmware device-tree
overlay. The host architecture is then unrelated to the fabric the core
runs in, so the existing soft-core and Zynq architecture list excludes the
very configurations that need it, x86_64 among them.

Add ADI_PCIE_OF_OVERLAY to the dependency list.

Signed-off-by: Rodrigo Alencar <rodrigo.alencar@analog.com>
@rodrigo455
rodrigo455 force-pushed the staging/misc-adi-pcie-of-overlay branch from 21220ee to c9bedaf Compare September 23, 2026 12:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

llm review Request a review from a LLM Reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants