HOWTO Create a Devicetree for Agilex SoC

Required environment

This write-up relies on a Linux development host environment with a bash shell.

This write-up references these built-in bash shell functions:

cd echo export read type

This write-up references these standard Linux utilities:

base64 cat column cp dd find git grep ls make mkdir scp sed sort sudo sync tar wget xz

This write-up references these programs provided by the Intel FPGA software tools installation:

quartus_pgm sopc-create-header-files

You can check your current environment by running this loop in your bash shell:

[]$ for NEXT in \
cd echo export read type \
base64 cat column cp dd find git grep ls make mkdir scp sed sort sudo sync tar wget xz \
quartus_pgm sopc-create-header-files
do
type -t ${NEXT:?} > /dev/null &&
{ echo "    located - ${NEXT:?}" ; } ||
{ echo "NOT located - ${NEXT:?}" ; }
done

If your development environment is missing any of the references from above, then you may need to install the packages or tools that provide those dependencies before you can execute some of the examples described below.

Overview

At the time of this writing the current Intel FPGA tools era is 22.2. The references below will be specific to that Intel FPGA software release environment.

This write-up will demonstrate how to create a devicetree for your own custom FPGA hardware implementation in an SoC FPGA device. As a reference and demonstration platform we will refer to materials that are prepared for use on the Agilex SoC development kit. The Golden Hardware Reference Design (GHRD) is a hardware example that is used to create a pre-built Linux SD card example called the Golden System Reference Design (GSRD). The GHRD and GSRD images are distributed on releases.rocketboards.org. We will use the GSRD reference design as the foundation of this write-up and demonstrate how we would create our own devicetree implementation for the GHRD FPGA hardware design as if we were attempting to do this for our own custom hardware platform.

In this write-up we provide many command line examples that are intended to be copied and pasted right on the command line of your Linux development host if you choose to evaluate them. We have made every attempt to create generic command examples that do not incorporate specific environmental requirements that your environment may not provide, but in some cases, you may find that you must edit the example commands to fit your environment more appropriately.

In this write-up we also provide some source file examples and patch examples that you will need to create and apply if you choose to evaluate this content directly. Since the source files and patch files require complex and specific white space that is challenging to represent accurately in various markdown languages that this write-up is distributed in, we have chosen to distribute these source and patch examples as xz compressed base64 encoded text along with command lines that you can use to recreate the precise output that is required for the source and patch files. Here is an example of how this works, you should be able to copy and paste the initial command line below along with all the base64 encoded data through the EOF line onto your Linux command line on your development host and have it produce the results that we show below:

[]$ cat << EOF | base64 -d -i | xz -d
/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4AJGAPRdAASdiQQG8RGxyToYaWl0rK7DBoB3YK3stuMt
gHWBpFTOaL6Q3l77rOjasGselnIn4jTcx7dH78l1cEuNQqasL+6EoEGJ3YTfPY8kk1Hn3SeCf5sN
9YrpTYB8eSZT36lBlNCuGBLE57vg+xwZvy44MmvtDbi6ZapUTEXN8JfQqXAsRzKyuxm91bd8Rmhj
HTTkbeNneZBqRLsR2vcF/XiP+QwFnSQKx2K5GMSQ8PI09lGfjvRoKaBIO6KAaEiFMjIxi6+jIfXM
lymiiHAW5VGrl1+yuBaFBCOZxeHSGll+OZYB2Vm3QzyscokCoknlAgthhLI8fuMARjY/RHmJ0LMA
AZACxwQAAJ3vYEixxGf7AgAAAAAEWVo=
EOF

When we place this decoded content into the write-up, we convert all the tab characters to spaces to maintain the desired representation of the original content but make it easier for the markdown language environments to represent the vertical alignment accurately. These white space substitutions will change the meaning of some of the patch contents, so you cannot simply cut and paste this text representation for its intended use:

        ranges = <
                S10_LW_H2F_BASE SYSID_QSYS_BASE SYSID_QSYS_BASE SYSID_QSYS_SPAN
                S10_LW_H2F_BASE LED_PIO_BASE    LED_PIO_BASE    LED_PIO_SPAN
                S10_LW_H2F_BASE BUTTON_PIO_BASE BUTTON_PIO_BASE BUTTON_PIO_SPAN
                S10_LW_H2F_BASE DIPSW_PIO_BASE  DIPSW_PIO_BASE  DIPSW_PIO_SPAN
                S10_LW_H2F_BASE ILC_BASE        ILC_BASE        ILC_SPAN
        >;

        ILC: ILC@ILC_UNIT_ADDRESS {
                compatible = "altr,ilc-1.0";
                reg = <S10_LW_H2F_BASE ILC_BASE ILC_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <
                        0       BUTTON_PIO_IRQ  IRQ_TYPE_LEVEL_HIGH
                        0       DIPSW_PIO_IRQ   IRQ_TYPE_LEVEL_HIGH
                >;
                altr,sw-fifo-depth = <32>;
        };

Throughout the rest of this write-up, we will provide the xz compressed base64 encoded content followed immediately by the decoded representation with the white space modifications as we just demonstrated above.

Initial Agilex SoC DK board setup, tools setup, and other setup

Start by getting the Agilex SoC DK board set up and configured to run the pre-built SD card image and get the tools and development environment properly configured.

Create a working directory and change into it:

[]$ mkdir agilex_soc_example
[]$ cd agilex_soc_example

Download the JIC image, SD card image and the GHRD project image for the Agilex SoC DK. At the time of this writing the latest version is the 2022.07 release:

[]$ wget https://releases.rocketboards.org/2022.07/gsrd/agilex_gsrd/\
ghrd_agfb014r24b2e2v.jic.tar.gz

[]$ wget https://releases.rocketboards.org/2022.07/gsrd/agilex_gsrd/sdimage.tar.gz

[]$ wget https://releases.rocketboards.org/2022.07/gsrd/agilex_gsrd/\
agilex_revb_soc_devkit_ghrd_enpirion_QPDS-22.2pro-21.1std.tar.gz

Extract the JIC image:

[]$ tar xf ghrd_agfb014r24b2e2v.jic.tar.gz

Extract the SD card image:

[]$ tar xf sdimage.tar.gz

Extract the GHRD project image into a new directory:

[]$ mkdir agilex_revb_soc_devkit_ghrd
[]$ tar -C agilex_revb_soc_devkit_ghrd \
-xf agilex_revb_soc_devkit_ghrd_enpirion_QPDS-22.2pro-21.1std.tar.gz

Program the JIC image into the SDM QSPI flash device, set the MSEL, SW1, dip switch block to 1-ON, 2-ON, 3-ON, 4-OFF power the board up and run the following command:

[]$ quartus_pgm -m jtag -c 1 -o pi\;ghrd_agfb014r24b2e2v.jic

Once the JIC image is programmed into the SDM QSPI flash device, power off the board and return the MSEL, SW1, dip switch block to 1-ON, 2-OFF, 3-OFF, 4-OFF.

Program the SD image onto a micro-SD card:

[]$ sudo dd if=gsrd-console-image-agilex.wic of=<path-to-sd-card> bs=16M
[]$ sync

Plug the micro-SD card into the Agilex SoC DK and boot the board to ensure that you have a working platform. Once you have booted into the Linux environment, verify the Linux kernel version that is being used so that we can create an appropriate devicetree of our own for that Linux kernel environment. Devicetrees are specific to each kernel release, the node, and binding requirements for the devicetree can and often do change from kernel release to kernel release.

Log into the target and run the 'uname' command to determine the kernel version:

agilex login: root
root@agilex:~# uname -a
Linux agilex 5.15.30-altera #1 SMP PREEMPT Fri Jun 24 03:52:26 UTC 2022 aarch64 GNU/Linux

In this case you can see that this version of the pre-built SD card contains a 5.15.30 kernel version, and it was built in June 2022. We will use that information to select the Linux kernel commit that we check out below.

Clone the linux-socfpga git repository and change into the directory:

[]$ git clone https://github.com/altera-opensource/linux-socfpga.git
[]$ cd linux-socfpga

We need to select a commit to check out of the repo that is on the branch that the kernel was built with above. We don't need the exact commit that was used to build the kernel we are running but we want something close to that kernel. The repo is consistently tagged with branch release tags in the form "rel_<branch-name>_<YY>.<MM>.<release>", so if we grep the git tags from the repo looking for the version that we see in the output above we can see the tags associated with that branch of the kernel from the repo:

[]$ git tag | grep 5.15.30
rel_socfpga-5.15.30-lts_22.08.02_pr
rel_socfpga-5.15.30-lts_22.09.01_pr

We can see that the "socfpga-5.15.30-lts" kernel tags are present, and we see that the earliest release was tagged in August of 2022. We will use the 22.08.02_pr tag as our target. We check out that commit on a new branch with the same name like this:

[]$ git checkout \
-b rel_socfpga-5.15.30-lts_22.08.02_pr \
rel_socfpga-5.15.30-lts_22.08.02_pr

If you do not have a cross compiler to compile the Linux kernel for Armv8 cores, download one now:

[]$ wget https://developer.arm.com/-/media/Files/downloads/gnu-a/10.3-2021.07/\
binrel/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz

[]$ tar xf gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz

Setup the environment so the Linux kernel build environment understands what we are building for:

[]$ export ARCH=arm64
[]$ export CROSS_COMPILE=\
<path-to>/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-

You may want to verify that the compiler can be located in your environment, you can test that with the type command:

[]$ type -P ${CROSS_COMPILE:?}gcc
<path-to>/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gcc

Agilex SoC devicetree orientation

Our strategy for building our own devicetree will be to leverage the base definitions for the Hard Processor System (HPS) that are up-streamed into the Linux source tree already. In fact, there is an example devicetree for the Agilex SoC DK that is already available in the source tree. If we look for the top-level devicetree file for the Agilex SoC DK we should find it here:

[]$ find -name socfpga_agilex_socdk.dts
./arch/arm64/boot/dts/intel/socfpga_agilex_socdk.dts

Or in your web browser you can look here.

This socfpga_agilex_socdk.dts file is the top-level devicetree file for the Agilex SoC DK board and we will want to create something that looks just like this for our own example, but we will need to add the FPGA specific devicetree requirements to our example. We may want to use this file as a starting reference point, but ultimately, we will need to create our own top-level devicetree file. This default devicetree example for the Agilex SoC DK is only concerned about the HPS hardened peripherals, which do not ever change, so this devicetree can be defined in such a way that it can apply to any other Agilex SoC design, and that is what this default example demonstrates. This top-level socfpga_agilex_socdk.dts file simply includes other more fundamental devicetree include files which define more of the lower-level details of the HPS core itself. In this top-level file we see only the top-level enabling of the resources defined in the lower level include files as well as any board level details that interact with the HPS peripherals that are enabled.

The socfpga_agilex_socdk.dts file includes the lowest level definition of the HPS for the Agilex SoC device in a file called socfpga_agilex.dtsi here:

[]$ find -name socfpga_agilex.dtsi
./arch/arm64/boot/dts/intel/socfpga_agilex.dtsi

Or in your web browser you can look here.

This socfpga_agilex.dtsi file contains the definition of all the low-level hardware built into the Agilex SoC HPS core. It carries the required clocking structure for the Linux clock framework as well as the devicetree bindings for all the HPS IO peripherals. If you look at this devicetree include file closely you will see that the peripherals that are internal to the HPS core are all enabled by default, i.e., they have no status binding which means they default to enabled. But all the IO peripherals that interact through device pins are marked as disabled, i.e., they all have a status binding that is set to disabled in this devicetree include file. If you look at the top-level file that includes this file, socfpga_agilex_socdk.dts you can see that one of the things it does is redefine the status binding for all the HPS IO peripherals that it wants to enable.

At the socfpga_agilex.dtsi level of the devicetree definition, they primarily try to define the bindings for all the peripherals that are not board specific, so you will see compatible, reg, interrupts, clocks and any other well-defined bindings that are unlikely to change for any implementation of the Agilex SoC HPS core. Then at the top-level devicetree file, for example in socfpga_agilex_socdk.dts you would need to enable the peripherals that you require and add any bindings to peripherals that are board specific parameters, and you can even override settings for bindings that are defined in the socfpga_agilex.dtsi level of the devicetree if needed.

You should familiarize yourself with the layout and contents of the Agilex SoC DK example devicetree top-level file and all the include files that it leverages. Our own devicetree example will begin by leveraging all that existing material.

Build and test the default Agilex SoC DK example devicetree

Build the default example for the Agilex SoC DK board that is in the kernel source tree. Make a copy of the original to modify and preserve the original example file.

Start by configuring the Linux kernel with the defconfig:

[]$ make defconfig

Make a copy of the socfpga_agilex_socdk.dts file for our own use:

[]$ cp \
./arch/arm64/boot/dts/intel/socfpga_agilex_socdk.dts \
./arch/arm64/boot/dts/intel/my_agilex_top.dts

Edit this top-level devicetree source file to create a unique value for the model binding so that we can identify this devicetree when we run it on the target:

[]$ sed -i "s/SoCFPGA Agilex SoCDK/DT TEST AGILEX SX DK/" \
./arch/arm64/boot/dts/intel/my_agilex_top.dts

To build the devicetree, simply call make with the name of the DTB file that you want to compile:

[]$ make intel/my_agilex_top.dtb

This works because the default Linux Makefile infrastructure has recipes for building devicetree sources and producing binary devicetree blobs. The Linux Makefile architecture passes all the devicetree sources through the GNU C preprocessor to preprocess macros and other syntax, and then it passes the preprocessed results through the DTC compiler to produce the resultant binary devicetree blob. You could do this on your own without leveraging the Linux kernel build environment if you wanted to, you would do something like this:

[]$ ${CROSS_COMPILE}gcc \
        -E \
        -nostdinc \
        -I./arch/arm64/boot/dts/intel \
        -I./include \
        -undef -D__DTS__ \
        -x assembler-with-cpp \
        -o arch/arm64/boot/dts/intel/my_agilex_top.dts.tmp \
        arch/arm64/boot/dts/intel/my_agilex_top.dts

[]$ ./scripts/dtc/dtc \
        -O dtb \
        -o arch/arm64/boot/dts/intel/my_agilex_top.dtb \
        -i arch/arm64/boot/dts/intel \
        -@ \
        arch/arm64/boot/dts/intel/my_agilex_top.dts.tmp

You should be able to locate the DTB that was just built:

[]$ find -name my_agilex_top.dtb
./arch/arm64/boot/dts/intel/my_agilex_top.dtb

Move this DTB file onto the Agilex SoC DK target. There are several ways to do this, we will demonstrate moving the file over Ethernet using scp to our target board and then positioning the new DTB where it needs to go.

Start by copying the file to the target:

[]$ scp ./arch/arm64/boot/dts/intel/my_agilex_top.dtb root@<TARGET-IP-ADDR>:/home/root/

We need to get the new DTB onto the FAT partition of the SD card, so we mount the FAT partition:

root@agilex:~# mount /dev/mmcblk0p1 /mnt

We copy our new DTB into the proper location:

root@agilex:~# mv my_agilex_top.dtb /mnt

Before we try out our new DTB, look at the model string that was defined in the original DTB that is running on the target:

root@agilex:~# (echo ""; cat /proc/device-tree/model; echo "" ;)

SoCFPGA Agilex SoCDK

We can reboot the target to have Linux load our new example DTB, remember to stop u-boot at the autoboot console:

root@agilex:~# reboot

...cut linux shutdown and u-boot startup messages...

Hit any key to stop autoboot:  0
SOCFPGA_AGILEX #

Once we reach the u-boot console we must manually boot the Linux kernel with our new devicetree since the default environment attempts to bootm from a configuration in the kernel FIT image on the FAT partition and we do not want that version of the devicetree to be used. We do want to reuse the FPGA core RBF configuration file and the Linux kernel from the FIT image.

Start by loading the kernel FIT image from mmc flash into memory:

SOCFPGA_AGILEX # load mmc 0:1 ${loadaddr} kernel.itb
15111707 bytes read in 687 ms (21 MiB/s)

Program the fpga-4 RBF image from the FIT image into the FPGA core logic and then enable the FPGA bridges:

SOCFPGA_AGILEX # imxtract ${loadaddr} fpga-4 ${kernel_addr_r}
## Copying 'fpga-4' subimage from FIT image at 01000000 ...
crc32+    Loading part 15 ... OK

SOCFPGA_AGILEX # fpga load 0 ${kernel_addr_r} ${filesize}
...FPGA reconfiguration OK!

SOCFPGA_AGILEX # bridge enable

Decompress the kernel image from the FIT image:

SOCFPGA_AGILEX # fdt addr ${loadaddr}
SOCFPGA_AGILEX # fdt get addr KERNEL_ADDR /images/kernel data
SOCFPGA_AGILEX # lzmadec ${KERNEL_ADDR} ${kernel_addr_r}
Uncompressed size: 34345472 = 0X20C1200

Load our custom devicetree image from mmc flash into memory:

SOCFPGA_AGILEX # load mmc 0:1 ${fdt_addr_r} my_agilex_top.dtb
16851 bytes read in 4 ms (4 MiB/s)

Boot the Linux kernel:

SOCFPGA_AGILEX # setenv bootargs "earlycon panic=-1 root=${mmcroot} rw rootwait"
SOCFPGA_AGILEX # booti ${kernel_addr_r} - ${fdt_addr_r}

The target should boot back into the Linux environment just as it did before, however none of the FPGA peripherals are defined in this default example devicetree. We will demonstrate how to add the FPGA peripherals into the devicetree next.

Once the target boots up to the Linux environment again, check the model string of the devicetree and make sure that it has updated to the value stored in the example that we just created and loaded onto the target:

root@agilex:~# (echo ""; cat /proc/device-tree/model; echo "" ;)

DT TEST AGILEX SX DK

What should the devicetree enable in the HPS?

We should be running the Agilex SoC DK example devicetree from the kernel source tree that we copied into the my_agilex_top.dts file and compiled into a DTB to load onto the target and run. In this devicetree that we copied from socfpga_agilex_socdk.dts, we can see that the following HPS IO peripherals have been enabled: gpio1, gmac0, mmc, uart0, usb0, and qspi.

There are a few ways to determine what HPS peripherals have been enabled in our hardware project and therefore should be enabled in the devicetree. One way to determine this is to open the HPS peripheral configuration in the Platform Designer system that the component is instantiated in, then you can manually look at the parameterization that selected the peripherals in your design. Another way to determine this is to look for the interface definitions that get output in the SOPCINFO file generated by Platform Designer. Here is an example sed parsing pattern to show us what HPS interfaces were enabled in the Agilex SoC GHRD design which is what we are running in the SD card image that we programmed into the board:

[]$ cat ../agilex_revb_soc_devkit_ghrd/qsys_top/qsys_top.sopcinfo | \
sed -E -e "/<module/,/<\/module>/P" -e "D" | \
sed -E -e "/kind=\"intel_agilex_hps\"/,/<\/module>/P" -e "D" | \
sed -E -e "/<boundary>/,/<\/boundary>/P" -e "D" | \
sed -E -e "/<ports>/,/<\/ports>/D" | \
sed -E -e "/<name>.*<\/name>/P" -e "D" | \
sed -E -e "s/^.*<name>(.*)<\/name>.*$/\1/" | \
sort | column
emac1                   f2h_axi_reset           h2f_lw_axi_clock
emac1_gtx_clk           f2h_axi_slave           h2f_lw_axi_master
emac1_md_clk            f2h_irq0                h2f_lw_axi_reset
emac1_rx_clk_in         f2h_irq1                h2f_reset
emac1_rx_reset          f2h_stm_hw_events       h2f_watchdog_rst
emac1_tx_clk_in         h2f_axi_clock           hps_emif
emac1_tx_reset          h2f_axi_master          hps_io
emac_ptp_ref_clock      h2f_axi_reset
f2h_axi_clock           h2f_cs

From the list above we can see that emac1 gets routed into the FPGA which means that we may eventually need to enable it in the devicetree, but we should not enable it until the FPGA logic that interfaces with that peripheral is properly programmed into the FPGA core logic. The interfaces represented by the f2h_* and h2f_* are bridge interfaces between the HPS and FPGA which do not really require specific devicetree enablement. The hps_emif interface represents the DRAM interface which also does not require devicetree enablement. The hps_io interface represents the HPS IO peripherals that get pin muxed off the device through the HPS IO bank and if we look at what ports it has enabled, we can understand other HPS IO peripherals that may require devicetree enablement:

[]$ cat ../agilex_revb_soc_devkit_ghrd/qsys_top/qsys_top.sopcinfo | \
sed -E -e "/<module/,/<\/module>/P" -e "D" | \
sed -E -e "/kind=\"intel_agilex_hps\"/,/<\/module>/P" -e "D" | \
sed -E -e "/<boundary>/,/<\/boundary>/P" -e "D" | \
sed -E -e "/<interface>/,/<\/interface>/P" -e "D" | \
sed -E -e "/<name>hps_io<\/name>/,/<\/interface>/P" -e "D" | \
sed -E -e "/<ports>/,/<\/ports>/P" -e "D" | \
sed -E -e "/<role>.*<\/role>/P" -e "D" | \
sed -E -e "s/^.*<role>(.*)<\/role>.*$/\1/" | \
sort | column
EMAC0_MDC       EMAC0_TXD0      gpio1_io5       SDMMC_D0        USB0_DATA3
EMAC0_MDIO      EMAC0_TXD1      hps_osc_clk     SDMMC_D1        USB0_DATA4
EMAC0_RX_CLK    EMAC0_TXD2      I2C1_SCL        SDMMC_D2        USB0_DATA5
EMAC0_RX_CTL    EMAC0_TXD3      I2C1_SDA        SDMMC_D3        USB0_DATA6
EMAC0_RXD0      gpio1_io0       jtag_tck        UART0_RX        USB0_DATA7
EMAC0_RXD1      gpio1_io1       jtag_tdi        UART0_TX        USB0_DIR
EMAC0_RXD2      gpio1_io19      jtag_tdo        USB0_CLK        USB0_NXT
EMAC0_RXD3      gpio1_io20      jtag_tms        USB0_DATA0      USB0_STP
EMAC0_TX_CLK    gpio1_io21      SDMMC_CCLK      USB0_DATA1
EMAC0_TX_CTL    gpio1_io4       SDMMC_CMD       USB0_DATA2

As you can see, the peripheral list that was enabled in Platform Designer for our HPS matches the list that we saw in the example devicetree, which makes sense. We see the emac0, gpio1, sdmmc, uart0 and usb0 peripherals represented in the pin mux list and the example devicetree enabled drivers for each of those. The example devicetree also enabled the qspi peripheral which is not allocated on HPS IO pins but it is allocated on the SDM IO pins of the device. The hps_osc_clk and jtag_* ports represent clock and debug interface pins which are not peripherals that require devicetree enablement. The I2C1_* ports represent an I2C interface that the default devicetree does not enable.

If you would like to locate the Linux device drivers and Linux documentation for the HPS peripherals, you can grep the Documentation and drivers directories in the kernel source tree for the compatibility strings that you observe in the HPS devicetree nodes. For Agilex SoC, you could for instance run this bash command sequence to do just that:

[]$ grep "compatible" ./arch/arm64/boot/dts/intel/socfpga_agilex.dtsi | \
sort -u | \
sed -e "s/^\s*\(.*\)/\\1/" | \
while read NEXT ; do \
echo $NEXT ; \
echo $NEXT | \
sed -e 's/[^"]*["]\([^"]*\)["][^"]*/\1\n/g' | \
while read NEXT1 ; do \
if [ "$NEXT1" != "syscon" ] ; then \
if [ -n "$NEXT1" ] ; then \
grep -Rl "$NEXT1" Documentation/* ; \
grep -Rl "$NEXT1" drivers/* ; \
fi ; \
fi ;
done | sort -u ; \
echo "" ; \
done

What should the devicetree enable in the FPGA?

The primary value of the devicetree is to inform kernel drivers and modules of the peripheral implementation parameters that the hardware design contains. If you are putting your own custom hardware into the FPGA, then you will be the one to decide how you plan to implement the software to interact with that hardware. If you plan to write a user space device driver for your hardware, you may find no need whatsoever in having devicetree entries that define your hardware configuration. Even if you plan to write a kernel mode driver for your hardware, you may or may not want to have your driver configured by parameters stored in a devicetree. A software driver only needs to have devicetree entries if it is built to use them and there are ways to build a driver such that it does not use devicetree entries. Your personal preferences and deployment goals for your product should dictate if and how you use the devicetree to configure your driver. On the other hand, there is no reason you cannot stuff the devicetree full of arbitrary information that no drivers or modules care about. The kernel exposes all the devicetree structure in the procfs and sysfs, so you can observe all the devicetree information from user space which may be useful for some implementations.

If you decide that you would prefer to use devicetree entries to inform your driver of your hardware configuration, then you need to know how to implement them properly in the devicetree. You may also use other standard components in the FPGA which have software device drivers that will rely on devicetree parameters to configure themselves at run time, so you need to ensure that you have properly configured those as well.

In the GHRD example FPGA hardware design that we are currently running on the Agilex SoC DK there are a few peripherals which have device drivers that are configured via devicetree. How can we identify what FPGA peripherals we have? And which of those need to be entered in our devicetree?

Again, we could open our Platform Designer system up and manually inspect it to see what peripherals we have instantiated in the FPGA, but we are going to use a utility called sopc-create-header-files to help us decode the data that is generated into the SOPCINFO file and extract the relevant information that we need into C formatted header files.

Start by creating a directory to capture the headers and generate the headers:

[]$ cd ..
[]$ mkdir pd_headers
[]$ sopc-create-header-files \
./agilex_revb_soc_devkit_ghrd/qsys_top/qsys_top.sopcinfo \
--output-dir pd_headers
[]$ cd pd_headers

That command should have produced several header files for our GHRD hardware system from the perspective of each master in the system, as well as some system wide perspectives:

[]$ ls
agilex_hps.h             jtg_mst_hps_m.h         qsys_top.h
fpga_m2ocm_pb.h          jtg_mst_hps_m_master.h  subsys_sgmii_emac1_sgmii_csr.h
jtg_mst_fpga_m.h         niosv_cpu.h
jtg_mst_fpga_m_master.h  periph_pb_cpu_0.h

We can use a few grep patterns to extract the base address of the FPGA peripherals in our hardware design:

[]$ cat qsys_top.h | \
grep -E -e "#define AGILEX_HPS_" | \
grep -E -e "_BASE\s"
#define AGILEX_HPS_OCM_BASE 0x0
#define AGILEX_HPS_SYSID_BASE 0x0
#define AGILEX_HPS_PERIPH_BUTTON_PIO_BASE 0x1060
#define AGILEX_HPS_PERIPH_DIPSW_PIO_BASE 0x1070
#define AGILEX_HPS_PERIPH_LED_PIO_BASE 0x1080
#define AGILEX_HPS_PERIPH_ILC_BASE 0x1100
#define AGILEX_HPS_SUBSYS_SGMII_EMAC1_ETH_TSE_0_BASE 0x3000
#define AGILEX_HPS_SUBSYS_SGMII_EMAC1_GMII_SGMII_ADAPTER_0_BASE 0x3040
#define AGILEX_HPS_SUBSYS_SGMII_EMAC1_SGMII_DEBUG_STATUS_PIO_BASE 0x3050

There are many more peripherals in this GHRD example than we care about for our devicetree example. And we can also observe how this output only contains the offset to the base address of the FPGA peripherals from the bridge that they are connected to, so we will need to add the offset to the base address of each bridge before we use these offsets to the peripheral in the devicetree definition. We can tell which peripherals are connected to each bridge on the HPS by opening the system in the Platform Designer GUI and visually inspecting it, or we could also parse this information from the connection properties captured in the SOPCINFO file. If we want to see the connections to the "h2f_axi_master" and the "h2f_lw_axi_master" on the HPS we can parse the SOPCINFO file like this:

[]$ cat ../agilex_revb_soc_devkit_ghrd/qsys_top/qsys_top.sopcinfo | \
sed -E -e "/^\s*<connection$/,/^.*>/P" -e "D" | \
sed -E -e "/name=/P" -e "D" | \
sed -E -e "/agilex_hps\./P" -e "D" | \
sed -E -e "/(h2f_lw_axi_master|h2f_axi_master)/P" -e "D" | \
sed -E -e "s/.*name=\"(.*)\".*/\1/" | \
sort
agilex_hps.h2f_axi_master/ocm.s1
agilex_hps.h2f_lw_axi_master/periph_pb_cpu_0.s0
agilex_hps.h2f_lw_axi_master/subsys_sgmii_emac1_sgmii_csr.s0
agilex_hps.h2f_lw_axi_master/sysid.control_slave

We can see that there is only one peripheral connected to the "h2f_axi_master" bridge, the onchip memory peripheral. That peripheral does not require a driver so there are no devicetree bindings required for it. The rest of the peripherals in the FPGA are connected through the "h2f_lw_axi_master" bridge, but we are only interested in two of those, the "sysid" and the "periph_pb_cpu_0". The "sysid" peripheral is a standard system ID peripheral which has a default Linux driver and the "periph_pb_cpu_0" peripheral is a bridge that connects to more peripherals downstream inside the FPGA. We can discover the peripherals connected to the "periph_pb_cpu_0" bridge by parsing the SOPCINFO file like this:

[]$ cat ../agilex_revb_soc_devkit_ghrd/qsys_top/qsys_top.sopcinfo | \
sed -E -e "/^\s*<connection$/,/^.*>/P" -e "D" | \
sed -E -e "/name=/P" -e "D" | \
sed -E -e "/periph_pb_cpu_0\./P" -e "D" | \
sed -E -e "/m0/P" -e "D" | \
sed -E -e "s/.*name=\"(.*)\".*/\1/" | \
sort
periph_pb_cpu_0.m0/periph_button_pio.s1
periph_pb_cpu_0.m0/periph_dipsw_pio.s1
periph_pb_cpu_0.m0/periph_ILC.avalon_slave
periph_pb_cpu_0.m0/periph_led_pio.s1

We can see that there are four peripherals connected here, three PIO peripherals and one Interrupt Latency Counter peripheral. For our devicetree example, we are only interested in the three PIO peripherals, the Interrupt Latency Counter and the system ID peripheral since they all have Linux device drivers that we can enable in the devicetree. All these peripherals are connected to the LW HPS AXI bridge so we will align their base address to that bridge in the devicetree.

Each peripheral in the devicetree is going to require certain fundamental bindings like the compatible, reg, and interrupts bindings. And some of the peripherals may require additional bindings that are unique to that peripheral. Here is how we find some of the basic information that each peripheral will require for the fundamental bindings.

We can find the base address and span of each peripheral required by the reg binding by looking at the general parameters exposed in the Platform Designer header data, a broad filter like this will expose much of it:

[]$ cat qsys_top.h | \
grep -E -e "#define AGILEX_HPS_"

Output not shown.

The way Agilex FPGA interrupts translate into the HPS GIC is architecture specific. We can use this filter to extract the interrupt information from the header file like this:

[]$ cat qsys_top.h | \
grep -E -e "#define AGILEX_HPS_" | \
grep -E -e "_IRQ\s"
#define AGILEX_HPS_PERIPH_BUTTON_PIO_IRQ 1
#define AGILEX_HPS_PERIPH_DIPSW_PIO_IRQ 0

We can see from this that we have an interrupt driven by two of the pio peripherals. These interrupt values indicate the position of these interrupts on the FPGA-to-HPS interrupt receiver which is a 64-bit vector. These pio interrupts are assigned to bits 0, and 1 of that vector. On Agilex SoC devices the 0th bit of that vector represents interrupt 49 into the GIC of the ARM processor. In the devicetree we must subtract off the first 32 interrupt locations that represent inter-processor interrupts. We can define an equation like this to represent these FPGA driven interrupts in the devicetree, FPGA-interrupt-vector + 49 - 32. And when you observe the devicetree examples that we create below you will see that we implement this equation for the interrupt entries.

For more information on how the ARM GIC expects us to format interrupt entries in the devicetree, you can look at this document:

[]$ find ../linux-socfpga/ -name arm,gic.yaml
../linux-socfpga/Documentation/devicetree/bindings/interrupt-controller/arm,gic.yaml

Or in your web browser you can look here.

The pio peripherals will need some additional information provided in their devicetree bindings, so while we are filtering the Platform Designer header, we can extract that information like this:

[]$ cat qsys_top.h | \
grep -E -e "#define AGILEX_HPS_" | \
grep -E -e "PIO_DATA_WIDTH\s" -e "PIO_EDGE_TYPE\s" -e "PIO_IRQ_TYPE\s"
#define AGILEX_HPS_PERIPH_BUTTON_PIO_DATA_WIDTH 4
#define AGILEX_HPS_PERIPH_BUTTON_PIO_EDGE_TYPE FALLING
#define AGILEX_HPS_PERIPH_BUTTON_PIO_IRQ_TYPE EDGE
#define AGILEX_HPS_PERIPH_DIPSW_PIO_DATA_WIDTH 4
#define AGILEX_HPS_PERIPH_DIPSW_PIO_EDGE_TYPE FALLING
#define AGILEX_HPS_PERIPH_DIPSW_PIO_IRQ_TYPE EDGE
#define AGILEX_HPS_PERIPH_LED_PIO_DATA_WIDTH 3
#define AGILEX_HPS_PERIPH_LED_PIO_EDGE_TYPE NONE
#define AGILEX_HPS_PERIPH_LED_PIO_IRQ_TYPE NONE
#define AGILEX_HPS_SUBSYS_SGMII_EMAC1_SGMII_DEBUG_STATUS_PIO_DATA_WIDTH 13
#define AGILEX_HPS_SUBSYS_SGMII_EMAC1_SGMII_DEBUG_STATUS_PIO_EDGE_TYPE RISING
#define AGILEX_HPS_SUBSYS_SGMII_EMAC1_SGMII_DEBUG_STATUS_PIO_IRQ_TYPE NONE

How did we know that the pio would require some additional binding information? And what other bindings might be needed by any of these peripherals? We can figure out what standard Altera peripherals may have drivers and require devicetree bindings by grep'ing for the string "altr," in the Documentation and drivers directories in the Linux source tree, like this:

[]$ grep -Rl "altr," ../linux-socfpga/Documentation/*

NOTE: with the Intel acquisition of Altera, you are now starting to see new drivers for Intel FPGA devices which carry the "intel," prefix rather than the "altr," prefix that the legacy Altera components were created with.

Out of the list created by the grep command above, we are interested in these three documents that cover the peripherals that we have in our GHRD hardware design:

[]$ find ../linux-socfpga/Documentation/ \
-name altera_sysid.txt -o \
-name gpio-altera.txt -o \
-name altera-interrupt-latency-counter.txt
../linux-socfpga/Documentation/devicetree/bindings/gpio/gpio-altera.txt
../linux-socfpga/Documentation/devicetree/bindings/misc/altera-interrupt-latency-counter.txt
../linux-socfpga/Documentation/devicetree/bindings/misc/altera_sysid.txt

These files can be viewed in a web browser here:

• gpio-altera.txt
• altera-interrupt-latency-counter.txt
• altera_sysid.txt

The one thing that each of these files will provide is the compatibility string that the driver expects. The pio and ILC peripherals have several other bindings that are required and optional that you can include in the devicetree.

With this information about where we can find the details for peripherals that exist inside our FPGA Platform Designer system and how to locate software driver documentation for existing Altera peripherals, we can now proceed to add the FPGA peripherals to our existing devicetree example.

Add the FPGA peripherals to the Agilex SoC devicetree

There are several ways to add the FPGA peripherals into the main devicetree, this example will demonstrate two practical and popular ways of doing this. First, we demonstrate how to create a devicetree overlay which can be dynamically applied to and removed from the main system devicetree at run-time. Then we demonstrate how to just embed our FPGA peripherals into the main system devicetree so that it loads at boot with the rest of the devicetree. Since we left the Agilex SoC DK in a state where it is running the base example devicetree with no FPGA contents, we will begin with the overlay example first.

Copy the qsys_top.h header into the devicetree source directory in the kernel source tree and change into the kernel source tree directory:

[]$ cp qsys_top.h ../linux-socfpga/arch/arm64/boot/dts/intel/
[]$ cd ../linux-socfpga

Devicetree overlay method

Create a new file called arch/arm64/boot/dts/intel/my_agilex_fpga_overlay.dts to store our FPGA peripheral definition as a devicetree overlay:

[]$ cat << EOF | base64 -d -i | xz -d > arch/arm64/boot/dts/intel/my_agilex_fpga_overlay.dts
/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4A5gA/pdABeKfUvNK1NbzTI+B3OUEJMYagjnn0bslkyF
7xuB8nrKCzWcgqNMVnytO8kU434lb6O/lVpXis+nN97FmMmUs2NSmBw+BzMMy9zaK6yGnL1+gDhJ
NSaOGp7SxBv9NFQb+t/EZduLl7lBQ/A6ng6jkddLTY2Wr0tJnArh9Igkc41FkLLhYu6utx4tAQ+c
W9b0H6/Vn/Ur90lI7IizHnZgl0rEOEk801kapWbgnvol+o7kcFRm9WZQP+Dhfnc7zejyWLH2QUG6
8NlkgCKSWk6OJHrJRxJGzzzsEsRMi0pDvAc7Ig2TdXm+aYXbhbgdOrzRCm93pmq3keAHULU/2CgB
TpvCoC5iPCdrLhGcrAX6jSg2Vnk/S+FgMKzaQZgYrxY+krG+HWnuhbYKncTCsFcKFf/QPYXxJvZQ
CGiF0Q9zhFr5HiBmO8xEnKFwMBuA6z0VvX/GbeoQnESuls8fsBfWyQDm5TpKoYUK4vtstldM6wzg
PgXL2Pz3hJGOlAyZUI2XqMSFxpJhxkub/KWtllN3Q/wzrWLknXqEPynheBDbWuR9wJNjy78w0RMO
yCWCTq4wiErwCd0vEqnmWDjxW5sc845Udc5zKm6ymnjHXY/FlwL5acIXO3a0gTrGEaFXlmk043rz
XCeUQBc2OYcaKcow/Q97oNiHc1ox9KajDV4SEudPtuCu6FOu8ny/OWZg06K/ACLkX3z660/C91Wh
YFMjufB23uPTxxNk9WKdr+AiCQTpacgZMDMlH6fLmLXIaVkcs7Nz+P8OxRVTfVnKrzPzemN64J2F
aEkntvGQTNjItwBRA0/XSYJB0zanEYsxTJ+TzLqPLvA7YJtCdGmIU0HxEppw6+3bC2kARffX6zaY
TNMjqPCTpLLpmnjWpt29QAfUej9dEitpmWA+p7OyiA3fXCkmaqrRVla7v4a6VjiWl07x2a4stuNV
CtlaCtTUSw5C+azPRdz7uqIJ1TnJT9/3hKbHJDqwXrX6yFN/QgLgAw57w9IsyWiRSjxoItFSfEoG
q65WqkHSsBOXC9Tt86tyCZMZYQQuYx3xWfDEJS26v/3awAVolSdw2a06YyzlbrQXoK+I5ueCxrDh
6Rl8PhjgChLRoiz0q7C+qT/6qtRFgoUo9HLvoB4s/tN7cI5YAgTVqinBWKmS+3WX7U48smvVEmK7
5r0aSZrcTQ0kY9AbkMQVYH+fWbkW0yAuKXMd+p6x9JlbKQadZp1yaRlCKtLoYuD3ojTm6I+Z7xML
3UFvFmHVoUhjkUhmzG7uDDBUJw0gVrmCWC/1jjcT+Bw4h5uewLOM/DzR4YN+ApcL6Ey+jbaO5Y6E
uo7Hq1Ob/JwGKZ38q89k9LMGLXcYXDoAAAAiFhTgtxEEuQABlgjhHAAACxblQ7HEZ/sCAAAAAARZ
Wg==
EOF

/*
 * SPDX-License-Identifier: MIT-0
 * Copyright (c) 2018-2022 Intel Corporation
 */

/dts-v1/;
/plugin/;

#include "qsys_top.h"
#include "dt-bindings/interrupt-controller/irq.h"
#include "dt-bindings/gpio/gpio.h"

/*
 * Redefine the macros from qsys_top.h into shorter macros and more useful
 * values that we can use directly in the device tree definition.  We will also
 * take care of the bridge offsets that we require for the true base address of
 * the H2F and LW_H2F bridge.
 */

#define AGILEX_H2F_BASE         0x80000000
#define AGILEX_LW_H2F_BASE      0xF9000000

#define SYSID_QSYS_UNIT_ADDRESS AGILEX_LW_H2F_BASE,AGILEX_HPS_SYSID_BASE
#define SYSID_QSYS_BASE         (AGILEX_LW_H2F_BASE + AGILEX_HPS_SYSID_BASE)
#define SYSID_QSYS_SPAN         AGILEX_HPS_SYSID_SPAN

#define LED_PIO_UNIT_ADDRESS    AGILEX_LW_H2F_BASE,AGILEX_HPS_PERIPH_LED_PIO_BASE
#define LED_PIO_BASE            (AGILEX_LW_H2F_BASE + AGILEX_HPS_PERIPH_LED_PIO_BASE)
#define LED_PIO_SPAN            AGILEX_HPS_PERIPH_LED_PIO_SPAN
#define LED_PIO_WIDTH           AGILEX_HPS_PERIPH_LED_PIO_DATA_WIDTH

#define BUTTON_PIO_UNIT_ADDRESS AGILEX_LW_H2F_BASE,AGILEX_HPS_PERIPH_BUTTON_PIO_BASE
#define BUTTON_PIO_BASE         (AGILEX_LW_H2F_BASE + AGILEX_HPS_PERIPH_BUTTON_PIO_BASE)
#define BUTTON_PIO_SPAN         AGILEX_HPS_PERIPH_BUTTON_PIO_SPAN
#define BUTTON_PIO_IRQ          (AGILEX_HPS_PERIPH_BUTTON_PIO_IRQ + 49 - 32)
#define BUTTON_PIO_WIDTH        AGILEX_HPS_PERIPH_BUTTON_PIO_DATA_WIDTH

#define DIPSW_PIO_UNIT_ADDRESS  AGILEX_LW_H2F_BASE,AGILEX_HPS_PERIPH_DIPSW_PIO_BASE
#define DIPSW_PIO_BASE          (AGILEX_LW_H2F_BASE + AGILEX_HPS_PERIPH_DIPSW_PIO_BASE)
#define DIPSW_PIO_SPAN          AGILEX_HPS_PERIPH_DIPSW_PIO_SPAN
#define DIPSW_PIO_IRQ           (AGILEX_HPS_PERIPH_DIPSW_PIO_IRQ + 49 - 32)
#define DIPSW_PIO_WIDTH         AGILEX_HPS_PERIPH_DIPSW_PIO_DATA_WIDTH

#define ILC_UNIT_ADDRESS        AGILEX_LW_H2F_BASE,AGILEX_HPS_PERIPH_ILC_BASE
#define ILC_BASE                (AGILEX_LW_H2F_BASE + AGILEX_HPS_PERIPH_ILC_BASE)
#define ILC_SPAN                AGILEX_HPS_PERIPH_ILC_SPAN

&{/soc/base_fpga_region} {

        external-fpga-config;

        #address-cells = <0x1>;
        #size-cells = <0x1>;

        ranges;

        SYSID_QSYS: SYSID_QSYS@SYSID_QSYS_UNIT_ADDRESS {
                compatible = "altr,sysid-1.0";
                reg = <SYSID_QSYS_BASE SYSID_QSYS_SPAN>;
        };

        LED_PIO: LED_PIO@LED_PIO_UNIT_ADDRESS {
                compatible = "altr,pio-1.0";
                reg = <LED_PIO_BASE LED_PIO_SPAN>;
                altr,ngpio = <LED_PIO_WIDTH>;
                #gpio-cells = <2>;
                gpio-controller;
        };

        BUTTON_PIO: BUTTON_PIO@BUTTON_PIO_UNIT_ADDRESS {
                compatible = "altr,pio-1.0";
                reg = <BUTTON_PIO_BASE BUTTON_PIO_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <0 BUTTON_PIO_IRQ IRQ_TYPE_LEVEL_HIGH>;
                altr,ngpio = <BUTTON_PIO_WIDTH>;
                #gpio-cells = <2>;
                gpio-controller;
                #interrupt-cells = <1>;
                interrupt-controller;
                altr,interrupt-type = <IRQ_TYPE_EDGE_FALLING>;
        };

        DIPSW_PIO: DIPSW_PIO@DIPSW_PIO_UNIT_ADDRESS {
                compatible = "altr,pio-1.0";
                reg = <DIPSW_PIO_BASE DIPSW_PIO_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <0 DIPSW_PIO_IRQ IRQ_TYPE_LEVEL_HIGH>;
                altr,ngpio = <DIPSW_PIO_WIDTH>;
                #gpio-cells = <2>;
                gpio-controller;
                #interrupt-cells = <1>;
                interrupt-controller;
                altr,interrupt-type = <IRQ_TYPE_EDGE_BOTH>;
        };

        ILC: ILC@ILC_UNIT_ADDRESS {
                compatible = "altr,ilc-1.0";
                reg = <ILC_BASE ILC_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <
                        0       BUTTON_PIO_IRQ  IRQ_TYPE_LEVEL_HIGH
                        0       DIPSW_PIO_IRQ   IRQ_TYPE_LEVEL_HIGH
                        >;
                altr,sw-fifo-depth = <32>;
        };

        leds {
                compatible = "gpio-leds";

                fpga0 {
                        label = "fpga_led0";
                        gpios = <&LED_PIO 0 GPIO_ACTIVE_HIGH>;
                };

                fpga1 {
                        label = "fpga_led1";
                        gpios = <&LED_PIO 1 GPIO_ACTIVE_HIGH>;
                };

                fpga2 {
                        label = "fpga_led2";
                        gpios = <&LED_PIO 2 GPIO_ACTIVE_HIGH>;
                };
        };

}; // base_fpga_region

Build this devicetree overlay file using the kernel build environment just like we built the main system devicetree example above:

[]$ make intel/my_agilex_fpga_overlay.dtb

Note that these warnings are expected and benign for our example:

arch/arm64/boot/dts/intel/my_agilex_fpga_overlay.dts:55.2-9:
Warning (ranges_format): /fragment@0/__overlay__:ranges: empty "ranges" property
but its #address-cells (1) differs from /fragment@0 (2)

arch/arm64/boot/dts/intel/my_agilex_fpga_overlay.dts:48.26-126.3:
Warning (avoid_default_addr_size): /fragment@0/__overlay__: Relying on default
#address-cells value

arch/arm64/boot/dts/intel/my_agilex_fpga_overlay.dts:48.26-126.3:
Warning (avoid_default_addr_size): /fragment@0/__overlay__: Relying on default
#size-cells value

You should be able to locate the DTB that was just built:

[]$ find -name my_agilex_fpga_overlay.dtb
./arch/arm64/boot/dts/intel/my_agilex_fpga_overlay.dtb

To test this devicetree overlay, copy it over to the target that is running the my_agilex_top.dtb example devicetree:

[]$ scp ./arch/arm64/boot/dts/intel/my_agilex_fpga_overlay.dtb root@<TARGET-IP-ADDR>:/home/root/

Place the my_agilex_fpga_overlay.dtb overlay file in the /lib/firmware directory, you may need to create that directory:

root@agilex:~# mkdir -p /lib/firmware
root@agilex:~# mv my_agilex_fpga_overlay.dtb /lib/firmware/

With the devicetree overlay DTB in the firmware directory, you can now apply it to the system devicetree. Do that by creating a directory under /sys/kernel/config/device-tree/overlays/, and then write the name of the devicetree overlay to the resultant path file that gets created in that new directory. There is a status file that gets created in that directory as well that informs you of the overlay status:

root@agilex:~# mkdir /sys/kernel/config/device-tree/overlays/my-agilex-fpga-overlay
root@agilex:~# ls /sys/kernel/config/device-tree/overlays/my-agilex-fpga-overlay
dtbo    path    status
root@agilex:~# cat /sys/kernel/config/device-tree/overlays/my-agilex-fpga-overlay/status
unapplied
root@agilex:~# echo my_agilex_fpga_overlay.dtb \
> /sys/kernel/config/device-tree/overlays/my-agilex-fpga-overlay/path
root@agilex:~# cat /sys/kernel/config/device-tree/overlays/my-agilex-fpga-overlay/status
applied

Once this overlay is applied to the main system devicetree, you should see the FPGA LEDs on the Agilex SoC DK turn on if they were in the uninitialized state after the FPGA was programmed at boot. In this version of the GSRD, the scroll_server application is running which means that the FPGA LEDs should scroll back and forth now that the FPGA peripherals are enabled. Even with the scroll server running, you should be able to toggle an FPGA LED at this point like this:

root@agilex:~# echo 1 > /sys/class/leds/fpga_led0/brightness
root@agilex:~# echo 0 > /sys/class/leds/fpga_led0/brightness

You should also be able to read the sysid values like this:

root@agilex:~# cat /sys/devices/platform/soc/soc:base_fpga_region/\
f9000000.SYSID_QSYS/sysid/id
2899692286
root@agilex:~# cat /sys/devices/platform/soc/soc:base_fpga_region/\
f9000000.SYSID_QSYS/sysid/timestamp
1657076120 (2022-7-6 2:55:20 UTC)

If you investigate the sysfs and procfs, as shown in the section below, you should be able to see all the new devicetree entries that were applied with the overlay. If you want to remove the overlay, just remove the directory that you created in /sys/kernel/config/device-tree/overlays/:

root@agilex:~# rmdir /sys/kernel/config/device-tree/overlays/my-agilex-fpga-overlay

Main system devicetree integration method

Create a new file called arch/arm64/boot/dts/intel/my_agilex_fpga.dtsi to store our FPGA peripheral definition as part of the main system devicetree:

[]$ cat << EOF | base64 -d -i | xz -d > arch/arm64/boot/dts/intel/my_agilex_fpga.dtsi
/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4A7LA9RdABeKfUvNK1NbzTI+B3OUEJMYagjnn0bslkyF
7xuB8nrKCzWcgqNMVnytO8kU434lb6O/lVpXis+nN97FmMmUs2NSmBw+BzMMy9zaK6yGnL1+gDhJ
NNH2JEe0oQyJ1mdBqrCCpJSTH6LbkKHAFpUymBW1Os+F4faXab6d0xnXOgF8ZPnjLIEq+qOxkAvj
cBrqm4ZAdhK4sOpQMTEZ/jbwGaul1G+PeA+p2/3LnB31oA4PXedfQe0j5HFeNcLzlxL3M1aDNgH8
qGz70cAjZaAP+F+8X1/KXI6Do7a2BomBSPm8y5CzPmrgk7Bk2ns+SM7CvnwrH8LFbVwsHfwJNGXj
MrqgksHnPb69OtZATpbPZwzjX0zLInuJDLTnyvZtIv20VAZQHA+kvzVK7nSC24+NfJ+OdO3y32NS
B9u9bShHBgjqoCQ0VmleP9Usz9BBz9RSq1KBqSv/7ytF4v/F/Is9nJ1SDYcDZ8cOpMmK74ImwYBX
tGf2HSdJzYFWCM+Er7hLI/WV3+frfv6z/iaIb2rfT6isl+3t6AxMI6HCR2rCPAm/GzHpxO2H1nqI
uiiwJlq3t0ms3byIVbXv8Weq+1q89Sa5AxHvjCG9QkKVW7Quag3g4SOI58R2s8UpmijVPHWN14qW
RushKLaafMpKkIX87Vx7tah3yDRp06MN8cCv1aoM0ZT1MYQcFF+4E8Zxb4mFpSp9dJWJT9c7to/w
y+FzHD1Zqu5deDmHH3KXMRWcUvyy58a5/qNjdUlWfSE/+LxFlR6iRu7SkfMWFKKwyELEnuBYXKts
fOJvamkWgY9CFy2rQOZ92cxEzVBV5pRXWsdI4Wd7ndJtIilOS2mX4igHR6JLC75QhAlw6/TV1H+G
S9EkTZbvM9sbAJWnJ04WPs+wAdeLwo83A2/cAK4JcJLAnH0vOESCgL6UGdaZIIhqyEZEK0cEqfV2
LUgFWFnclCoUXBqz2nlp0MzUNOIdT4ZBSGbt1BF+WlgSzUL0MyHxEY38n4C43gFCNL+6MepPYjqq
kcdeNw2pjsIofyluC0e7QXSI9xwauRf5rfXhFbqQ0G/bMnwasftP7vG0S11giQwwonkZ7pei5VxL
mywDFkMHosGsu2bOAh6XzX870gC4OLffDBiu06aHeitYi4kw6LNqmnFyk5a+HGykgm9DZ2MH+wVB
jMd4OXsE/Q1+NkMG3q6eXJlTd5MZWYtZ1y5GEGtAABRf36aN/EIv+YgM85f5CzPqaR5BzVnFIuzw
Tk+7lc2lv6TtaKPQMK59k9kN/NFByQaeBpegrRqgwbLaAtLHjaLYBjLOAJoJRGuTqsjlAAHwB8wd
AABJcIJMscRn+wIAAAAABFla
EOF

/*
 * SPDX-License-Identifier: MIT-0
 * Copyright (c) 2018-2022 Intel Corporation
 */

#include "qsys_top.h"
#include "dt-bindings/interrupt-controller/irq.h"
#include "dt-bindings/gpio/gpio.h"

/*
 * Redefine the macros from qsys_top.h into shorter macros and more useful
 * values that we can use directly in the device tree definition.  We will also
 * take care of the bridge offsets that we require for the true base address of
 * the H2F and LW_H2F bridge.
 */

#define AGILEX_H2F_BASE         0x80000000
#define AGILEX_LW_H2F_BASE      0xF9000000

#define SYSID_QSYS_UNIT_ADDRESS AGILEX_LW_H2F_BASE,AGILEX_HPS_SYSID_BASE
#define SYSID_QSYS_BASE         (AGILEX_LW_H2F_BASE + AGILEX_HPS_SYSID_BASE)
#define SYSID_QSYS_SPAN         AGILEX_HPS_SYSID_SPAN

#define LED_PIO_UNIT_ADDRESS    AGILEX_LW_H2F_BASE,AGILEX_HPS_PERIPH_LED_PIO_BASE
#define LED_PIO_BASE            (AGILEX_LW_H2F_BASE + AGILEX_HPS_PERIPH_LED_PIO_BASE)
#define LED_PIO_SPAN            AGILEX_HPS_PERIPH_LED_PIO_SPAN
#define LED_PIO_WIDTH           AGILEX_HPS_PERIPH_LED_PIO_DATA_WIDTH

#define BUTTON_PIO_UNIT_ADDRESS AGILEX_LW_H2F_BASE,AGILEX_HPS_PERIPH_BUTTON_PIO_BASE
#define BUTTON_PIO_BASE         (AGILEX_LW_H2F_BASE + AGILEX_HPS_PERIPH_BUTTON_PIO_BASE)
#define BUTTON_PIO_SPAN         AGILEX_HPS_PERIPH_BUTTON_PIO_SPAN
#define BUTTON_PIO_IRQ          (AGILEX_HPS_PERIPH_BUTTON_PIO_IRQ + 49 - 32)
#define BUTTON_PIO_WIDTH        AGILEX_HPS_PERIPH_BUTTON_PIO_DATA_WIDTH

#define DIPSW_PIO_UNIT_ADDRESS  AGILEX_LW_H2F_BASE,AGILEX_HPS_PERIPH_DIPSW_PIO_BASE
#define DIPSW_PIO_BASE          (AGILEX_LW_H2F_BASE + AGILEX_HPS_PERIPH_DIPSW_PIO_BASE)
#define DIPSW_PIO_SPAN          AGILEX_HPS_PERIPH_DIPSW_PIO_SPAN
#define DIPSW_PIO_IRQ           (AGILEX_HPS_PERIPH_DIPSW_PIO_IRQ + 49 - 32)
#define DIPSW_PIO_WIDTH         AGILEX_HPS_PERIPH_DIPSW_PIO_DATA_WIDTH

#define ILC_UNIT_ADDRESS        AGILEX_LW_H2F_BASE,AGILEX_HPS_PERIPH_ILC_BASE
#define ILC_BASE                (AGILEX_LW_H2F_BASE + AGILEX_HPS_PERIPH_ILC_BASE)
#define ILC_SPAN                AGILEX_HPS_PERIPH_ILC_SPAN

/ {
soc {

base_fpga_region {

        external-fpga-config;

        #address-cells = <0x2>;
        #size-cells = <0x1>;

        ranges = <
                AGILEX_LW_H2F_BASE      SYSID_QSYS_BASE SYSID_QSYS_BASE SYSID_QSYS_SPAN
                AGILEX_LW_H2F_BASE      LED_PIO_BASE    LED_PIO_BASE    LED_PIO_SPAN
                AGILEX_LW_H2F_BASE      BUTTON_PIO_BASE BUTTON_PIO_BASE BUTTON_PIO_SPAN
                AGILEX_LW_H2F_BASE      DIPSW_PIO_BASE  DIPSW_PIO_BASE  DIPSW_PIO_SPAN
                AGILEX_LW_H2F_BASE      ILC_BASE        ILC_BASE        ILC_SPAN
                >;

        SYSID_QSYS: SYSID_QSYS@SYSID_QSYS_UNIT_ADDRESS {
                compatible = "altr,sysid-1.0";
                reg = <AGILEX_LW_H2F_BASE       SYSID_QSYS_BASE SYSID_QSYS_SPAN>;
        };

        LED_PIO: LED_PIO@LED_PIO_UNIT_ADDRESS {
                compatible = "altr,pio-1.0";
                reg = <AGILEX_LW_H2F_BASE       LED_PIO_BASE LED_PIO_SPAN>;
                altr,ngpio = <LED_PIO_WIDTH>;
                #gpio-cells = <2>;
                gpio-controller;
        };

        BUTTON_PIO: BUTTON_PIO@BUTTON_PIO_UNIT_ADDRESS {
                compatible = "altr,pio-1.0";
                reg = <AGILEX_LW_H2F_BASE       BUTTON_PIO_BASE BUTTON_PIO_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <0 BUTTON_PIO_IRQ IRQ_TYPE_LEVEL_HIGH>;
                altr,ngpio = <BUTTON_PIO_WIDTH>;
                #gpio-cells = <2>;
                gpio-controller;
                #interrupt-cells = <1>;
                interrupt-controller;
                altr,interrupt-type = <IRQ_TYPE_EDGE_FALLING>;
        };

        DIPSW_PIO: DIPSW_PIO@DIPSW_PIO_UNIT_ADDRESS {
                compatible = "altr,pio-1.0";
                reg = <AGILEX_LW_H2F_BASE       DIPSW_PIO_BASE DIPSW_PIO_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <0 DIPSW_PIO_IRQ IRQ_TYPE_LEVEL_HIGH>;
                altr,ngpio = <DIPSW_PIO_WIDTH>;
                #gpio-cells = <2>;
                gpio-controller;
                #interrupt-cells = <1>;
                interrupt-controller;
                altr,interrupt-type = <IRQ_TYPE_EDGE_BOTH>;
        };

        ILC: ILC@ILC_UNIT_ADDRESS {
                compatible = "altr,ilc-1.0";
                reg = <AGILEX_LW_H2F_BASE       ILC_BASE ILC_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <
                        0       BUTTON_PIO_IRQ  IRQ_TYPE_LEVEL_HIGH
                        0       DIPSW_PIO_IRQ   IRQ_TYPE_LEVEL_HIGH
                        >;
                altr,sw-fifo-depth = <32>;
        };
};

}; // soc
}; // /

Edit the arch/arm64/boot/dts/intel/my_agilex_top.dts file to include our new FPGA devicetree definition:

[]$ cat << EOF | base64 -d -i | xz -d | bash
/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4AG4ANhdADmZSuojhxSi3Mv8MfjO12CEtsMK+7liVcns
z5YLtM3IyUrqx0d48jihm/tdVePAIZEqRPqwJGxL6orv3fxjc7uCCkFs6jbBDLfxzAW3Z9gWEYRi
r/QWIrGvNPbhcrHeIpuoJ8E9FDG86nDdVg28MA4y/nLQ/sYauZz2Mfr2/InKV01eqZDSDvSroSUt
XhgWaXkAw02DinxF/96hwnlcx1kG2OzlG6TQhLVgyRmO7a2p0/qYCTp/5aMkcxLzi6MwMzG4GIoV
j+BXcTNJz6EzCywF8r+sNpLidQCMpVxT8rZAgwAB9AG5AwAAKxkKErHEZ/sCAAAAAARZWg==
EOF

set -x
sed \
-i.bak \
-e '/#include "socfpga_agilex.dtsi"/a #include "my_agilex_fpga.dtsi"' \
-e '/compatible = "gpio-leds";/a \
\
                fpga0 {\
                        label = "fpga_led0";\
                        gpios = <&LED_PIO 0 GPIO_ACTIVE_HIGH>;\
                };\
\
                fpga1 {\
                        label = "fpga_led1";\
                        gpios = <&LED_PIO 1 GPIO_ACTIVE_HIGH>;\
                };\
\
                fpga2 {\
                        label = "fpga_led2";\
                        gpios = <&LED_PIO 2 GPIO_ACTIVE_HIGH>;\
                };\
' \
./arch/arm64/boot/dts/intel/my_agilex_top.dts

Build the devicetree:

[]$ make intel/my_agilex_top.dtb

You should be able to locate the DTB that was just built:

[]$ find -name my_agilex_top.dtb
./arch/arm64/boot/dts/intel/my_agilex_top.dtb

Copy the new system DTB over to the target:

[]$ scp ./arch/arm64/boot/dts/intel/my_agilex_top.dtb root@<TARGET-IP-ADDR>:/home/root/

Move the DTB onto the FAT partition:

root@agilex:~# mount /dev/mmcblk0p1 /mnt
root@agilex:~# mv my_agilex_top.dtb /mnt/

Now reboot the target, remember to stop u-boot at the autoboot console:

root@agilex:~# reboot

...cut linux shutdown and u-boot startup messages...

Hit any key to stop autoboot:  0
SOCFPGA_AGILEX #

Just like above, once we reach the u-boot console we must manually boot the Linux kernel with our new devicetree since the default environment attempts to bootm from a configuration in the kernel FIT image on the FAT partition and we do not want that version of the devicetree to be used. We do want to reuse the FPGA core RBF configuration file and the Linux kernel from the FIT image.

Start by loading the kernel FIT image from mmc flash into memory:

SOCFPGA_AGILEX # load mmc 0:1 ${loadaddr} kernel.itb
15111707 bytes read in 687 ms (21 MiB/s)

Program the fpga-4 RBF image from the FIT image into the FPGA core logic and then enable the FPGA bridges:

SOCFPGA_AGILEX # imxtract ${loadaddr} fpga-4 ${kernel_addr_r}
## Copying 'fpga-4' subimage from FIT image at 01000000 ...
crc32+    Loading part 15 ... OK

SOCFPGA_AGILEX # fpga load 0 ${kernel_addr_r} ${filesize}
...FPGA reconfiguration OK!

SOCFPGA_AGILEX # bridge enable

Decompress the kernel image from the FIT image:

SOCFPGA_AGILEX # fdt addr ${loadaddr}
SOCFPGA_AGILEX # fdt get addr KERNEL_ADDR /images/kernel data
SOCFPGA_AGILEX # lzmadec ${KERNEL_ADDR} ${kernel_addr_r}
Uncompressed size: 34345472 = 0X20C1200

Load our custom devicetree image from mmc flash into memory:

SOCFPGA_AGILEX # load mmc 0:1 ${fdt_addr_r} my_agilex_top.dtb
16851 bytes read in 4 ms (4 MiB/s)

Boot the Linux kernel:

SOCFPGA_AGILEX # setenv bootargs "earlycon panic=-1 root=${mmcroot} rw rootwait"
SOCFPGA_AGILEX # booti ${kernel_addr_r} - ${fdt_addr_r}

When the kernel boots you should again see the FPGA LEDs turn on as the drivers are loaded based on the devicetree configuration and the scroll server should scroll them back and forth. If you investigate the sysfs and procfs, as shown in the section below, you should be able to see all the new devicetree entries that were added to the main system devicetree.

Observing devicetree evidence in the sysfs and procfs

If you would like to observe the devicetree evidence in the sysfs, you can search for the names that we used in our FPGA section and see where various entries are maintained in the sysfs that reference them:

root@agilex:~# find /sys \
-name "*SYSID*" -o \
-name "*PIO*" -o \
-name "*ILC*" | \
sort
/sys/bus/platform/devices/f9000000.SYSID_QSYS
/sys/bus/platform/devices/f9001060.BUTTON_PIO
/sys/bus/platform/devices/f9001070.DIPSW_PIO
/sys/bus/platform/devices/f9001080.LED_PIO
/sys/bus/platform/devices/f9001100.ILC
/sys/bus/platform/drivers/altera_gpio/f9001060.BUTTON_PIO
/sys/bus/platform/drivers/altera_gpio/f9001070.DIPSW_PIO
/sys/bus/platform/drivers/altera_gpio/f9001080.LED_PIO
/sys/bus/platform/drivers/altera_ilc/f9001100.ILC
/sys/bus/platform/drivers/altera_sysid/f9000000.SYSID_QSYS
/sys/devices/platform/soc/soc:base_fpga_region/fpga_region/region0/f9000000.SYSID_QSYS
/sys/devices/platform/soc/soc:base_fpga_region/fpga_region/region0/f9001060.BUTTON_PIO
/sys/devices/platform/soc/soc:base_fpga_region/fpga_region/region0/f9001070.DIPSW_PIO
/sys/devices/platform/soc/soc:base_fpga_region/fpga_region/region0/f9001080.LED_PIO
/sys/devices/platform/soc/soc:base_fpga_region/fpga_region/region0/f9001100.ILC
/sys/firmware/devicetree/base/__symbols__/BUTTON_PIO
/sys/firmware/devicetree/base/__symbols__/DIPSW_PIO
/sys/firmware/devicetree/base/__symbols__/ILC
/sys/firmware/devicetree/base/__symbols__/LED_PIO
/sys/firmware/devicetree/base/__symbols__/SYSID_QSYS
/sys/firmware/devicetree/base/soc/base_fpga_region/BUTTON_PIO@0xF9000000,0x1060
/sys/firmware/devicetree/base/soc/base_fpga_region/DIPSW_PIO@0xF9000000,0x1070
/sys/firmware/devicetree/base/soc/base_fpga_region/ILC@0xF9000000,0x1100
/sys/firmware/devicetree/base/soc/base_fpga_region/LED_PIO@0xF9000000,0x1080
/sys/firmware/devicetree/base/soc/base_fpga_region/SYSID_QSYS@0xF9000000,0x0

You can find similar evidence in the procfs as well:

root@agilex:~# find /proc/device-tree/soc/base_fpga_region/ -maxdepth 1 | sort
/proc/device-tree/soc/base_fpga_region/
/proc/device-tree/soc/base_fpga_region/#address-cells
/proc/device-tree/soc/base_fpga_region/#size-cells
/proc/device-tree/soc/base_fpga_region/BUTTON_PIO@0xF9000000,0x1060
/proc/device-tree/soc/base_fpga_region/DIPSW_PIO@0xF9000000,0x1070
/proc/device-tree/soc/base_fpga_region/ILC@0xF9000000,0x1100
/proc/device-tree/soc/base_fpga_region/LED_PIO@0xF9000000,0x1080
/proc/device-tree/soc/base_fpga_region/SYSID_QSYS@0xF9000000,0x0
/proc/device-tree/soc/base_fpga_region/compatible
/proc/device-tree/soc/base_fpga_region/external-fpga-config
/proc/device-tree/soc/base_fpga_region/fpga-mgr
/proc/device-tree/soc/base_fpga_region/name
/proc/device-tree/soc/base_fpga_region/ranges

Notes about peripherals in this Agilex SoC GSRD example

The SYSID_QSYS, LED_PIO, DIPSW_PIO, and BUTTON_PIO peripherals should all work as expected.

In this version of the GSRD, the ILC peripheral will report errors as the driver loads. This is because the ILC driver does not support the new GPIO IRQ framework that is implemented in the modern kernel. The current ILC driver expects to be able to share IRQs with the peripherals that it monitors, and the modern GPIO framework no longer supports that:

root@agilex:~# dmesg | grep ilc
[    4.768577] altera_ilc f9001100.ILC: Failed to register interrupt handler
[    4.775559] altera_ilc f9001100.ILC: Failed to register interrupt handler
[    4.782469] altera_ilc f9001100.ILC: Driver successfully loaded