HOWTO Create a Devicetree for Stratix 10 SoC

Required environment

This write-up relies on a Linux development host environment with a bash shell.

This write-up references these built-in bash shell functions:

cd echo export read type

This write-up references these standard Linux utilities:

base64 cat column cp dd find git grep ls make mkdir scp sed sort sudo sync tar wget xz

This write-up references these programs provided by the Intel FPGA software tools installation:

quartus_pgm sopc-create-header-files

You can check your current environment by running this loop in your bash shell:

[]$ for NEXT in \
cd echo export read type \
base64 cat column cp dd find git grep ls make mkdir scp sed sort sudo sync tar wget xz \
quartus_pgm sopc-create-header-files
do
type -t ${NEXT:?} > /dev/null &&
{ echo "    located - ${NEXT:?}" ; } ||
{ echo "NOT located - ${NEXT:?}" ; }
done

If your development environment is missing any of the references from above, then you may need to install the packages or tools that provide those dependencies before you can execute some of the examples described below.

Overview

At the time of this writing the current Intel FPGA tools era is 22.2. The references below will be specific to that Intel FPGA software release environment.

This write-up will demonstrate how to create a devicetree for your own custom FPGA hardware implementation in an SoC FPGA device. As a reference and demonstration platform we will refer to materials that are prepared for use on the Stratix 10 SoC development kit. The Golden Hardware Reference Design (GHRD) is a hardware example that is used to create a pre-built Linux SD card example called the Golden System Reference Design (GSRD). The GHRD and GSRD images are distributed on releases.rocketboards.org. We will use the GSRD reference design as the foundation of this write-up and demonstrate how we would create our own devicetree implementation for the GHRD FPGA hardware design as if we were attempting to do this for our own custom hardware platform.

In this write-up we provide many command line examples that are intended to be copied and pasted right on the command line of your Linux development host if you choose to evaluate them. We have made every attempt to create generic command examples that do not incorporate specific environmental requirements that your environment may not provide, but in some cases, you may find that you must edit the example commands to fit your environment more appropriately.

In this write-up we also provide some source file examples and patch examples that you will need to create and apply if you choose to evaluate this content directly. Since the source files and patch files require complex and specific white space that is challenging to represent accurately in various markdown languages that this write-up is distributed in, we have chosen to distribute these source and patch examples as xz compressed base64 encoded text along with command lines that you can use to recreate the precise output that is required for the source and patch files. Here is an example of how this works, you should be able to copy and paste the initial command line below along with all the base64 encoded data through the EOF line onto your Linux command line on your development host and have it produce the results that we show below:

[]$ cat << EOF | base64 -d -i | xz -d
/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4AJGAPRdAASdiQQG8RGxyToYaWl0rK7DBoB3YK3stuMt
gHWBpFTOaL6Q3l77rOjasGselnIn4jTcx7dH78l1cEuNQqasL+6EoEGJ3YTfPY8kk1Hn3SeCf5sN
9YrpTYB8eSZT36lBlNCuGBLE57vg+xwZvy44MmvtDbi6ZapUTEXN8JfQqXAsRzKyuxm91bd8Rmhj
HTTkbeNneZBqRLsR2vcF/XiP+QwFnSQKx2K5GMSQ8PI09lGfjvRoKaBIO6KAaEiFMjIxi6+jIfXM
lymiiHAW5VGrl1+yuBaFBCOZxeHSGll+OZYB2Vm3QzyscokCoknlAgthhLI8fuMARjY/RHmJ0LMA
AZACxwQAAJ3vYEixxGf7AgAAAAAEWVo=
EOF

When we place this decoded content into the write-up, we convert all the tab characters to spaces to maintain the desired representation of the original content but make it easier for the markdown language environments to represent the vertical alignment accurately. These white space substitutions will change the meaning of some of the patch contents, so you cannot simply cut and paste this text representation for its intended use:

        ranges = <
                S10_LW_H2F_BASE SYSID_QSYS_BASE SYSID_QSYS_BASE SYSID_QSYS_SPAN
                S10_LW_H2F_BASE LED_PIO_BASE    LED_PIO_BASE    LED_PIO_SPAN
                S10_LW_H2F_BASE BUTTON_PIO_BASE BUTTON_PIO_BASE BUTTON_PIO_SPAN
                S10_LW_H2F_BASE DIPSW_PIO_BASE  DIPSW_PIO_BASE  DIPSW_PIO_SPAN
                S10_LW_H2F_BASE ILC_BASE        ILC_BASE        ILC_SPAN
        >;

        ILC: ILC@ILC_UNIT_ADDRESS {
                compatible = "altr,ilc-1.0";
                reg = <S10_LW_H2F_BASE ILC_BASE ILC_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <
                        0       BUTTON_PIO_IRQ  IRQ_TYPE_LEVEL_HIGH
                        0       DIPSW_PIO_IRQ   IRQ_TYPE_LEVEL_HIGH
                >;
                altr,sw-fifo-depth = <32>;
        };

Throughout the rest of this write-up, we will provide the xz compressed base64 encoded content followed immediately by the decoded representation with the white space modifications as we just demonstrated above.

Initial Stratix 10 SoC DK board setup, tools setup, and other setup

Start by getting the Stratix 10 SoC DK board set up and configured to run the pre-built SD card image and get the tools and development environment properly configured.

Create a working directory and change into it:

[]$ mkdir s10_soc_example
[]$ cd s10_soc_example

Download the JIC image, SD card image and the GHRD project image for the Stratix 10 SoC DK. At the time of this writing the latest version is the 2022.07 release:

[]$ wget https://releases.rocketboards.org/2022.07/gsrd/s10_gsrd/\
ghrd_1sx280lu2f50e2vg.jic.tar.gz

[]$ wget https://releases.rocketboards.org/2022.07/gsrd/s10_gsrd/sdimage.tar.gz

[]$ wget https://releases.rocketboards.org/2022.07/gsrd/s10_gsrd/\
s10_soc_devkit_ghrd_QPDS-22.2pro-21.1std.tar.gz

Extract the JIC image:

[]$ tar xf ghrd_1sx280lu2f50e2vg.jic.tar.gz

Extract the SD card image:

[]$ tar xf sdimage.tar.gz

Extract the GHRD project image into a new directory:

[]$ mkdir s10_soc_devkit_ghrd
[]$ tar -C s10_soc_devkit_ghrd -xf s10_soc_devkit_ghrd_QPDS-22.2pro-21.1std.tar.gz

Program the JIC image into the SDM QSPI flash device, set the MSEL, SW2, dip switch block to 1-ON, 2-ON, 3-ON, 4-OFF power the board up and run the following command:

[]$ quartus_pgm -m jtag -c 1 -o pi\;ghrd_1sx280lu2f50e2vg.jic

Once the JIC image is programmed into the SDM QSPI flash device, power off the board and return the MSEL, SW2, dip switch block to 1-ON, 2-OFF, 3-OFF, 4-OFF.

Program the SD image onto a micro-SD card:

[]$ sudo dd if=gsrd-console-image-stratix10.wic of=<path-to-sd-card> bs=16M
[]$ sync

Plug the micro-SD card into the Stratix 10 SoC DK and boot the board to ensure that you have a working platform. Once you have booted into the Linux environment, verify the Linux kernel version that is being used so that we can create an appropriate devicetree of our own for that Linux kernel environment. Devicetrees are specific to each kernel release, the node, and binding requirements for the devicetree can and often do change from kernel release to kernel release.

Log into the target and run the 'uname' command to determine the kernel version:

stratix10 login: root
root@stratix10:~# uname -a
Linux stratix10 5.15.30-altera #1 SMP PREEMPT Fri Jun 24 03:52:26 UTC 2022 aarch64 GNU/Linux

In this case you can see that this version of the pre-built SD card contains a 5.15.30 kernel version, and it was built in June 2022. We will use that information to select the Linux kernel commit that we check out below.

Clone the linux-socfpga git repository and change into the directory:

[]$ git clone https://github.com/altera-opensource/linux-socfpga.git
[]$ cd linux-socfpga

We need to select a commit to check out of the repo that is on the branch that the kernel was built with above. We don't need the exact commit that was used to build the kernel we are running but we want something close to that kernel. The repo is consistently tagged with branch release tags in the form "rel_<branch-name>_<YY>.<MM>.<release>", so if we grep the git tags from the repo looking for the version that we see in the output above we can see the tags associated with that branch of the kernel from the repo:

[]$ git tag | grep 5.15.30
rel_socfpga-5.15.30-lts_22.08.02_pr
rel_socfpga-5.15.30-lts_22.09.01_pr

We can see that the "socfpga-5.15.30-lts" kernel tags are present, and we see that the earliest release was tagged in August of 2022. We will use the 22.08.02_pr tag as our target. We check out that commit on a new branch with the same name like this:

[]$ git checkout \
-b rel_socfpga-5.15.30-lts_22.08.02_pr \
rel_socfpga-5.15.30-lts_22.08.02_pr

If you do not have a cross compiler to compile the Linux kernel for Armv8 cores, download one now:

[]$ wget https://developer.arm.com/-/media/Files/downloads/gnu-a/10.3-2021.07/\
binrel/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz

[]$ tar xf gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu.tar.xz

Setup the environment so the Linux kernel build environment understands what we are building for:

[]$ export ARCH=arm64
[]$ export CROSS_COMPILE=\
<path-to>/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-

You may want to verify that the compiler can be located in your environment, you can test that with the type command:

[]$ type -P ${CROSS_COMPILE:?}gcc
<path-to>/gcc-arm-10.3-2021.07-x86_64-aarch64-none-linux-gnu/bin/aarch64-none-linux-gnu-gcc

Stratix 10 SoC devicetree orientation

Our strategy for building our own devicetree will be to leverage the base definitions for the Hard Processor System (HPS) that are up-streamed into the Linux source tree already. In fact, there is an example devicetree for the Stratix 10 SoC DK that is already available in the source tree. If we look for the top-level devicetree file for the Stratix 10 SoC DK we should find it here:

[]$ find -name socfpga_stratix10_socdk.dts
./arch/arm64/boot/dts/altera/socfpga_stratix10_socdk.dts

Or in your web browser you can look here.

This socfpga_stratix10_socdk.dts file is the top-level devicetree file for the Stratix 10 SoC DK board and we will want to create something that looks just like this for our own example, but we will need to add the FPGA specific devicetree requirements to our example. We may want to use this file as a starting reference point, but ultimately, we will need to create our own top-level devicetree file. This default devicetree example for the Stratix 10 SoC DK is only concerned about the HPS hardened peripherals, which do not ever change, so this devicetree can be defined in such a way that it can apply to any other Stratix 10 SoC design, and that is what this default example demonstrates. This top-level socfpga_stratix10_socdk.dts file simply includes other more fundamental devicetree include files which define more of the lower-level details of the HPS core itself. In this top-level file we see only the top-level enabling of the resources defined in the lower level include files as well as any board level details that interact with the HPS peripherals that are enabled.

The socfpga_stratix10_socdk.dts file includes the lowest level definition of the HPS for the Stratix 10 SoC device in a file called socfpga_stratix10.dtsi here:

[]$ find -name socfpga_stratix10.dtsi
./arch/arm64/boot/dts/altera/socfpga_stratix10.dtsi

Or in your web browser you can look here.

This socfpga_stratix10.dtsi file contains the definition of all the low-level hardware built into the Stratix 10 SoC HPS core. It carries the required clocking structure for the Linux clock framework as well as the devicetree bindings for all the HPS IO peripherals. If you look at this devicetree include file closely you will see that the peripherals that are internal to the HPS core are all enabled by default, i.e., they have no status binding which means they default to enabled. But all the IO peripherals that interact through device pins are marked as disabled, i.e., they all have a status binding that is set to disabled in this devicetree include file. If you look at the top-level file that includes this file, socfpga_stratix10_socdk.dts you can see that one of the things it does is redefine the status binding for all the HPS IO peripherals that it wants to enable.

At the socfpga_stratix10.dtsi level of the devicetree definition, they primarily try to define the bindings for all the peripherals that are not board specific, so you will see compatible, reg, interrupts, clocks and any other well-defined bindings that are unlikely to change for any implementation of the Stratix 10 SoC HPS core. Then at the top-level devicetree file, for example in socfpga_stratix10_socdk.dts you would need to enable the peripherals that you require and add any bindings to peripherals that are board specific parameters, and you can even override settings for bindings that are defined in the socfpga_stratix10.dtsi level of the devicetree if needed.

This version of the socfpga_stratix10_socdk.dts file includes a devicetree include file called socfpga_stratix10_qse.dtsi which implements several multi-rate Ethernet drivers for FPGA based IP. We will ignore this functionality in our example and simply comment out that line of code from the top-level devicetree. We only want to discuss the basic GHRD components in this example.

You should familiarize yourself with the layout and contents of the Stratix 10 SoC DK example devicetree top-level file and all the include files that it leverages. Our own devicetree example will begin by leveraging all that existing material.

Build and test the default Stratix 10 SoC DK example devicetree

Build the default example for the Stratix 10 SoC DK board that is in the kernel source tree. Make a copy of the original to modify and preserve the original example file.

Start by configuring the Linux kernel with the defconfig:

[]$ make defconfig

Make a copy of the socfpga_stratix10_socdk.dts file for our own use:

[]$ cp \
./arch/arm64/boot/dts/altera/socfpga_stratix10_socdk.dts \
./arch/arm64/boot/dts/altera/my_s10_top.dts

Edit this top-level devicetree source file to create a unique value for the model binding so that we can identify this devicetree when we run it on the target:

[]$ sed -i "s/SoCFPGA Stratix 10 SoCDK/DT TEST S10 SX DK/" \
./arch/arm64/boot/dts/altera/my_s10_top.dts

And remove the include file reference to socfpga_stratix10_qse.dtsi:

[]$ sed -i "/socfpga_stratix10_qse.dtsi/d" \
./arch/arm64/boot/dts/altera/my_s10_top.dts

To build the devicetree, simply call make with the name of the DTB file that you want to compile:

[]$ make altera/my_s10_top.dtb

This works because the default Linux Makefile infrastructure has recipes for building devicetree sources and producing binary devicetree blobs. The Linux Makefile architecture passes all the devicetree sources through the GNU C preprocessor to preprocess macros and other syntax, and then it passes the preprocessed results through the DTC compiler to produce the resultant binary devicetree blob. You could do this on your own without leveraging the Linux kernel build environment if you wanted to, you would do something like this:

[]$ ${CROSS_COMPILE}gcc \
        -E \
        -nostdinc \
        -I./arch/arm64/boot/dts/altera \
        -I./include \
        -undef -D__DTS__ \
        -x assembler-with-cpp \
        -o arch/arm64/boot/dts/altera/my_s10_top.dts.tmp \
        arch/arm64/boot/dts/altera/my_s10_top.dts

[]$ ./scripts/dtc/dtc \
        -O dtb \
        -o arch/arm64/boot/dts/altera/my_s10_top.dtb \
        -i arch/arm64/boot/dts/altera \
        -@ \
        arch/arm64/boot/dts/altera/my_s10_top.dts.tmp

You should be able to locate the DTB that was just built:

[]$ find -name my_s10_top.dtb
./arch/arm64/boot/dts/altera/my_s10_top.dtb

Move this DTB file onto the Stratix 10 SoC DK target. There are several ways to do this, we will demonstrate moving the file over Ethernet using scp to our target board and then positioning the new DTB where it needs to go.

Start by copying the file to the target:

[]$ scp ./arch/arm64/boot/dts/altera/my_s10_top.dtb root@<TARGET-IP-ADDR>:/home/root/

We need to get the new DTB onto the FAT partition of the SD card, so we mount the FAT partition:

root@stratix10:~# mount /dev/mmcblk0p1 /mnt

We copy our new DTB into the proper location:

root@stratix10:~# mv my_s10_top.dtb /mnt

Before we try out our new DTB, look at the model string that was defined in the original DTB that is running on the target:

root@stratix10:~# (echo ""; cat /proc/device-tree/model; echo "" ;)

SoCFPGA Stratix 10 SoCDK

We can reboot the target to have Linux load our new example DTB, remember to stop u-boot at the autoboot console:

root@stratix10:~# reboot

...cut linux shutdown and u-boot startup messages...

Hit any key to stop autoboot:  0
SOCFPGA_STRATIX10 #

Once we reach the u-boot console we must manually boot the Linux kernel with our new devicetree since the default environment attempts to bootm from a configuration in the kernel FIT image on the FAT partition and we do not want that version of the devicetree to be used. We do want to reuse the FPGA core RBF configuration file and the Linux kernel from the FIT image.

Start by loading the kernel FIT image from mmc flash into memory:

SOCFPGA_STRATIX10 # load mmc 0:1 ${loadaddr} kernel.itb
16430839 bytes read in 754 ms (20.8 MiB/s)

Program the fpga-4 RBF image from the FIT image into the FPGA core logic and then enable the FPGA bridges:

SOCFPGA_STRATIX10 # imxtract ${loadaddr} fpga-4
## Copying 'fpga-4' subimage from FIT image at 01000000 ...
crc32+

SOCFPGA_STRATIX10 # fpga load 0 ${fileaddr} ${filesize}
....FPGA reconfiguration OK!

SOCFPGA_STRATIX10 # bridge enable

Decompress the kernel image from the FIT image:

SOCFPGA_STRATIX10 # fdt addr ${loadaddr}
SOCFPGA_STRATIX10 # fdt get addr KERNEL_ADDR /images/kernel data
SOCFPGA_STRATIX10 # lzmadec ${KERNEL_ADDR} ${kernel_addr_r}
Uncompressed size: 34345472 = 0X20C1200

Load our custom devicetree image from mmc flash into memory:

SOCFPGA_STRATIX10 # load mmc 0:1 ${fdt_addr_r} my_s10_top.dtb
17358 bytes read in 5 ms (3.3 MiB/s)

Boot the Linux kernel:

SOCFPGA_STRATIX10 # setenv bootargs "earlycon panic=-1 root=${mmcroot} rw rootwait"
SOCFPGA_STRATIX10 # booti ${kernel_addr_r} - ${fdt_addr_r}

The target should boot back into the Linux environment just as it did before, however none of the FPGA peripherals are defined in this default example devicetree. We will demonstrate how to add the FPGA peripherals into the devicetree next.

Once the target boots up to the Linux environment again, check the model string of the devicetree and make sure that it has updated to the value stored in the example that we just created and loaded onto the target:

root@stratix10:~# (echo ""; cat /proc/device-tree/model; echo "" ;)

DT TEST S10 SX DK

What should the devicetree enable in the HPS?

We should be running the Stratix 10 SoC DK example devicetree from the kernel source tree that we copied into the my_s10_top.dts file and compiled into a DTB to load onto the target and run. In this devicetree that we copied from socfpga_stratix10_socdk.dts, we can see that the following HPS IO peripherals have been enabled: gpio1, gmac0, mmc, uart0, usb0, i2c1, and qspi. The include file socfpga_stratix10.dtsi enables usbphy0.

There are a few ways to determine what HPS peripherals have been enabled in our hardware project and therefore should be enabled in the devicetree. One way to determine this is to open the HPS peripheral configuration in the Platform Designer system that the component is instantiated in, then you can manually look at the parameterization that selected the peripherals in your design. Another way to determine this is to look for the interface definitions that get output in the SOPCINFO file generated by Platform Designer. Here is an example sed parsing pattern to show us what HPS interfaces were enabled in the Stratix 10 SoC GHRD design which is what we are running in the SD card image that we programmed into the board:

[]$ cat ../s10_soc_devkit_ghrd/qsys_top/qsys_top.sopcinfo | \
sed -E -e "/<module/,/<\/module>/P" -e "D" | \
sed -E -e "/kind=\"altera_stratix10_hps\"/,/<\/module>/P" -e "D" | \
sed -E -e "/<boundary>/,/<\/boundary>/P" -e "D" | \
sed -E -e "/<ports>/,/<\/ports>/D" | \
sed -E -e "/<name>.*<\/name>/P" -e "D" | \
sed -E -e "s/^.*<name>(.*)<\/name>.*$/\1/" | \
sort | column
emac1                   emac_ptp_ref_clock      f2sdram2_data
emac1_gtx_clk           f2h_axi_clock           f2sdram2_reset
emac1_md_clk            f2h_axi_reset           h2f_axi_clock
emac1_rx_clk_in         f2h_axi_slave           h2f_axi_master
emac1_rx_reset          f2h_irq0                h2f_axi_reset
emac1_tx_clk_in         f2h_irq1                h2f_lw_axi_clock
emac1_tx_reset          f2h_stm_hw_events       h2f_lw_axi_master
emac2                   f2sdram0_clock          h2f_lw_axi_reset
emac2_gtx_clk           f2sdram0_data           h2f_reset
emac2_md_clk            f2sdram0_reset          h2f_watchdog_rst
emac2_rx_clk_in         f2sdram1_clock          hps_emif
emac2_rx_reset          f2sdram1_data           hps_io
emac2_tx_clk_in         f2sdram1_reset
emac2_tx_reset          f2sdram2_clock

From the list above we can see that emac1 and emac2 get routed into the FPGA which means that we may eventually need to enable them in the devicetree, but we should not enable them until the FPGA logic that interfaces with those peripherals is properly programmed into the FPGA core logic. The interfaces represented by the f2h_*, f2sdram*, and h2f_* are bridge interfaces between the HPS and FPGA which do not really require specific devicetree enablement. The hps_emif interface represents the DRAM interface which also does not require devicetree enablement. The hps_io interface represents the HPS IO peripherals that get pin muxed off the device through the HPS IO bank and if we look at what ports it has enabled, we can understand other HPS IO peripherals that may require devicetree enablement:

[]$ cat ../s10_soc_devkit_ghrd/qsys_top/qsys_top.sopcinfo | \
sed -E -e "/<module/,/<\/module>/P" -e "D" | \
sed -E -e "/kind=\"altera_stratix10_hps\"/,/<\/module>/P" -e "D" | \
sed -E -e "/<boundary>/,/<\/boundary>/P" -e "D" | \
sed -E -e "/<interface>/,/<\/interface>/P" -e "D" | \
sed -E -e "/<name>hps_io<\/name>/,/<\/interface>/P" -e "D" | \
sed -E -e "/<ports>/,/<\/ports>/P" -e "D" | \
sed -E -e "/<role>.*<\/role>/P" -e "D" | \
sed -E -e "s/^.*<role>(.*)<\/role>.*$/\1/" | \
sort | column
hps_io_gpio_gpio1_io0           hps_io_phery_emac0_TXD2
hps_io_gpio_gpio1_io1           hps_io_phery_emac0_TXD3
hps_io_gpio_gpio1_io19          hps_io_phery_i2c1_SCL
hps_io_gpio_gpio1_io20          hps_io_phery_i2c1_SDA
hps_io_gpio_gpio1_io21          hps_io_phery_sdmmc_CCLK
hps_io_gpio_gpio1_io4           hps_io_phery_sdmmc_CMD
hps_io_gpio_gpio1_io5           hps_io_phery_sdmmc_D0
hps_io_hps_ocs_clk              hps_io_phery_sdmmc_D1
hps_io_jtag_tck                 hps_io_phery_sdmmc_D2
hps_io_jtag_tdi                 hps_io_phery_sdmmc_D3
hps_io_jtag_tdo                 hps_io_phery_uart0_RX
hps_io_jtag_tms                 hps_io_phery_uart0_TX
hps_io_phery_emac0_MDC          hps_io_phery_usb0_CLK
hps_io_phery_emac0_MDIO         hps_io_phery_usb0_DATA0
hps_io_phery_emac0_RX_CLK       hps_io_phery_usb0_DATA1
hps_io_phery_emac0_RX_CTL       hps_io_phery_usb0_DATA2
hps_io_phery_emac0_RXD0         hps_io_phery_usb0_DATA3
hps_io_phery_emac0_RXD1         hps_io_phery_usb0_DATA4
hps_io_phery_emac0_RXD2         hps_io_phery_usb0_DATA5
hps_io_phery_emac0_RXD3         hps_io_phery_usb0_DATA6
hps_io_phery_emac0_TX_CLK       hps_io_phery_usb0_DATA7
hps_io_phery_emac0_TX_CTL       hps_io_phery_usb0_DIR
hps_io_phery_emac0_TXD0         hps_io_phery_usb0_NXT
hps_io_phery_emac0_TXD1         hps_io_phery_usb0_STP

As you can see, the peripheral list that was enabled in Platform Designer for our HPS matches the list that we saw in the example devicetree, which makes sense. We see the gpio1, emac0, i2c1, sdmmc, uart0 and usb0 peripherals represented in the pin mux list and the example devicetree enabled drivers for each of those. The example devicetree also enabled the qspi peripheral which is not allocated on HPS IO pins but it is allocated on the SDM IO pins of the device. The hps_io_hps_ocs_clk and hps_io_jtag_* ports represent clock and debug interface pins which are not peripherals that require devicetree enablement.

If you would like to locate the Linux device drivers and Linux documentation for the HPS peripherals, you can grep the Documentation and drivers directories in the kernel source tree for the compatibility strings that you observe in the HPS devicetree nodes. For Stratix 10 SoC, you could for instance run this bash command sequence to do just that:

[]$ grep "compatible" ./arch/arm64/boot/dts/altera/socfpga_stratix10.dtsi | \
sort -u | \
sed -e "s/^\s*\(.*\)/\\1/" | \
while read NEXT ; do \
echo $NEXT ; \
echo $NEXT | \
sed -e 's/[^"]*["]\([^"]*\)["][^"]*/\1\n/g' | \
while read NEXT1 ; do \
if [ "$NEXT1" != "syscon" ] ; then \
if [ -n "$NEXT1" ] ; then \
grep -Rl "$NEXT1" Documentation/* ; \
grep -Rl "$NEXT1" drivers/* ; \
fi ; \
fi ;
done | sort -u ; \
echo "" ; \
done

What should the devicetree enable in the FPGA?

The primary value of the devicetree is to inform kernel drivers and modules of the peripheral implementation parameters that the hardware design contains. If you are putting your own custom hardware into the FPGA, then you will be the one to decide how you plan to implement the software to interact with that hardware. If you plan to write a user space device driver for your hardware, you may find no need whatsoever in having devicetree entries that define your hardware configuration. Even if you plan to write a kernel mode driver for your hardware, you may or may not want to have your driver configured by parameters stored in a devicetree. A software driver only needs to have devicetree entries if it is built to use them and there are ways to build a driver such that it does not use devicetree entries. Your personal preferences and deployment goals for your product should dictate if and how you use the devicetree to configure your driver. On the other hand, there is no reason you cannot stuff the devicetree full of arbitrary information that no drivers or modules care about. The kernel exposes all the devicetree structure in the procfs and sysfs, so you can observe all the devicetree information from user space which may be useful for some implementations.

If you decide that you would prefer to use devicetree entries to inform your driver of your hardware configuration, then you need to know how to implement them properly in the devicetree. You may also use other standard components in the FPGA which have software device drivers that will rely on devicetree parameters to configure themselves at run time, so you need to ensure that you have properly configured those as well.

In the GHRD example FPGA hardware design that we are currently running on the Stratix 10 SoC DK there are a few peripherals which have device drivers that are configured via devicetree. How can we identify what FPGA peripherals we have? And which of those need to be entered in our devicetree?

Again, we could open our Platform Designer system up and manually inspect it to see what peripherals we have instantiated in the FPGA, but we are going to use a utility called sopc-create-header-files to help us decode the data that is generated into the SOPCINFO file and extract the relevant information that we need into C formatted header files.

Start by creating a directory to capture the headers and generate the headers:

[]$ cd ..
[]$ mkdir pd_headers
[]$ sopc-create-header-files \
./s10_soc_devkit_ghrd/qsys_top/qsys_top.sopcinfo \
--output-dir pd_headers
[]$ cd pd_headers

That command should have produced several header files for our GHRD hardware system from the perspective of each master in the system, as well as some system wide perspectives:

[]$ ls
fpga_m2ocm_pb.h               jtg_mst_f2sdram_m_2_master.h
frz_bdg_0.h                   jtg_mst_fpga_m.h
hps_mge_mge_rcfg_0.h          jtg_mst_fpga_m_master.h
hps_mge_pb_0.h                jtg_mst_hps_m.h
hps_mge_pb_mge_rcfg_0.h       jtg_mst_hps_m_master.h
jtg_mst_f2sdram_m_0.h         periph_pb_cpu_0.h
jtg_mst_f2sdram_m_0_master.h  pr_region_0_pr_mm_bdg_pr_region_0_0.h
jtg_mst_f2sdram_m_1.h         qsys_top.h
jtg_mst_f2sdram_m_1_master.h  s10_hps.h
jtg_mst_f2sdram_m_2.h

We can use a few grep patterns to extract the base address of the FPGA peripherals in our hardware design:

[]$ cat qsys_top.h | \
grep -E -e "#define S10_HPS_" | \
grep -E -e "_BASE\s"
#define S10_HPS_OCM_BASE 0x0
#define S10_HPS_SYSID_BASE 0x0
#define S10_HPS_MGE_LED_PIO_BASE 0x100
#define S10_HPS_MGE_RCFG_PIO_BASE 0x110
#define S10_HPS_FRZ_CTRL_0_BASE 0x450
#define S10_HPS_PR_REGION_0_PR_SYSID_PR_REGION_0_0_BASE 0x800
#define S10_HPS_PR_REGION_0_PR_OCM_PR_REGION_0_0_BASE 0x900
#define S10_HPS_PERIPH_BUTTON_PIO_BASE 0x1060
#define S10_HPS_PERIPH_DIPSW_PIO_BASE 0x1070
#define S10_HPS_PERIPH_LED_PIO_BASE 0x1080
#define S10_HPS_PERIPH_ILC_BASE 0x1100
#define S10_HPS_HPS_MGE_ALT_MGE_PHY_1_BASE 0x3000
#define S10_HPS_HPS_MGE_HPS_TO_MGE_GMII_ADAPTER_1_BASE 0x3040
#define S10_HPS_HPS_MGE_ALT_MGE_PHY_2_BASE 0x3080
#define S10_HPS_HPS_MGE_HPS_TO_MGE_GMII_ADAPTER_2_BASE 0x30c0

There are many more peripherals in this GHRD example than we care about for our devicetree example. And we can also observe how this output only contains the offset to the base address of the FPGA peripherals from the bridge that they are connected to, so we will need to add the offset to the base address of each bridge before we use these offsets to the peripheral in the devicetree definition. We can tell which peripherals are connected to each bridge on the HPS by opening the system in the Platform Designer GUI and visually inspecting it, or we could also parse this information from the connection properties captured in the SOPCINFO file. If we want to see the connections to the "h2f_axi_master" and the "h2f_lw_axi_master" on the HPS we can parse the SOPCINFO file like this:

[]$ cat ../s10_soc_devkit_ghrd/qsys_top/qsys_top.sopcinfo | \
sed -E -e "/^\s*<connection$/,/^.*>/P" -e "D" | \
sed -E -e "/name=/P" -e "D" | \
sed -E -e "/s10_hps\./P" -e "D" | \
sed -E -e "/(h2f_lw_axi_master|h2f_axi_master)/P" -e "D" | \
sed -E -e "s/.*name=\"(.*)\".*/\1/" | \
sort
s10_hps.h2f_axi_master/ocm.s1
s10_hps.h2f_lw_axi_master/frz_bdg_0.slv_bridge_to_sr
s10_hps.h2f_lw_axi_master/frz_ctrl_0.avl_csr
s10_hps.h2f_lw_axi_master/hps_mge_pb_0.s0
s10_hps.h2f_lw_axi_master/mge_led_pio.s1
s10_hps.h2f_lw_axi_master/mge_rcfg_pio.s1
s10_hps.h2f_lw_axi_master/periph_pb_cpu_0.s0
s10_hps.h2f_lw_axi_master/sysid.control_slave

We can see that there is only one peripheral connected to the "h2f_axi_master" bridge, the onchip memory peripheral. That peripheral does not require a driver so there are no devicetree bindings required for it. The rest of the peripherals in the FPGA are connected through the "h2f_lw_axi_master" bridge, but we are only interested in two of those, the "sysid" and the "periph_pb_cpu_0". The "sysid" peripheral is a standard system ID peripheral which has a default Linux driver and the "periph_pb_cpu_0" peripheral is a bridge that connects to more peripherals downstream inside the FPGA. We can discover the peripherals connected to the "periph_pb_cpu_0" bridge by parsing the SOPCINFO file like this:

[]$ cat ../s10_soc_devkit_ghrd/qsys_top/qsys_top.sopcinfo | \
sed -E -e "/^\s*<connection$/,/^.*>/P" -e "D" | \
sed -E -e "/name=/P" -e "D" | \
sed -E -e "/periph_pb_cpu_0\./P" -e "D" | \
sed -E -e "/m0/P" -e "D" | \
sed -E -e "s/.*name=\"(.*)\".*/\1/" | \
sort
periph_pb_cpu_0.m0/periph_button_pio.s1
periph_pb_cpu_0.m0/periph_dipsw_pio.s1
periph_pb_cpu_0.m0/periph_ILC.avalon_slave
periph_pb_cpu_0.m0/periph_led_pio.s1

We can see that there are four peripherals connected here, three PIO peripherals and one Interrupt Latency Counter peripheral. For our devicetree example, we are only interested in the three PIO peripherals, the Interrupt Latency Counter and the system ID peripheral since they all have Linux device drivers that we can enable in the devicetree. All these peripherals are connected to the LW HPS AXI bridge so we will align their base address to that bridge in the devicetree.

Each peripheral in the devicetree is going to require certain fundamental bindings like the compatible, reg, and interrupts bindings. And some of the peripherals may require additional bindings that are unique to that peripheral. Here is how we find some of the basic information that each peripheral will require for the fundamental bindings.

We can find the base address and span of each peripheral required by the reg binding by looking at the general parameters exposed in the Platform Designer header data, a broad filter like this will expose much of it:

[]$ cat qsys_top.h | \
grep -E -e "#define S10_HPS_"

Output not shown.

The way Stratix 10 FPGA interrupts translate into the HPS GIC is architecture specific. We can use this filter to extract the interrupt information from the header file like this:

[]$ cat qsys_top.h | \
grep -E -e "#define S10_HPS_" | \
grep -E -e "_IRQ\s"
#define S10_HPS_FRZ_CTRL_0_IRQ 11
#define S10_HPS_PERIPH_BUTTON_PIO_IRQ 1
#define S10_HPS_PERIPH_DIPSW_PIO_IRQ 0

We can see from this that we have an interrupt driven by two of the pio peripherals along with a freeze controller which we do not care about in this example. These interrupt values indicate the position of these interrupts on the FPGA-to-HPS interrupt receiver which is a 64-bit vector. These pio interrupts are assigned to bits 0, and 1 of that vector. On Stratix 10 SoC devices the 0th bit of that vector represents interrupt 49 into the GIC of the ARM processor. In the devicetree we must subtract off the first 32 interrupt locations that represent inter-processor interrupts. We can define an equation like this to represent these FPGA driven interrupts in the devicetree, FPGA-interrupt-vector + 49 - 32. And when you observe the devicetree examples that we create below you will see that we implement this equation for the interrupt entries.

For more information on how the ARM GIC expects us to format interrupt entries in the devicetree, you can look at this document:

[]$ find ../linux-socfpga/ -name arm,gic.yaml
../linux-socfpga/Documentation/devicetree/bindings/interrupt-controller/arm,gic.yaml

Or in your web browser you can look here.

The pio peripherals will need some additional information provided in their devicetree bindings, so while we are filtering the Platform Designer header, we can extract that information like this:

[]$ cat qsys_top.h | \
grep -E -e "#define S10_HPS_PERIPH_" | \
grep -E -e "PIO_DATA_WIDTH\s" -e "PIO_EDGE_TYPE\s" -e "PIO_IRQ_TYPE\s"
#define S10_HPS_PERIPH_BUTTON_PIO_DATA_WIDTH 4
#define S10_HPS_PERIPH_BUTTON_PIO_EDGE_TYPE FALLING
#define S10_HPS_PERIPH_BUTTON_PIO_IRQ_TYPE EDGE
#define S10_HPS_PERIPH_DIPSW_PIO_DATA_WIDTH 4
#define S10_HPS_PERIPH_DIPSW_PIO_EDGE_TYPE FALLING
#define S10_HPS_PERIPH_DIPSW_PIO_IRQ_TYPE EDGE
#define S10_HPS_PERIPH_LED_PIO_DATA_WIDTH 3
#define S10_HPS_PERIPH_LED_PIO_EDGE_TYPE NONE
#define S10_HPS_PERIPH_LED_PIO_IRQ_TYPE NONE

How did we know that the pio would require some additional binding information? And what other bindings might be needed by any of these peripherals? We can figure out what standard Altera peripherals may have drivers and require devicetree bindings by grep'ing for the string "altr," in the Documentation and drivers directories in the Linux source tree, like this:

[]$ grep -Rl "altr," ../linux-socfpga/Documentation/*

NOTE: with the Intel acquisition of Altera, you are now starting to see new drivers for Intel FPGA devices which carry the "intel," prefix rather than the "altr," prefix that the legacy Altera components were created with.

Out of the list created by the grep command above, we are interested in these three documents that cover the peripherals that we have in our GHRD hardware design:

[]$ find ../linux-socfpga/Documentation/ \
-name altera_sysid.txt -o \
-name gpio-altera.txt -o \
-name altera-interrupt-latency-counter.txt
../linux-socfpga/Documentation/devicetree/bindings/gpio/gpio-altera.txt
../linux-socfpga/Documentation/devicetree/bindings/misc/altera-interrupt-latency-counter.txt
../linux-socfpga/Documentation/devicetree/bindings/misc/altera_sysid.txt

These files can be viewed in a web browser here:

• gpio-altera.txt
• altera-interrupt-latency-counter.txt
• altera_sysid.txt

The one thing that each of these files will provide is the compatibility string that the driver expects. The pio and ILC peripherals have several other bindings that are required and optional that you can include in the devicetree.

With this information about where we can find the details for peripherals that exist inside our FPGA Platform Designer system and how to locate software driver documentation for existing Altera peripherals, we can now proceed to add the FPGA peripherals to our existing devicetree example.

Add the FPGA peripherals to the Stratix 10 SoC devicetree

There are several ways to add the FPGA peripherals into the main devicetree, this example will demonstrate two practical and popular ways of doing this. First, we demonstrate how to create a devicetree overlay which can be dynamically applied to and removed from the main system devicetree at run-time. Then we demonstrate how to just embed our FPGA peripherals into the main system devicetree so that it loads at boot with the rest of the devicetree. Since we left the Stratix 10 SoC DK in a state where it is running the base example devicetree with no FPGA contents, we will begin with the overlay example first.

Copy the qsys_top.h header into the devicetree source directory in the kernel source tree and change into the kernel source tree directory:

[]$ cp qsys_top.h ../linux-socfpga/arch/arm64/boot/dts/altera/
[]$ cd ../linux-socfpga

Devicetree overlay method

Create a new file called arch/arm64/boot/dts/altera/my_s10_fpga_overlay.dts to store our FPGA peripheral definition as a devicetree overlay:

[]$ cat << EOF | base64 -d -i | xz -d > arch/arm64/boot/dts/altera/my_s10_fpga_overlay.dts
/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4A5jBAddABeKfUvNK1NbzTI+B3OUEJMYagjnn0bslkyF
7xuB8nrKCzWcgqNMVnytO8kU434lb6O/lVpXis+nN97FmMmUs2NSmBw+BzMMy9zaK6yGnL1+gDhJ
NSaOGp7SxBv9NFQb+t/EZduLl7lBQ/A6ng6jkddLTY2Wr0tJnArh9Igkc41FkLLhYu6utx4tAQ+c
W9b0H6/Vn/Ur90lI7IizHnZgl0rEOEk801kapWbgnvol+o7kcFRm9WZQP+Dhfnc7zejyWLH2QUG6
8NlkgCKSWk6OJHrJRxJGzzzsEsRMi0pDvAc7Ig2TdXm+aYXbhbgdOrzRCm93pmq3keAHULU/2CgB
TpvCoC5iPCdrLhGcrAX6jSg2Vnk/S+FgMKzaQZgYrxY+krG+HWnuhbYKncTCsFcKFf/QPYXxJvZQ
CGiF0Q9zhFr5HiBmO8xEnKFwMBuA7dPtsQSp06gXt+VFYINxNqoUsZhQ+Jwd4hKfgcLyKcT9IYmm
vx4S3TojlIgms5EzvjKo9BiLKW+Ni4u+BIJszbED79nawLRjQ8pDPgCIBNBpvIpQGFJt+NorgGlS
dYtf3u3Ahezb01zbCyiRJvfPyukkjSd7KZ47bwtqJzX+o+uQKYJ8tjNpk5JoYjcVH9wOPV6W0kS6
hFknq49ZmmRmcozk/mzdUxCTMAHD6SmR4+41LhaFjfLCl7cFqV4qFPYCUBFEq6YsX1Pjm2cUc3nf
Pl0RjvQhFlc96ROfRISamKK9XilUofInu1LnwJCiAYrIEJkJC6sveoT5dGeWruy++utnBoCPftfz
y0iKuncg69XBbdbHCN0X6330tuYAd44ZhFnd2ZA3TCShnW25BFTijunNePWwPr7qZ6P3KIZ8N/UO
g78DGSA4oMeJQFhzpy90mX9u47bZJtRpXhUNakMJZvIe0OIKUrwVN8t9fVK8ZPVuIhG01wjG5lAb
iLbVIVsJ3Q3+/IeRPzIbmWbP5hExTkAGSEPXQLYVlilBF7ncoCw4wmQmNSJ0M5US0wToceLDf55h
eJc9gLri2OKD51HZ9QGbe5S/eOjSNYbYNRafoHs6pvfdUHAZ77IBPtUFW7brcp5ZGPN2HmiPpkfn
bNLPc+siohPGuu5XJrMHzGn6dvtAbvzgxbAv0ywDurcUmHrptP8KsSWEnWGamLO51ioZfBMRzcCI
39hPovaSxgeyHiGMD3CBp95MDcka/GVm9cZAfjWkuauspEB1hd1mHDvKQQO2iEHal2hQhyhBu187
G3wjJPoevQsyUCJ05kJKr/MZ7/P5DyxFNhhFkVGzFV4bzoLnhmO6f6a+5jhDlOzsoZ2qr4DJJBmL
OCgt8DlBTAiRD/JxiHtVPxRTvc+VPgkQlOo+cX+a2oG0rJlZAABb6wierFCccQABowjkHAAAJ3CM
ILHEZ/sCAAAAAARZWg==
EOF

/*
 * SPDX-License-Identifier: MIT-0
 * Copyright (c) 2018-2022 Intel Corporation
 */

/dts-v1/;
/plugin/;

#include "qsys_top.h"
#include "dt-bindings/interrupt-controller/irq.h"
#include "dt-bindings/gpio/gpio.h"

/*
 * Redefine the macros from qsys_top.h into shorter macros and more useful
 * values that we can use directly in the device tree definition.  We will also
 * take care of the bridge offsets that we require for the true base address of
 * the H2F and LW_H2F bridge.
 */

#define S10_H2F_BASE    0x80000000
#define S10_LW_H2F_BASE 0xF9000000

#define SYSID_QSYS_UNIT_ADDRESS S10_LW_H2F_BASE,S10_HPS_SYSID_BASE
#define SYSID_QSYS_BASE         (S10_LW_H2F_BASE + S10_HPS_SYSID_BASE)
#define SYSID_QSYS_SPAN         S10_HPS_SYSID_SPAN

#define LED_PIO_UNIT_ADDRESS    S10_LW_H2F_BASE,S10_HPS_PERIPH_LED_PIO_BASE
#define LED_PIO_BASE            (S10_LW_H2F_BASE + S10_HPS_PERIPH_LED_PIO_BASE)
#define LED_PIO_SPAN            S10_HPS_PERIPH_LED_PIO_SPAN
#define LED_PIO_WIDTH           S10_HPS_PERIPH_LED_PIO_DATA_WIDTH

#define BUTTON_PIO_UNIT_ADDRESS S10_LW_H2F_BASE,S10_HPS_PERIPH_BUTTON_PIO_BASE
#define BUTTON_PIO_BASE         (S10_LW_H2F_BASE + S10_HPS_PERIPH_BUTTON_PIO_BASE)
#define BUTTON_PIO_SPAN         S10_HPS_PERIPH_BUTTON_PIO_SPAN
#define BUTTON_PIO_IRQ          (S10_HPS_PERIPH_BUTTON_PIO_IRQ + 49 - 32)
#define BUTTON_PIO_WIDTH        S10_HPS_PERIPH_BUTTON_PIO_DATA_WIDTH

#define DIPSW_PIO_UNIT_ADDRESS  S10_LW_H2F_BASE,S10_HPS_PERIPH_DIPSW_PIO_BASE
#define DIPSW_PIO_BASE          (S10_LW_H2F_BASE + S10_HPS_PERIPH_DIPSW_PIO_BASE)
#define DIPSW_PIO_SPAN          S10_HPS_PERIPH_DIPSW_PIO_SPAN
#define DIPSW_PIO_IRQ           (S10_HPS_PERIPH_DIPSW_PIO_IRQ + 49 - 32)
#define DIPSW_PIO_WIDTH         S10_HPS_PERIPH_DIPSW_PIO_DATA_WIDTH

#define ILC_UNIT_ADDRESS        S10_LW_H2F_BASE,S10_HPS_PERIPH_ILC_BASE
#define ILC_BASE                (S10_LW_H2F_BASE + S10_HPS_PERIPH_ILC_BASE)
#define ILC_SPAN                S10_HPS_PERIPH_ILC_SPAN

#define FRZ_CTRL_0_IRQ          (S10_HPS_FRZ_CTRL_0_IRQ + 49 - 32)

&{/soc/base_fpga_region} {

        external-fpga-config;

        #address-cells = <0x1>;
        #size-cells = <0x1>;

        ranges;

        SYSID_QSYS: SYSID_QSYS@SYSID_QSYS_UNIT_ADDRESS {
                compatible = "altr,sysid-1.0";
                reg = <SYSID_QSYS_BASE SYSID_QSYS_SPAN>;
        };

        LED_PIO: LED_PIO@LED_PIO_UNIT_ADDRESS {
                compatible = "altr,pio-1.0";
                reg = <LED_PIO_BASE LED_PIO_SPAN>;
                altr,ngpio = <LED_PIO_WIDTH>;
                #gpio-cells = <2>;
                gpio-controller;
        };

        BUTTON_PIO: BUTTON_PIO@BUTTON_PIO_UNIT_ADDRESS {
                compatible = "altr,pio-1.0";
                reg = <BUTTON_PIO_BASE BUTTON_PIO_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <0 BUTTON_PIO_IRQ IRQ_TYPE_LEVEL_HIGH>;
                altr,ngpio = <BUTTON_PIO_WIDTH>;
                #gpio-cells = <2>;
                gpio-controller;
                #interrupt-cells = <1>;
                interrupt-controller;
                altr,interrupt-type = <IRQ_TYPE_EDGE_FALLING>;
        };

        DIPSW_PIO: DIPSW_PIO@DIPSW_PIO_UNIT_ADDRESS {
                compatible = "altr,pio-1.0";
                reg = <DIPSW_PIO_BASE DIPSW_PIO_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <0 DIPSW_PIO_IRQ IRQ_TYPE_LEVEL_HIGH>;
                altr,ngpio = <DIPSW_PIO_WIDTH>;
                #gpio-cells = <2>;
                gpio-controller;
                #interrupt-cells = <1>;
                interrupt-controller;
                altr,interrupt-type = <IRQ_TYPE_EDGE_BOTH>;
        };

        ILC: ILC@ILC_UNIT_ADDRESS {
                compatible = "altr,ilc-1.0";
                reg = <ILC_BASE ILC_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <
                        0       FRZ_CTRL_0_IRQ  IRQ_TYPE_LEVEL_HIGH
                        0       BUTTON_PIO_IRQ  IRQ_TYPE_LEVEL_HIGH
                        0       DIPSW_PIO_IRQ   IRQ_TYPE_LEVEL_HIGH
                        >;
                altr,sw-fifo-depth = <32>;
        };

        leds {
                compatible = "gpio-leds";

                fpga0 {
                        label = "fpga_led0";
                        gpios = <&LED_PIO 0 GPIO_ACTIVE_HIGH>;
                };

                fpga1 {
                        label = "fpga_led1";
                        gpios = <&LED_PIO 1 GPIO_ACTIVE_HIGH>;
                };

                fpga2 {
                        label = "fpga_led2";
                        gpios = <&LED_PIO 2 GPIO_ACTIVE_HIGH>;
                };
        };

}; // base_fpga_region

Build this devicetree overlay file using the kernel build environment just like we built the main system devicetree example above:

[]$ make altera/my_s10_fpga_overlay.dtb

Note that these warnings are expected and benign for our example:

arch/arm64/boot/dts/altera/my_s10_fpga_overlay.dts:55.2-9:
Warning (ranges_format): /fragment@0/__overlay__:ranges: empty "ranges" property
but its #address-cells (1) differs from /fragment@0 (2)

arch/arm64/boot/dts/altera/my_s10_fpga_overlay.dts:48.26-126.3:
Warning (avoid_default_addr_size): /fragment@0/__overlay__: Relying on default
#address-cells value

arch/arm64/boot/dts/altera/my_s10_fpga_overlay.dts:48.26-126.3:
Warning (avoid_default_addr_size): /fragment@0/__overlay__: Relying on default
#size-cells value

You should be able to locate the DTB that was just built:

[]$ find -name my_s10_fpga_overlay.dtb
./arch/arm64/boot/dts/altera/my_s10_fpga_overlay.dtb

To test this devicetree overlay, copy it over to the target that is running the my_s10_top.dtb example devicetree:

[]$ scp ./arch/arm64/boot/dts/altera/my_s10_fpga_overlay.dtb root@<TARGET-IP-ADDR>:/home/root/

Place the my_s10_fpga_overlay.dtb overlay file in the /lib/firmware directory, you may need to create that directory:

root@stratix10:~# mkdir -p /lib/firmware
root@stratix10:~# mv my_s10_fpga_overlay.dtb /lib/firmware/

With the devicetree overlay DTB in the firmware directory, you can now apply it to the system devicetree. Do that by creating a directory under /sys/kernel/config/device-tree/overlays/, and then write the name of the devicetree overlay to the resultant path file that gets created in that new directory. There is a status file that gets created in that directory as well that informs you of the overlay status:

root@stratix10:~# mkdir /sys/kernel/config/device-tree/overlays/my-s10-fpga-overlay
root@stratix10:~# ls /sys/kernel/config/device-tree/overlays/my-s10-fpga-overlay
dtbo    path    status
root@stratix10:~# cat /sys/kernel/config/device-tree/overlays/my-s10-fpga-overlay/status
unapplied
root@stratix10:~# echo my_s10_fpga_overlay.dtb \
> /sys/kernel/config/device-tree/overlays/my-s10-fpga-overlay/path
root@stratix10:~# cat /sys/kernel/config/device-tree/overlays/my-s10-fpga-overlay/status
applied

Once this overlay is applied to the main system devicetree, you should see the FPGA LEDs on the Stratix 10 SoC DK turn off if they were in the uninitialized state after the FPGA was programmed at boot. In this version of the GSRD, the scroll_server application is running which means that the FPGA LEDs should scroll back and forth now that the FPGA peripherals are enabled. Even with the scroll server running, you should be able to toggle an FPGA LED at this point like this:

root@stratix10:~# echo 1 > /sys/class/leds/fpga_led0/brightness
root@stratix10:~# echo 0 > /sys/class/leds/fpga_led0/brightness

You should also be able to read the sysid values like this:

root@stratix10:~# cat /sys/bus/platform/devices/f9000000.SYSID_QSYS/sysid/id
2899692286
root@stratix10:~# cat /sys/bus/platform/devices/f9000000.SYSID_QSYS/sysid/timestamp
1657724789 (2022-7-13 15:6:29 UTC)

If you investigate the sysfs and procfs, as shown in the section below, you should be able to see all the new devicetree entries that were applied with the overlay. If you want to remove the overlay, just remove the directory that you created in /sys/kernel/config/device-tree/overlays/:

root@stratix10:~# rmdir /sys/kernel/config/device-tree/overlays/my-s10-fpga-overlay

Main system devicetree integration method

Create a new file called arch/arm64/boot/dts/altera/my_s10_fpga.dtsi to store our FPGA peripheral definition as part of the main system devicetree:

[]$ cat << EOF | base64 -d -i | xz -d > arch/arm64/boot/dts/altera/my_s10_fpga.dtsi
/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4A6wA+FdABeKfUvNK1NbzTI+B3OUEJMYagjnn0bslkyF
7xuB8nrKCzWcgqNMVnytO8kU434lb6O/lVpXis+nN97FmMmUs2NSmBw+BzMMy9zaK6yGnL1+gDhJ
NNH2JEe0oQyJ1mdBqrCCpJSTH6LbkKHAFpUymBW1Os+F4faXab6d0xnXOgF8ZPnjLIEq+qOxkAvj
cBrqm4ZAdhK4sOpQMTEZ/jbwGaul1G+PeA+p2/3LnB31oA4PXedfQe0j5HFeNcLzlxL3M1aDNgH8
qGz70cAjZaAP+F+8X1/KXI6Do7a2BomBSPm8y5CzPmrgk7Bk2ns+SM7CvnwrH8LFbVwsHfwJNGXj
MrqgksHnPb69OtZATpbPZwzjX0zLInuJDLTnyvZtIv20VAZQHA+kvzVK7nSC24+NfJ+OdO3y32NS
B9u9bTwjhzCLb7IxfRF9LtBT1MbCPQ7JRbgQcwimQ6apc/EC4Tj62ME3YpbIBdEdYXVEMO/CwSPo
/Sw8Ve0novqMeBnbJ/71P+mQpottELXJlasxnsxPL5+iJfyoAhmGZsmPHliRLhzOJQeTHH415Lgc
tTs1Coph5kdkQsIaAIcowMwTyVpoothi5Ru8hj338eufSJK3bDM1VL05/5aUR7QV7gZ9IwrNQSSe
T74+UlQw4Fg8p78d0pcDwbTEf/FFOoMSyW7YFFT09yR1mR2pxOjdzeO7DI+WasI+PPWj8ZnJtPKP
8faYVafShCkj7fPgo2XxPrw8zvaJ0FbeyM2sVnBUTN6y0jemShtHrIEnNKLp6WcvJ+b/IFJNhdeh
iRZjXYxmGMC8VpFmtuJ3ol5WFxlTyP7809CtaD//+ZHBXMqSXG6J2/h+1CFDEP+mSLo17KEKZ9AU
/fSrdgJRaaqz+na1L//nT5A6bx3ODbKxBPY5CDZtoUcgVs74HPUplUDnc60XNa7/88NVuzx4OJay
9L2q0QBHoXL2t+LH+DtkDYKd8IPle/6znAsDVF8926JJfbPyhmWGRU3njg3M9Q6teIE7pIls3jH1
l5eNpTufwnvwyQQIldS1MVpde8Bt1tUqI96dJkKjjtNgTJPkh7j7/qzwd+O9tzleuwm0GzZuEIrD
MZC0ED90cduS5Y27cxGN5Tx1ywOhd8WwrO7qlXHCwp8pEzilt75lIK/BuFa4UpybdJ5+r9jvVn7B
ALPIjU9YpD7E7Luj6YF2meCnJrEYpdTCd2WO3UrtBr1c3R5v+OK+1hf39FOm+3Dw88EaLLX3yKTK
uRiw/QgNJdLku0uQCJkGYVxzdaZpw7OzAwRlvQzJK+b+3CrQrNFWdraPUZtJFw91FyxSVPj8BgAA
AAB6bK59pHGLkQAB/QexHQAA1htJabHEZ/sCAAAAAARZWg==
EOF

/*
 * SPDX-License-Identifier: MIT-0
 * Copyright (c) 2018-2022 Intel Corporation
 */

#include "qsys_top.h"
#include "dt-bindings/interrupt-controller/irq.h"
#include "dt-bindings/gpio/gpio.h"

/*
 * Redefine the macros from qsys_top.h into shorter macros and more useful
 * values that we can use directly in the device tree definition.  We will also
 * take care of the bridge offsets that we require for the true base address of
 * the H2F and LW_H2F bridge.
 */

#define S10_H2F_BASE    0x80000000
#define S10_LW_H2F_BASE 0xF9000000

#define SYSID_QSYS_UNIT_ADDRESS S10_LW_H2F_BASE,S10_HPS_SYSID_BASE
#define SYSID_QSYS_BASE         (S10_LW_H2F_BASE + S10_HPS_SYSID_BASE)
#define SYSID_QSYS_SPAN         S10_HPS_SYSID_SPAN

#define LED_PIO_UNIT_ADDRESS    S10_LW_H2F_BASE,S10_HPS_PERIPH_LED_PIO_BASE
#define LED_PIO_BASE            (S10_LW_H2F_BASE + S10_HPS_PERIPH_LED_PIO_BASE)
#define LED_PIO_SPAN            S10_HPS_PERIPH_LED_PIO_SPAN
#define LED_PIO_WIDTH           S10_HPS_PERIPH_LED_PIO_DATA_WIDTH

#define BUTTON_PIO_UNIT_ADDRESS S10_LW_H2F_BASE,S10_HPS_PERIPH_BUTTON_PIO_BASE
#define BUTTON_PIO_BASE         (S10_LW_H2F_BASE + S10_HPS_PERIPH_BUTTON_PIO_BASE)
#define BUTTON_PIO_SPAN         S10_HPS_PERIPH_BUTTON_PIO_SPAN
#define BUTTON_PIO_IRQ          (S10_HPS_PERIPH_BUTTON_PIO_IRQ + 49 - 32)
#define BUTTON_PIO_WIDTH        S10_HPS_PERIPH_BUTTON_PIO_DATA_WIDTH

#define DIPSW_PIO_UNIT_ADDRESS  S10_LW_H2F_BASE,S10_HPS_PERIPH_DIPSW_PIO_BASE
#define DIPSW_PIO_BASE          (S10_LW_H2F_BASE + S10_HPS_PERIPH_DIPSW_PIO_BASE)
#define DIPSW_PIO_SPAN          S10_HPS_PERIPH_DIPSW_PIO_SPAN
#define DIPSW_PIO_IRQ           (S10_HPS_PERIPH_DIPSW_PIO_IRQ + 49 - 32)
#define DIPSW_PIO_WIDTH         S10_HPS_PERIPH_DIPSW_PIO_DATA_WIDTH

#define ILC_UNIT_ADDRESS        S10_LW_H2F_BASE,S10_HPS_PERIPH_ILC_BASE
#define ILC_BASE                (S10_LW_H2F_BASE + S10_HPS_PERIPH_ILC_BASE)
#define ILC_SPAN                S10_HPS_PERIPH_ILC_SPAN

#define FRZ_CTRL_0_IRQ          (S10_HPS_FRZ_CTRL_0_IRQ + 49 - 32)

/ {
soc {

base_fpga_region {

        external-fpga-config;

        #address-cells = <0x2>;
        #size-cells = <0x1>;

        ranges = <
                S10_LW_H2F_BASE SYSID_QSYS_BASE SYSID_QSYS_BASE SYSID_QSYS_SPAN
                S10_LW_H2F_BASE LED_PIO_BASE    LED_PIO_BASE    LED_PIO_SPAN
                S10_LW_H2F_BASE BUTTON_PIO_BASE BUTTON_PIO_BASE BUTTON_PIO_SPAN
                S10_LW_H2F_BASE DIPSW_PIO_BASE  DIPSW_PIO_BASE  DIPSW_PIO_SPAN
                S10_LW_H2F_BASE ILC_BASE        ILC_BASE        ILC_SPAN
                >;

        SYSID_QSYS: SYSID_QSYS@SYSID_QSYS_UNIT_ADDRESS {
                compatible = "altr,sysid-1.0";
                reg = <S10_LW_H2F_BASE  SYSID_QSYS_BASE SYSID_QSYS_SPAN>;
        };

        LED_PIO: LED_PIO@LED_PIO_UNIT_ADDRESS {
                compatible = "altr,pio-1.0";
                reg = <S10_LW_H2F_BASE  LED_PIO_BASE LED_PIO_SPAN>;
                altr,ngpio = <LED_PIO_WIDTH>;
                #gpio-cells = <2>;
                gpio-controller;
        };

        BUTTON_PIO: BUTTON_PIO@BUTTON_PIO_UNIT_ADDRESS {
                compatible = "altr,pio-1.0";
                reg = <S10_LW_H2F_BASE  BUTTON_PIO_BASE BUTTON_PIO_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <0 BUTTON_PIO_IRQ IRQ_TYPE_LEVEL_HIGH>;
                altr,ngpio = <BUTTON_PIO_WIDTH>;
                #gpio-cells = <2>;
                gpio-controller;
                #interrupt-cells = <1>;
                interrupt-controller;
                altr,interrupt-type = <IRQ_TYPE_EDGE_FALLING>;
        };

        DIPSW_PIO: DIPSW_PIO@DIPSW_PIO_UNIT_ADDRESS {
                compatible = "altr,pio-1.0";
                reg = <S10_LW_H2F_BASE  DIPSW_PIO_BASE DIPSW_PIO_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <0 DIPSW_PIO_IRQ IRQ_TYPE_LEVEL_HIGH>;
                altr,ngpio = <DIPSW_PIO_WIDTH>;
                #gpio-cells = <2>;
                gpio-controller;
                #interrupt-cells = <1>;
                interrupt-controller;
                altr,interrupt-type = <IRQ_TYPE_EDGE_BOTH>;
        };

        ILC: ILC@ILC_UNIT_ADDRESS {
                compatible = "altr,ilc-1.0";
                reg = <S10_LW_H2F_BASE  ILC_BASE ILC_SPAN>;
                interrupt-parent = <&intc>;
                interrupts = <
                        0       FRZ_CTRL_0_IRQ  IRQ_TYPE_LEVEL_HIGH
                        0       BUTTON_PIO_IRQ  IRQ_TYPE_LEVEL_HIGH
                        0       DIPSW_PIO_IRQ   IRQ_TYPE_LEVEL_HIGH
                        >;
                altr,sw-fifo-depth = <32>;
        };
};

}; // soc
}; // /

Edit the arch/arm64/boot/dts/altera/my_s10_top.dts file to include our new FPGA devicetree definition:

[]$ cat << EOF | base64 -d -i | xz -d | bash
/Td6WFoAAATm1rRGAgAhARwAAAAQz1jM4AG2AN1dADmZSuojhxSi3Mv8MfjO12CEtsMK+7liVcns
z5YLtM3IyUrqx0d48jihm/tdXznrLSs+MOPkF3tycJSyGAFERupj4GAZSnixKKN1DB3rAVjCD6rh
vhqeA7guyEXqPRVlONAnpqVQeQONeYWtoxwU1FKmMY3zs6xc8aND+jinGOEx3SdETDiCZ7RcfgBc
H77vdj+quSqJ2KRTPm+onLksOwv5L/FbrWnFw+/GB8c3BsepOO8A4irmL8sanS5/8d2qgXPobrKA
dmc0AMqwzxTCmiMdzwR9t/ss5Xep5NtlAAAAAK+tls7XzxpIAAH5AbcDAADGfUtOscRn+wIAAAAA
BFla
EOF

set -x
sed \
-i.bak \
-e '/#include "socfpga_stratix10.dtsi"/a #include "my_s10_fpga.dtsi"' \
-e '/compatible = "gpio-leds";/a \
\
                fpga0 {\
                        label = "fpga_led0";\
                        gpios = <&LED_PIO 0 GPIO_ACTIVE_HIGH>;\
                };\
\
                fpga1 {\
                        label = "fpga_led1";\
                        gpios = <&LED_PIO 1 GPIO_ACTIVE_HIGH>;\
                };\
\
                fpga2 {\
                        label = "fpga_led2";\
                        gpios = <&LED_PIO 2 GPIO_ACTIVE_HIGH>;\
                };\
' \
./arch/arm64/boot/dts/altera/my_s10_top.dts

Build the devicetree:

[]$ make altera/my_s10_top.dtb

You should be able to locate the DTB that was just built:

[]$ find -name my_s10_top.dtb
./arch/arm64/boot/dts/altera/my_s10_top.dtb

Copy the new system DTB over to the target:

[]$ scp ./arch/arm64/boot/dts/altera/my_s10_top.dtb root@<TARGET-IP-ADDR>:/home/root/

Move the DTB onto the FAT partition:

root@stratix10:~# mount /dev/mmcblk0p1 /mnt
root@stratix10:~# mv my_s10_top.dtb /mnt/

Now reboot the target, remember to stop u-boot at the autoboot console:

root@stratix10:~# reboot

...cut linux shutdown and u-boot startup messages...

Hit any key to stop autoboot:  0
SOCFPGA_STRATIX10 #

Just like above, once we reach the u-boot console we must manually boot the Linux kernel with our new devicetree since the default environment attempts to bootm from a configuration in the kernel FIT image on the FAT partition and we do not want that version of the devicetree to be used. We do want to reuse the FPGA core RBF configuration file and the Linux kernel from the FIT image.

Start by loading the kernel FIT image from mmc flash into memory:

SOCFPGA_STRATIX10 # load mmc 0:1 ${loadaddr} kernel.itb
16430839 bytes read in 754 ms (20.8 MiB/s)

Program the fpga-4 RBF image from the FIT image into the FPGA core logic and then enable the FPGA bridges:

SOCFPGA_STRATIX10 # imxtract ${loadaddr} fpga-4
## Copying 'fpga-4' subimage from FIT image at 01000000 ...
crc32+

SOCFPGA_STRATIX10 # fpga load 0 ${fileaddr} ${filesize}
....FPGA reconfiguration OK!

SOCFPGA_STRATIX10 # bridge enable

Decompress the kernel image from the FIT image:

SOCFPGA_STRATIX10 # fdt addr ${loadaddr}
SOCFPGA_STRATIX10 # fdt get addr KERNEL_ADDR /images/kernel data
SOCFPGA_STRATIX10 # lzmadec ${KERNEL_ADDR} ${kernel_addr_r}
Uncompressed size: 34345472 = 0X20C1200

Load our custom devicetree image from mmc flash into memory:

SOCFPGA_STRATIX10 # load mmc 0:1 ${fdt_addr_r} my_s10_top.dtb
17358 bytes read in 5 ms (3.3 MiB/s)

Boot the Linux kernel:

SOCFPGA_STRATIX10 # setenv bootargs "earlycon panic=-1 root=${mmcroot} rw rootwait"
SOCFPGA_STRATIX10 # booti ${kernel_addr_r} - ${fdt_addr_r}

When the kernel boots you should again see the FPGA LEDs turn off as the drivers are loaded based on the devicetree configuration and the scroll server should scroll them back and forth. If you investigate the sysfs and procfs, as shown in the section below, you should be able to see all the new devicetree entries that were added to the main system devicetree.

Observing devicetree evidence in the sysfs and procfs

If you would like to observe the devicetree evidence in the sysfs, you can search for the names that we used in our FPGA section and see where various entries are maintained in the sysfs that reference them:

root@stratix10:~# find /sys \
-name "*SYSID*" -o \
-name "*PIO*" -o \
-name "*ILC*" | \
sort
/sys/bus/platform/devices/f9000000.SYSID_QSYS
/sys/bus/platform/devices/f9001060.BUTTON_PIO
/sys/bus/platform/devices/f9001070.DIPSW_PIO
/sys/bus/platform/devices/f9001080.LED_PIO
/sys/bus/platform/devices/f9001100.ILC
/sys/bus/platform/drivers/altera_gpio/f9001060.BUTTON_PIO
/sys/bus/platform/drivers/altera_gpio/f9001070.DIPSW_PIO
/sys/bus/platform/drivers/altera_gpio/f9001080.LED_PIO
/sys/bus/platform/drivers/altera_ilc/f9001100.ILC
/sys/bus/platform/drivers/altera_sysid/f9000000.SYSID_QSYS
/sys/devices/platform/soc/soc:base_fpga_region/fpga_region/region0/f9000000.SYSID_QSYS
/sys/devices/platform/soc/soc:base_fpga_region/fpga_region/region0/f9001060.BUTTON_PIO
/sys/devices/platform/soc/soc:base_fpga_region/fpga_region/region0/f9001070.DIPSW_PIO
/sys/devices/platform/soc/soc:base_fpga_region/fpga_region/region0/f9001080.LED_PIO
/sys/devices/platform/soc/soc:base_fpga_region/fpga_region/region0/f9001100.ILC
/sys/firmware/devicetree/base/__symbols__/BUTTON_PIO
/sys/firmware/devicetree/base/__symbols__/DIPSW_PIO
/sys/firmware/devicetree/base/__symbols__/ILC
/sys/firmware/devicetree/base/__symbols__/LED_PIO
/sys/firmware/devicetree/base/__symbols__/SYSID_QSYS
/sys/firmware/devicetree/base/soc/base_fpga_region/BUTTON_PIO@0xF9000000,0x1060
/sys/firmware/devicetree/base/soc/base_fpga_region/DIPSW_PIO@0xF9000000,0x1070
/sys/firmware/devicetree/base/soc/base_fpga_region/ILC@0xF9000000,0x1100
/sys/firmware/devicetree/base/soc/base_fpga_region/LED_PIO@0xF9000000,0x1080
/sys/firmware/devicetree/base/soc/base_fpga_region/SYSID_QSYS@0xF9000000,0x0

You can find similar evidence in the procfs as well:

root@stratix10:~# find /proc/device-tree/soc/base_fpga_region/ -maxdepth 1 | sort
/proc/device-tree/soc/base_fpga_region/
/proc/device-tree/soc/base_fpga_region/#address-cells
/proc/device-tree/soc/base_fpga_region/#size-cells
/proc/device-tree/soc/base_fpga_region/BUTTON_PIO@0xF9000000,0x1060
/proc/device-tree/soc/base_fpga_region/DIPSW_PIO@0xF9000000,0x1070
/proc/device-tree/soc/base_fpga_region/ILC@0xF9000000,0x1100
/proc/device-tree/soc/base_fpga_region/LED_PIO@0xF9000000,0x1080
/proc/device-tree/soc/base_fpga_region/SYSID_QSYS@0xF9000000,0x0
/proc/device-tree/soc/base_fpga_region/compatible
/proc/device-tree/soc/base_fpga_region/external-fpga-config
/proc/device-tree/soc/base_fpga_region/fpga-mgr
/proc/device-tree/soc/base_fpga_region/name
/proc/device-tree/soc/base_fpga_region/ranges

Notes about peripherals in this Stratix 10 SoC GSRD example

The SYSID_QSYS, LED_PIO, DIPSW_PIO, and BUTTON_PIO peripherals should all work as expected.

In this version of the GSRD, the ILC peripheral will report errors as the driver loads. This is because the ILC driver does not support the new GPIO IRQ framework that is implemented in the modern kernel. The current ILC driver expects to be able to share IRQs with the peripherals that it monitors, and the modern GPIO framework no longer supports that:

root@stratix10:~# dmesg | grep ilc
[    4.822455] altera_ilc f9001100.ILC: Failed to register interrupt handler
[    4.829378] altera_ilc f9001100.ILC: Failed to register interrupt handler
[    4.836429] altera_ilc f9001100.ILC: Driver successfully loaded