Overview

This project demonstrates how to access HPS Memory via the FPGA to HPS bridges when the HPS is running Linux, but is also applicable to other OS’s. It includes the effect of:

• The memory allocation method used in Linux which sets the memory attributes including security and privilege
• The interaction with Arm® AMBA® AXI™ Side Band Signal settings required for secure, privileged, and cached transfers from the FPGA.
• The Firewall settings for the HPS bridges, which can override AXI transaction attributes

This example allocates memory using two methods:

1. Manually reserved memory at Kernel boot time

• Allocation at Kernel boot time using the Kernel boot argument mem=1g to limit the Linux kernel to the lower 1GB (of 2GB) SDRAM
• This creates a static 1G manually reserved area
• This area will typically be marked as Privileged and non-secure, but this is implementation dependant

2. Runtime allocation from the Contiguous Memory Allocator Area.

• Kernel allocated memory and is an applicable approach for Kernel drivers and dynamic allocation of DMA Memory and the Linux Kernel’s DMA API.
• An open source 3rd party driver is used to allocate this memory.
• This memory will typically be marked as privileged as it has been kernel allocated, and may be marked as non-secure.

For further reading:

• Linux Kernel DMA API, and CMA documentation can be found in the Linux Kernel Documentation.
• Information on Memory Attributes in Arm® architectures can be found in the Arm Architecture documentation available on arm.com.

FPGA Firewall settings The current Yocto u-boot recipe for Intel Agilex has a patch that modifies the fpga2soc bit on the soc_noc_fw_mpfe_csr_inst_0_mpfe_scr register. This allows non-secure access through F2S bridge. For more detail see : Intel® Agilex™ Hard Processor System Address Map and Register Definitions

High Level Diagram :

AMBA ACE-LITE Signaling : The attributes of the AXI transaction from the FPGA are controlled by the Intel Cache Coherency Translator IP The AXUSER signal is used to steer transactions either directly to the SDRAM or via the HPS Cache Coherency Unit (CCU)

 ACE-LITE Signaling: 
   #[1:0]     - ardomain    
   #[3:2]     - arbar        
   #[7:4]     - arsnoop     
   #[1:8]   - arcache      
   #[13:12] - awdomain      
   #[15:14] - awbar        
   #[18:16] - awsnoop       
   #[22:19] - awcache       
   #[28:23] - axuser[7:2]    
   #[31:29] - axprot        

For further reading:

• The AMBA® AXI™ and ACE™ Protocol Specification available form arm.com contains more information on the AXI Side band signals.
• The Bridges chapter of the Intel® Agilex™ Hard Processor System Technical Reference Manual contains information and examples bridge settings for AXI Side Band Signals.

The demonstration script runs 4 tests on via a simple bash script. All the tests are run using a DMA in the FPGA fabric, controlled from the HPS.

• 2 sets of Read and Write Buffers are created for the DMA to use
• One set of Buffers is created in the manually reserved memory region
• One set of Buffers is created in the Contiguous Memory Allocator (CMA) Area
• The Read Buffers are populated with test data from user space prior to the test running.
• See Test Script for full detail on AXI Attribute settings for each test.

The Intel Cache Coherency Translator is set prior to each test, to set the transaction routing (SDRAM Direct or via CCU) and the AXI Transaction attributes.

Tests

1. SDRAM Direct : AXI attributes set to non-cached, and privileged access.
   • Configure the DMA to Read, and then Write buffers in the manually reserved area of SDRAM
   • Read back the buffer from user space to confirm the content
   • Note: This test should fail due to Non-secure transactions being blocked by the FPGA Firewall.
2. SDRAM Direct : AXI attributes set to non-cached, and privileged and secure access.
   • Configure the DMA to Read, and then Write to a buffers in the manually reserved area of SDRAM
   • Read back from user space
3. Non-cached transactions via the CCU : Privileged and secure access.
   • Configure the DMA to Read, and then Write buffers in the CMA reserved area in non cached mode
   • Read back from user space
   • This test can be edited to use the manually reserved area of SDRAM
4. Cached transactions via the CCU : Privileged and secure access.
   • Configure the DMA to Read, and then Write buffers in the CMA reserved area in cached mode
   • Read back from user space
   • This test can be edited to use the manually reserved area of SDRAM

Prerequisites

• Agilex Intel SoC FPGA Development Kit
  • Mini USB cable for serial output
  • USB cable for on-board Intel FPGA Download Cable II
  • SD/MMC HPS Daughtercard
  • Micro SD card.

• Host PC
  • Linux distribution with kernel-headers/ kernel-devel and Binutils packages properly installed.
  • Intel Quartus Prime Pro Edition v22.1
  • Serial terminal (for example Minicom on Linux and TeraTerm or PuTTY on Windows)

Building the Example

Set up

Let's create a folder to store everything:

sudo rm -rf agilex_gsrd
mkdir agilex_gsrd
cd agilex_gsrd
export TOP_FOLDER=$(pwd) 

Download the compiler toolchain, add it to the PATH variable, to be used by the GHRD makefile to build the HPS Debug FSBL:

cd $TOP_FOLDER
wget https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/\
gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz
tar xf gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz
rm -f gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz
export PATH=`pwd`/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/bin:$PATH
export ARCH=arm64
export CROSS_COMPILE=aarch64-none-linux-gnu-

Building GHRD

This project requires the following IPs:

• Intel Cache Coherency Translator CCU's Overview
• Modular Scatter-Gather DMA mSGDMA's Overview

Enable Quartus tools to be called from command line:

export QUARTUS_ROOTDIR=~/intelFPGA_pro/24.1/quartus/
export PATH=$QUARTUS_ROOTDIR/bin:$QUARTUS_ROOTDIR/linux64:$QUARTUS_ROOTDIR/../qsys/bin:$PATH

Download and build GHRD as it comes by default:

cd $TOP_FOLDER
rm -rf ghrd-socfpga agilex_soc_devkit_ghrd
git clone -b QPDS24.1_REL_GSRD_PR https://github.com/altera-opensource/ghrd-socfpga
mv ghrd-socfpga/agilex_soc_devkit_ghrd .
rm -rf ghrd-socfpga
cd agilex_soc_devkit_ghrd
# disable sgmii and partial reconfiguration - to decrease build time
export HPS_ENABLE_SGMII=0
export ENABLE_PARTIAL_RECONFIGURATION=0
make scrub_clean_all
make generate_from_tcl
unset HPS_ENABLE_SGMII
unset ENABLE_PARTIAL_RECONFIGURATION
cd software/hps_debug/
make
cd ../../
quartus ghrd_agfb014r24b2e2v.qpf &

This should have generated the design and opened it with quartus. Now we go to files and open qsys_top.qsys to be:

Remove connection fromm agilex_hps.f2h_axi_slave → jtg_mast hps_m_master as follows:

Right-click on agilex_hps name and select "Edit Parameters" then configure it as follows and click on finish:

• FPGA to HPS slave interface
  • Interface specification → ACE-lite
  • Enable/Data width → 128-bit
  • Interface destination → custom
    

Search and add Intel Cache Coherency Translator IP then configure it as follows:

• General
  • CONTROL_INTERFACE → CSR
  • Address Width → 32
  • Data Width → 128
  • AXM ID Width → 5
  • AXM ID Width → 5
• ACE-Lite Transaction Control for read Channel
  • ARCACHE_OVERRIDE_EN → Enable
• ACE-Lite Transaction Control for Write Channel
  • ARCACHE_OVERRIDE_EN → Enable
• User Selection
  • AxPROT_OVERRIDE_EN → Enable

Add Modular Scatter-Gather DMA IP with the following configurations:

• DMA Settings
  • Data Width → 128
  • Data Path FIFO Depth → 512
  • Descriptor FIFO Depth → 64
  • Response Port → Memory-Mapped
  • Maximum Transfer Length → 4KB
  • Burst Enable
  • Maximum Burst Count → 64
• Extended Feature Options
  • Enable Extended Feature Support
  • Programmable Burst Enable
• Pre-Fetching Options
  • Data Width of Descriptor read/write master data path → 128

Move Intel Cache Coherency Translator just below agilex_hps with the blue arrow buttons and then connect it as follows:

• Clock → clk_100.out_clk
• reset → rst_in.out_reset
• csr_clock → clk_100.out_clk
• csr_reset → rst_in.out_reset
• csr → agilex_hps.h2f_lw_axi_master
• m0 → agilex_hps.f2h_axi_slave
• s0 → jtg_mst.hps_m_master

Move Modular Scatter-Gather DMA below Intel Cache Coherency Translator (Module name should be msgdma_0) and connect it as follows:

• Clock → clk_100.out_clk
• reset → rst_in.out_reset
• csr → agilex_hps.h2f_lw_axi_master
• descriptor_slave → agilex_hps.h2f_lw_axi_master
• response → agilex_hps.h2f_lw_axi_master
• mm_read → intel_cache_coherency_translator_0.s0
• mm_write → intel_cache_coherency_translator_0.s0

Use the padlock icon on the "Base" column to lock all the base addresses with Intel Cache Coherency translator and Modular Scatter-Gather DMA as the only exceptions. After that go to system → Assign Base Addresses and click it. Once the process is finished If you go to Address Map tab it should look similar to this:

If everything is ok. There shouldn't be any error on the system Messages tray and you can click on generate HDL now. Finally close Platform Designer and start building on Quartus by either clicking or pressing CTL-L.

Now let's create the debug sof necessary for later steps:

quartus_pfg -c output_files/ghrd_agfb014r24b2e2v.sof ghrd_agfb014r24b2e2v_hps_debug.sof -o hps_path=software/hps_debug/hps_debug.ihex
mv ghrd_agfb014r24b2e2v_hps_debug.sof output_files/
cd $TOP_FOLDER

The following files are created:

• $TOP_FOLDER/agilex_soc_devkit_ghrd/output_files/ghrd_agfb014r24b2e2v.sof - FPGA configuration file, without HPS FSBL
• $TOP_FOLDER/agilex_soc_devkit_ghrd/software/hps_debug/hps_debug.ihex - HPS Debug FSBL
• $TOP_FOLDER/agilex_soc_devkit_ghrd/output_files/ghrd_agfb014r24b2e2v_hps_debug.sof - FPGA configuration file, with HPS Debug FSBL

Create the Core RBF file to be used in the rootfs created by Yocto by using the HPS Debug SOF built by the GHRD makefile:

cd $TOP_FOLDER
rm -f *jic* *rbf*
  quartus_pfg -c agilex_soc_devkit_ghrd/output_files/ghrd_agfb014r24b2e2v_hps_debug.sof \
  ghrd_agfb014r24b2e2v.jic \
  -o device=MT25QU02G \
  -o flash_loader=AGFB014R24B2E2V \
  -o mode=ASX4 \
  -o hps=1
rm ghrd_agfb014r24b2e2v.hps.jic

Note we are also creating an HPS JIC file, but we are discarding it, as it has the HPS Debug FSBL, while the final image needs to have the U-Boot SPL created by the Yocto recipes.

Setting Up Yocto Build System

1. First, make sure you have Yocto system requirements met: https://docs.yoctoproject.org/3.4.1/ref-manual/system-requirements.html#supported-linux-distributions.

The command to install the required packages on Ubuntu 20.04 is:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install openssh-server mc libgmp3-dev libmpc-dev gawk wget git diffstat unzip texinfo gcc \
build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping \
python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev pylint3 xterm python3-subunit mesa-common-dev zstd \
liblz4-tool git fakeroot build-essential ncurses-dev xz-utils libssl-dev bc flex libelf-dev bison xinetd \
tftpd tftp nfs-kernel-server libncurses5 libc6-i386 libstdc++6:i386 libgcc++1:i386 lib32z1 \
device-tree-compiler curl mtd-utils u-boot-tools net-tools swig -y

2. Clone the Yocto script and prepare the build:

cd $TOP_FOLDER
rm -rf gsrd_socfpga
git clone -b nanbield https://github.com/altera-opensource/gsrd_socfpga
cd gsrd_socfpga
. agilex7_dk_si_agf014ea-gsrd-build.sh
build_setup

Note: Run the following commands to set up again the yocto build environments, if you closed the current window (for example when rebooting the Linux host) and want to resume the next steps:

cd $TOP_FOLDER/gsrd_socfpga
. ./poky/oe-init-build-env agilex7_dk_si_agf014ea-gsrd-rootfs/

Adding our custom .rbf

1. Copy the rebuilt files to $WORKSPACE/meta-intel-fpga-refdes/recipes-bsp/ghrd/files using the following names, as expected by the yocto recipes:

• agilex7_dk_si_agf014ea_gsrd_ghrd.core.rbf

In our case we just copy the core.ghrd file in the Yocto recipe location:

CORE_RBF=$WORKSPACE/meta-intel-fpga-refdes/recipes-bsp/ghrd/files/agilex7_dk_si_agf014ea_gsrd_ghrd.core.rbf
ln -s $TOP_FOLDER/ghrd_agfb014r24b2e2v.core.rbf $CORE_RBF

2. In the Yocto recipe $WORKSPACE/meta-intel-fpga-refdes/recipes-bsp/ghrd/hw-ref-design.bb modify the agilex_gsrd_code file location:

SRC_URI:agilex7_dk_si_agf014ea ?= "\
      ${GHRD_REPO}/agilex7_dk_si_agf014ea_gsrd_${ARM64_GHRD_CORE_RBF};name=agilex7_dk_si_agf014ea_gsrd_core \
      ${GHRD_REPO}/agilex7_dk_si_agf014ea_nand_${ARM64_GHRD_CORE_RBF};name=agilex7_dk_si_agf014ea_nand_core \
      ${GHRD_REPO}/agilex7_dk_si_agf014ea_pr_${ARM64_GHRD_CORE_RBF};name=agilex7_dk_si_agf014ea_pr_core \
      ${GHRD_REPO}/agilex7_dk_si_agf014ea_pr_persona0.rbf;name=agilex7_dk_si_agf014ea_pr_persona0 \
      ${GHRD_REPO}/agilex7_dk_si_agf014ea_pr_persona1.rbf;name=agilex7_dk_si_agf014ea_pr_persona1 \
      "

to look like this:

SRC_URI:agilex7_dk_si_agf014ea ?= "\
      file://agilex7_dk_si_agf014ea_gsrd_ghrd.core.rbf;sha256sum=xxxxxxxxx \
      ${GHRD_REPO}/agilex7_dk_si_agf014ea_nand_${ARM64_GHRD_CORE_RBF};name=agilex7_dk_si_agf014ea_nand_core \
      ${GHRD_REPO}/agilex7_dk_si_agf014ea_pr_${ARM64_GHRD_CORE_RBF};name=agilex7_dk_si_agf014ea_pr_core \
      ${GHRD_REPO}/agilex7_dk_si_agf014ea_pr_persona0.rbf;name=agilex7_dk_si_agf014ea_pr_persona0 \
      ${GHRD_REPO}/agilex7_dk_si_agf014ea_pr_persona1.rbf;name=agilex7_dk_si_agf014ea_pr_persona1 \
      "

using the following commands:

OLD_URI="\${GHRD_REPO}\/agilex7_dk_si_agf014ea_gsrd_\${ARM64_GHRD_CORE_RBF};name=agilex7_dk_si_agf014ea_gsrd_core"
CORE_SHA=$(sha256sum $CORE_RBF | cut -f1 -d" ")
NEW_URI="file:\/\/agilex7_dk_si_agf014ea_gsrd_ghrd.core.rbf;sha256sum=$CORE_SHA"
sed -i "s/$OLD_URI/$NEW_URI/g" $WORKSPACE/meta-intel-fpga-refdes/recipes-bsp/ghrd/hw-ref-design.bb

3. In the same Yocto recipe delete the old SHA256 checksum for the file:

SRC_URI[name=agilex7_dk_si_agf014ea_pr_core.sha256sum] = "69245470a36274f1f002cb275748664902d9262f577cae4bba92813cb48fdeaf"

by using the following commands:

sed -i "/agilex7_dk_si_agf014ea_gsrd_core\.sha256sum/d" $WORKSPACE/meta-intel-fpga-refdes/recipes-bsp/ghrd/hw-ref-design.bb

Setting kernel configurations and device tree changes

We need to set the following kernel configurations:

1. CONFIG_OF_UNITTEST=y
2. CONFIG_ALTERA_SYSID=y
3. CONFIG_ALTERA_MSGDMA=y
4. CONFIG_FPGA_BRIDGE=y
5. CONFIG_ALTERA_FREEZE_BRIDGE=y
6. CONFIG_FPGA_REGION=y
7. CONFIG_OF_FPGA_REGION=y
8. CONFIG_GDB_SCRIPTS=y
9. CONFIG_DEBUG_PAGEALLOC=y
10. CONFIG_DEBUG_PAGEALLOC_ENABLE_DEFAULT=y
11. CONFIG_PAGE_OWNER=y
12. CONFIG_PTDUMP_DEBUGFS=y
13. CONFIG_DEBUG_MEMORY_INIT=y
14. CONFIG_DEBUG_PER_CPU_MAPS=y
15. CONFIG_DEBUG_INFO
16. CONFIG_DEBUG_KOBJECT=y
17. CONFIG_STRICT_DEVMEM=n​

Luckily we don't need to add them manually. Yocto allows to add fragments and make the process easier. We can do this by executing the following commands:

#Lets add the fragment file into sgmi.scc
echo "kconf hardware bridgesfragment.cfg" >> $WORKSPACE/meta-intel-fpga-refdes/recipes-kernel/linux/linux-socfpga-lts/sgmii.scc

#Download the kernel fragment.
wget https://www.rocketboards.org/foswiki/pub/Projects/SettingUpAndUsingBridgesOnAgilex/bridgesfragment.cfg -P $WORKSPACE/meta-intel-fpga-refdes/recipes-kernel/linux/linux-socfpga-lts

It is also necessary to add the following changes to arch/arm64/boot/dts/intel/socfpga_agilex_socdk.dts to define how the memory will be allocated by udmabuf:

 reserved-memory {
                #address-cells = <2>; 
                #size-cells = <2>;

                dma0_mem:dma0@0x10000000 {
                 compatible= "shared-dma-pool";
                 reusable;
                 reg=<0x0  0x10000000 0x0 0x20000000>;
                 label="dma0_mem";

         };  

        };  
        soc {
                clocks {
                        osc1 {
                                clock-frequency = <25000000>;
                        };  
                };  
                udmabuf0: udmabuf@0x10000000 {
                  compatible = "ikwzm,u-dma-buf";
                  device-name = "udmabuf0";
                  minor-number = <0>;
                  size = <0x20000000>;
                  memory-region = <&dma0_mem>;
                  dma-coherent = <1>;
          };  
        };  

To do this we need to add a previously defined .patch inside the Yocto kernel recipe by executing the following commands:

#Lets add the patch file 
FILENAME=$WORKSPACE/meta-intel-fpga-refdes/recipes-kernel/linux/linux-socfpga-lts_%.bbappend
awk '/file:\/\/fit_kernel_agilex7_dk_si_agf014ea.its/ {sub(/"$/, " file://socfpga_agilex_socdk.dts.patch\"")} 1' $FILENAME > temp && mv temp $FILENAME
#Download the Device tree patch.
wget https://www.rocketboards.org/foswiki/pub/Projects/SettingUpAndUsingBridgesOnAgilex/socfpga_agilex_socdk.dts.patch -P $WORKSPACE/meta-intel-fpga-refdes/recipes-kernel/linux/linux-socfpga-lts/

Finally we'll add the out of tree udma-buf kernel driver into our main rootfs with the following commands:

devtool create-workspace buff
devtool add https://github.com/ikwzm/udmabuf
bitbake-layers create-layer $WORKSPACE/meta-bridges
bitbake-layers add-layer $WORKSPACE/meta-bridges
devtool finish udmabuf  $WORKSPACE/meta-bridges
echo 'CORE_IMAGE_EXTRA_INSTALL+="udmabuf"' >> $WORKSPACE/agilex7_dk_si_agf014ea-gsrd-rootfs/conf/local.conf

Setting propper fpga2soc permissions

Current Yocto's u-boot recipe has a patch that modify the fpga2soc bit inside soc_noc_fw_mpfe_csr_inst_0_mpfe_scr register allowing non-secure access through F2S bridge. Due to demonstration purposes we to need remove that patch and set the U-Boot to its default state by executing the following command:

 sed -i '21,30d'  $WORKSPACE/meta-intel-fpga-refdes/recipes-bsp/u-boot/files/0001-HSD-14015655818-arm-dts-soc64-agilex-Enable-F2SDRAM-.patch

Configuring U-boot script

We need to reserve memory buffer by updating bootargs in

$WORKSPACE/meta-intel-fpga-refdes/recipes-bsp/u-boot/files/uboot.txt

according to the following example:

echo "Trying to boot Linux from device ${target}";

if test ${target} = "mmc0"; then
        if test -e ${devtype} ${devnum}:${distro_bootpart} /${bootfile}; then
                echo "Found kernel in mmc0";
                mmc rescan;
                fatload ${devtype} ${devnum}:${distro_bootpart} ${loadaddr} ${bootfile};
                setenv bootargs "earlycon panic=-1 root=${mmcroot} rw rootwait";
                bootm ${loadaddr}#board-${board_id};
                exit;
        fi
fi

if test ${target} = "qspi"; then
        sf probe;
        sf read ${loadaddr} ${qspibootimageaddr} ${bootimagesize};
        setenv bootargs "earlycon root=/dev/mtdblock1 rw rootfstype=jffs2 rootwait";
        bootm ${loadaddr}#board-${board_id};
fi

if test ${target} = "nand"; then
        ubi part root;
        ubi readvol ${loadaddr} kernel;
        setenv bootargs "earlycon panic=-1 root=${nandroot} rw rootwait rootfstype=ubifs ubi.mtd=1";
        bootm ${loadaddr}#board-${board_id};
fi

to be

echo "Trying to boot Linux from device ${target}";

if test ${target} = "mmc0"; then
        if test -e ${devtype} ${devnum}:${distro_bootpart} /${bootfile}; then
                echo "Found kernel in mmc0";
                mmc rescan;
                fatload ${devtype} ${devnum}:${distro_bootpart} ${loadaddr} ${bootfile};
                setenv bootargs "earlycon panic=-1 root=${mmcroot} rw rootwait cma=230M mem=1G​";
                bootm ${loadaddr}#board-${board_id};
                exit;
        fi
fi

if test ${target} = "qspi"; then
        sf probe;
        sf read ${loadaddr} ${qspibootimageaddr} ${bootimagesize};
        setenv bootargs "earlycon root=/dev/mtdblock1 rw rootfstype=jffs2 rootwait cma=230M mem=1G​";
        bootm ${loadaddr}#board-${board_id};
fi

if test ${target} = "nand"; then
        ubi part root;
        ubi readvol ${loadaddr} kernel;
        setenv bootargs "earlycon panic=-1 root=${nandroot} rw rootwait cma=230M mem=1G​ rootfstype=ubifs ubi.mtd=1";
        bootm ${loadaddr}#board-${board_id};
fi

This can be achieved by executing the following command:

OLD_BOOTARGS="rootwait"
NEW_BOOTARGS="rootwait cma=230M mem=1G​"
sed  -i "s/$OLD_BOOTARGS/$NEW_BOOTARGS/g" $WORKSPACE/meta-intel-fpga-refdes/recipes-bsp/u-boot/files/uboot.txt
sed -i "s/$(echo -ne '\u200b')//g" $WORKSPACE/meta-intel-fpga-refdes/recipes-bsp/u-boot/files/uboot.txt

Building Linux rootfs

Now that we are setup we have to execute the following command to start building:

cd $WORKSPACE
bitbake_image
package

Once the build is completed successfully, you will see the following two folders are created:

• agilex7_dk_si_agf014ea-gsrd-rootfs: area used by OpenEmbedded build system for builds. Description of build directory structure - https://docs.yoctoproject.org/ref-manual/structure.html#the-build-directory-build
• agilex7_dk_si_agf014ea-gsrd-images: the build script copies here relevant files built by Yocto from the agilex7_dk_si_agf014ea-gsrd-rootfs/tmp/deploy/images/agilex7_dk_si_agf014ea folder, but also other relevant files.

The two most relevant files created in the $TOP_FOLDER/gsrd_socfpga/agilex7_dk_si_agf014ea-gsrd-images folder are:

File	Description
sdimage.tar.gz	SD Card Image
u-boot-agilex-socdk-gsrd-atf/u-boot-spl-dtb.hex	U-Boot SPL Hex file

Building QSPI Flash Image

The QSPI image will contain the FPGA configuration data and the HPS FSBL and it can be built using the following command:

cd $TOP_FOLDER
rm -f *jic* *rbf*
 quartus_pfg -c agilex_soc_devkit_ghrd/output_files/ghrd_agfb014r24b2e2v.sof \
  ghrd_agfb014r24b2e2v.jic \
  -o hps_path=gsrd_socfpga/agilex7_dk_si_agf014ea-gsrd-images/u-boot-agilex7-socdk-gsrd-atf/u-boot-spl-dtb.hex \
  -o device=MT25QU02G \
  -o flash_loader=AGFB014R24B2E2V \
  -o mode=ASX4 \
  -o hps=1

The following files will be created:

• $TOP_FOLDER/ghrd_agfb014r24b2e2v.hps.jic - Flash image for HPS First configuration bitstream, phase 1: HPS and DDR
• $TOP_FOLDER/ghrd.core.rbf - HPS First configuration bitstream, phase 2: FPGA fabric, discarded, as we already have it on the SD card

Running the Example

You can download the sample script from dmaTest_cache_v1.0.run by executing:

wget https://www.rocketboards.org/foswiki/pub/Projects/SettingUpAndUsingBridgesOnAgilex/dmaTest_cache_v1.0.run

This script can be transferred to the target board.

One of the first very thing you should take care about is the base addresses. If yours are different from the following:

Then you should modify the addresses specified in the following part of the escript where:

1. intel_cache_coherency_translator_0.csr → icct_csr
2. msgdma_0.csr → msgdma_0.csr
3. msgdma_0.descriptor_slave → msgdma_descriptorlave
4. msgadma_0.response → msgdma_response

 68 #Bridges pointers
 69 icct_csr=0xf9000010
 70 msgdma_csr=0xf9000040
 71 msgdma_descriptorslave=0xf9000020
 72 msgda_response=0xf9000008

if everything is set up you can see how to run the script by executing this in the following way:

root@agilex:~# bash dmaTest_cache_v1.0.run -h
Usage: dmaTest_cache_v1.0.run options
                Options:
                        --help, -h
                        --privilaged-only, -p
                        --privilaged-secure, -s
                        --ccu-uncached, -u
                        --ccu-cached, -c

Now lets execute secure test:

 bash dmaTest_cache_v1.0.run -s 
read sysID perif,  should read back 0xACD5CAFE
/dev/mem opened.
Memory mapped at address 0xffff96e6f000.
Read at address  0xF9000000 (0xffff96e6f000): 0xACD5CAFE
Setup AXI Security Bridge via CSR H2F_LW + 0x8
Read ACE-LITE Bridge Mode
/dev/mem opened.
Memory mapped at address 0xffffb13b5000.
Read at address  0xF9000008 (0xffffb13b5008): 0x3C108208
########################################
2 SDRAM Direct Secure Protected 
Setup AXI Security Bridge via CSR H2F_LW + 0x8
Set ACE-LITE Bridge to SDRAM Direct Mode  : 0x3C108208
/dev/mem opened.
Memory mapped at address 0xffff94fc7000.
Read at address  0xF9000008 (0xffff94fc7008): 0x3C108208
Write at address 0xF9000008 (0xffff94fc7008): 0x3C108208, readback 0x3C108208
########################################
########################################
########################################
Read ACE-LITE Bridge Mode
/dev/mem opened.
Memory mapped at address 0xffffa5af9000.
Read at address  0xF9000008 (0xffffa5af9008): 0x3C108208
########################################
Creating populated data buffer at 0x40001000 in unused memory (above 1GB) 
Check data by reading 1st address
/dev/mem opened.
Memory mapped at address 0xffff97b39000.
Read at address  0x40001000 (0xffff97b39000): 0xFEAD0000
########################################
Creating populated data buffer at in CMA reserved cached memory at 0x10002000 from userspace
Check data by reading 1st address
/dev/mem opened.
Memory mapped at address 0xffff8b029000.
Read at address  0x10002000 (0xffff8b029000): 0xEEEEEEEE
########################################
finished data writes
########################################
########################################
msgdma : Stop dispatcher
/dev/mem opened.
Memory mapped at address 0xffff9377e000.
Read at address  0xF9000044 (0xffff9377e044): 0x00000000
Write at address 0xF9000044 (0xffff9377e044): 0x00000001, readback 0x00000001
msgdma : Stop descriptors
/dev/mem opened.
Memory mapped at address 0xffff8fb14000.
Read at address  0xF9000044 (0xffff8fb14044): 0x00000001
Write at address 0xF9000044 (0xffff8fb14044): 0x00000020, readback 0x00000020
msgdma : Reset Dispatcher
/dev/mem opened.
Memory mapped at address 0xffffa72e3000.
Read at address  0xF9000044 (0xffffa72e3044): 0x00000020
Write at address 0xF9000044 (0xffffa72e3044): 0x00000002, readback 0x00000002
msgdma : Read address,  lower 32bits, upper 32bits : 0x0000000040001000
/dev/mem opened.
Memory mapped at address 0xffffb6dba000.
Read at address  0xF9000020 (0xffffb6dba020): 0x00000000
Write at address 0xF9000020 (0xffffb6dba020): 0x40001000, readback 0x40001000
/dev/mem opened.
Memory mapped at address 0xffffbb061000.
Read at address  0xF9000034 (0xffffbb061034): 0x00000000
Write at address 0xF9000034 (0xffffbb061034): 0x00000000, readback 0x00000000
msgdma : Write address, lower 32bits, upper 32bits : 0x0000000040000000
/dev/mem opened.
Memory mapped at address 0xffff9b65a000.
Read at address  0xF9000024 (0xffff9b65a024): 0x00000000
Write at address 0xF9000024 (0xffff9b65a024): 0x40000000, readback 0x40000000
/dev/mem opened.
Memory mapped at address 0xffff9128a000.
Read at address  0xF9000038 (0xffff9128a038): 0x00000000
Write at address 0xF9000038 (0xffff9128a038): 0x00000000, readback 0x00000000
msgdma : length in bytes 4068,  128bit x 32
/dev/mem opened.
Memory mapped at address 0xffff839d3000.
Read at address  0xF9000028 (0xffff839d3028): 0x00000000
Write at address 0xF9000028 (0xffff839d3028): 0x00000200, readback 0x00000200
msgdma : Burst counts 32, sequence number = 1
/dev/mem opened.
Memory mapped at address 0xffffa755b000.
Read at address  0xF900002C (0xffffa755b02c): 0x00000000
Write at address 0xF900002C (0xffffa755b02c): 0x20200001, readback 0x20200001
msgdma : Check Status
/dev/mem opened.
Memory mapped at address 0xffffaf25e000.
Read at address  0xF9000040 (0xffffaf25e040): 0x0000000A
msgdma : Control,  wait for resp
/dev/mem opened.
Memory mapped at address 0xffffbe1ff000.
Read at address  0xF900003C (0xffffbe1ff03c): 0x00000000
Write at address 0xF900003C (0xffffbe1ff03c): 0x02000000, readback 0x02000000
########################################
########################################
########################################
########################################
msgdma : Control,  wait for resp and go
/dev/mem opened.
Memory mapped at address 0xffffac92f000.
Read at address  0xF900003C (0xffffac92f03c): 0x00000000
Write at address 0xF900003C (0xffffac92f03c): 0x82000000, readback 0x82000000
########################################
msgdma : Check Status
/dev/mem opened.
Memory mapped at address 0xffffb6c58000.
Read at address  0xF9000040 (0xffffb6c58040): 0x00000002
ReadBack from 0x0000000040000000
/dev/mem opened.
Memory mapped at address 0xffff8ecff000.
Read at address  0x40000000 (0xffff8ecff000): 0xFEAD0000
/dev/mem opened.
Memory mapped at address 0xffffac213000.
Read at address  0x40000004 (0xffffac213004): 0xFEAD0004
/dev/mem opened.
Memory mapped at address 0xffffb6fd6000.
Read at address  0x40000008 (0xffffb6fd6008): 0xFEAD0008
/dev/mem opened.
Memory mapped at address 0xffffbdc8c000.
Read at address  0x4000000C (0xffffbdc8c00c): 0xFEAD000C
/dev/mem opened.
Memory mapped at address 0xffffa36ae000.
Read at address  0x40000010 (0xffffa36ae010): 0xFEAD0010