Intel Agilex® 7 FPGA F-Series PCIe Root Port Reference Design

Summary

Required Components

• Root Port Host Board
  • Intel Agilex® 7 FPGA F-Series FPGA Transceiver-SoC Development Kit
• Example End Points
  • NVMe
• Pre-compiled Software/Firmware
• Tools and software
  • Linux Development Computer (Ubuntu, CentOS, or similar) with a SD Card reader
    • Make sure you have the yum resources i.e.:
      • yum groupinstall "Development tools"
      • yum install ncurses ncurses-devel
      • yum install uboot-tools
      • gcc version 5 or later
  • Intel® Quartus® Prime Design Suite software versions 23.3
  • A serial terminal application, such as Minicom, Tera Term or Putty

Helpful Reference Documentation

• MCDMA
• Intel Agilex® 7 FPGA F-Series FPGA Transceiver-SoC Development Kit
• P-tile Avalon® Intel® FPGA IPs for PCI Express

Release Contents

Hardware Description

Root Port Design

Root Port Reference Design Hardware Block Diagram

Memory Map

HPS H2F Memory Map

HPS LWH2F Memory Map

PCIe BAM interface

Intel SSD D7 P5510 Non-Volatile Memory express (NVMe) End Point

Hardware Setup

• SW1: ON-ON-ON-ON
• SW2: All ON
• SW3: All OFF
• SW4: ON-OFF-ON-OFF

Setup Intel SSD D7 P5510 NVMe End Point

Agilex 7 F-Series SoC Development Kit with SSD End Point

Create SD Card for Agilex 7 F-Series SoC Development Kit

Creating SD card on Linux

1. Extract sdimage.tar.gz file ( after extraction gsrd-console-image-agilex.wic should be seen)

   tar xf sdimage.tar.gz 

2. Determine the device associated with the SD card on the host

   cat /proc/partitions (Let's assume it is /dev/sdx.) 

3. Use dd utility to write the SD image to the SD card

   sudo dd if=gsrd-console-image-agilex.wic of=/dev/sdx bs=1M 

   Note we are using sudo to be able to write to the card
4. Use sync utility to flush the changes to the SD card

   sudo sync 

Creating SD card on Windows

1. Download the sdimage.tar.gz listed in the Release Contents section above.
2. Run tar xvfz sdimage.tar.gz to extract the SD Card binary image (sdimage.img).
3. Insert the SD Card to be updated into an SD Card reader attached to the computer.
4. Use Win32DiskImager to write the image to the SD card.

SD Card Programming using Win32DiskImager Tool

Replace kernel.itb in SD card with the PCIe root port enable kernel.itb

1. Download the PCIe root port kernel.itb from the Release Contents.
2. Replace the kernel.itb in the SD with the PCIe root port enable kernel.itb.

Booting the System

Configuring Serial Connection

1. The virtual serial port is usually named /dev/ttyUSB0. In order to determine the device name associated with the virtual serial port on your host PC, please perform the following:
   • Use the following command to determine which USB serial devices are already installed: ls /dev/ttyUSB*
   • Connect mini USB cable from J7 to the PC. This will enable the PC to communicate with the board, even if the board is not powered yet.
   • Use the ls /dev/ttyUSB* command command again to determine which new USB serial device appeared.
2. Install minicom application on host PC, if not installed.
   • On Ubuntu, use sudo apt-get install minicom
3. Configure minicom.

   sudo minicom -s

• Serial Device: /dev/ttyUSB0 (edit to match the system as necessary)
• Bps/Par/Bits: 115200 8N1
• Hardware Flow Control: No
• Software Flow Control: No
• Hit [ESC] to return to the main configuration menu

Programming *.sof file Using JTAG Configuration

1. Please make sure switches on the Agilex 7 F-Series SoC Development Kit are configured as below.
   • SW1: ON-ON-ON-ON
   • SW2: All ON
   • SW3: All OFF
   • SW4: ON-OFF-ON-OFF
2. With the board powered down, insert the SD Card created above into the SD Card slot.
3. Connect miniUSB cable between host PC and miniUSB port (J7) on the board and connect mini USB cable between host PC and mini USB port (J7) on OOBE Daughter Card.
4. Power on the Agilex 7 SoC Development Kit and open a serial port terminal application on the host PC such as TeraTerm, Putty, Minicom on host PC with settings below.
   • Baud-rate: 115,200
   • Parity: none
   • Flow control: none
   • Stop bits: 1
5. Open Quartus Prime Programmer and select Agilex Si/SoC Dev Kit for the Hardware Setup.
6. Click Auto Detect to detect the AGILEX_HPS device in the JTAG scan chain.
7. Select the AGB014R24B device followed by selection of 10M16SA.
8. Select the AGB014R24B device from the JTAG scan chain and click on Change File.
9. Navigate to the location of the pre-built sof file and click OK.
10. Check the Program/Configure checkbox. Click Start to begin configuring the device.

Programmer configuration

Discovery of Intel SSD End Point

fio (flexible I/O tester) on NVMe

• Read fio command: fio --filename=/dev/nvme0n1 --rw=read --gtod_reduce=1 --blocksize=64k --size=2G --iodepth=2 --group_reporting --name=myjob --ioengine=libaio --numjobs= no of job
• Write fio command: fio --filename=/dev/nvme0n1 --rw=write --gtod_reduce=1 --blocksize=64k --size=2G --iodepth=2 --group_reporting --name=myjob --ioengine=libaio --numjobs= no of job

• fio --filename=/dev/nvme0n1 --rw=read --gtod_reduce=1 --blocksize=64k --size=8G --iodepth=2 --group_reporting --name=myjob --ioengine=libaio --numjobs= no of job
• fio --filename=/dev/nvme0n1 --rw=write --gtod_reduce=1 --blocksize=64k --size=8G --iodepth=2 --group_reporting --name=myjob --ioengine=libaio --numjobs= no of job

Intel SSD NVMe Write/Read Throughput

Rebuilding Source Files

Rebuild Hardware Design

1. Setting up Enviroment.
2. Building the Hardware Design.
   • You can rebuild the hardware via Intel Quartus Prime Pro GUI or CLI.
   • If you refer to Agilex 7 Golden System Reference Design (GSRD) User Manuals to rebuilding, make sure the dir path is correct when you execute the command.
3. Building Core RBF.

Rebuilding Source Files

Build Agilex PCIe Root Port and Integrating U-boot FSBL

• The project is available here
• Follow the instructions listed at  Compiling Hardware Design and Integrating U-boot FSBL to compile the hardware design and integrating the FSBL.

Compiling Linux

1. Download this patch to add PCIe root port node into socfpga_agilex_etile_4_channel.dtsi in ~/linux-socfpga/arch/arm64/boot/dts/intel/.
2. Run commands below to apply the patch and build Linux kernel and Linux device tree.

$ wget https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz 
$ tar xf gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz 
$ export PATH=`pwd`/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/bin:$PATH 
$ export ARCH=arm64 
$ export CROSS_COMPILE=aarch64-none-linux-gnu- 
$ git clone https://github.com/altera-opensource/linux-socfpga -b QPDS23.3_REL_GSRD_PR 
$ cd linux_socfpga 
$ apply patch (Ex patch -p1 < SW/6_1_38_agilex7_pcierp.patch ) 
$ make defconfig ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- 
$ make menuconfig ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu- , add CONFIG_PCIEAER and required PCI EP drivers ( Ex CONFIG_BLK_DEV_NVME for nvme storage) 

menuconfig

$ make -j`nproc` ARCH=arm64 CROSS_COMPILE=aarch64-none-linux-gnu-

• kernel image should be avaliable at "arch/arm64/boot/Image"
• dtb file should be avaliable at "arch/arm64/boot/dts/intel/socfpga_agilex_socdk.dtb"

Generating the kernel.itb

cd ..
mkdir kernel_itb
cd kernel_itb
mv file.rbf agilex_gsrd_ghrd.core.rbf
copy ../fit_kernel_agilex_pci.its and ../file.rbf

cp ../linux-socfpga/arch/arm64/boot/dts/intel/socfpga_agilex_socdk.dtb 
cp ../linux-socfpga/arch/arm64/boot/Image
xz --format=lzma Image 

mkimage -f fit_kernel_agilex_pci.its kernel.itb

Crafting Device Tree Manually for PCIe root port enablement

• Agilex SoC PCIe Root Port device tree descriptions
• PCIe controller dts node descriptions
• PCIe MSI controller dts node descriptions

soc { 

                aglx_hps_bridges: bridge@80000000 { 
                        compatible = "simple-bus"; 
                        reg = <0x80000000 0x20200000>, 
                              <0xf9000000 0x00100000>; 
                        reg-names = "axi_h2f", "axi_h2f_lw"; 
                        #address-cells = <0x2>; 
                        #size-cells = <0x1>; 
                        ranges = <0x00000000 0x00000000 0x80000000 0x00040000>, 
                                 <0x00000000 0x10000000 0x90000000 0x10000000>, 
                                 <0x00000000 0x20000000 0xa0000000 0x00200000>, 
                                 <0x00000001 0x00010000 0xf9010000 0x00008000>, 
                                 <0x00000001 0x00018000 0xf9018000 0x00000080>, 
                                 <0x00000001 0x00018080 0xf9018080 0x00000010>; 

                        pcie_0_pcie_aglx: pcie@200000000 { 
                                       compatible = "altr,pcie-root-port-3.0"; 
                                       reg = <0x00000000 0x20000000 0x00200000>, 
                                             <0x00000000 0x10000000 0x10000000>, 
                                             <0x00000001 0x00010000 0x00008000>; 
                                       reg-names = "Hip", "Txs", "Cs"; 
                                       /* p-tile 
                                       port_conf_stat = <0x104000>;*/ 
                                       /* f-tile */ 
                                       port_conf_stat = <0x14000>; 
                                       interrupt-parent = <&intc>; 
                                       interrupts = <0x0 0x14 0x4>; 
                                       interrupt-controller; 
                                       #interrupt-cells = <0x1>; 
                                       device_type = "pci"; 
                                       bus-range = <0x0000000 0x000000ff>; 
                                       ranges = <0x82000000 0x00000000 0x00000000 0x00000000 0x10000000 0x00000000 0x10000000>; 
                                       msi-parent = <&pcie_0_msi_irq>; 
                                       #address-cells = <0x3>; 
                                       #size-cells = <0x2>; 
                                       dma-coherent; 
                                       interrupt-map-mask = <0x0 0x0 0x0 0x7>; 
                                       interrupt-map = <0x0 0x0 0x0 0x1 &pcie_0_pcie_aglx 0x1>, 
                                                      <0x0 0x0 0x0 0x2 &pcie_0_pcie_aglx 0x2>, 
                                                      <0x0 0x0 0x0 0x3 &pcie_0_pcie_aglx 0x3>, 
                                                      <0x0 0x0 0x0 0x4 &pcie_0_pcie_aglx 0x4>; 
                                       }; //end pcie@0x010000000 (pcie_0_pcie_aglx) 

                          pcie_0_msi_irq: msi@10008080 { 
                                       compatible = "altr,msi-1.0"; 
                                       reg = <0x00000001 0x00018080 0x00000010>, 
                                             <0x00000001 0x00018000 0x00000080>; 
                                       reg-names = "csr", "vector_slave"; 
                                       interrupt-parent = <&intc>; 
                                       interrupts = <0x0 0x13 0x4>; 
                                       msi-controller = <0x1>; 
                                       num-vectors = <0x20>; 
                                       }; //end msi@0x100008000 (pcie_0_msi_irq) 
                        }; 
	}; 
}; 

Creating and Updating SD Card