This is a sustaining user manual for Stratix 10 SoC Design Example for 10Gbe with IEEE1588 PTP Capability starting with GHRD ver. 21.4. Hardware and software buildflow are provided to guide you to integrate QSE IP block to hardware design and build the relative compatible Kernel version. You may refer to release content for available prebuild binaries.

Related information

Existing version:

• Stratix 10 SoC Design Example for 10Gbe with IEEE1588 PTP Capability v21.2 (v21.2 - stable version)
• Stratix 10 SoC Design Example for 10Gbe with IEEE1588 PTP Capability (v20.1 - closed)

For design example update flow, please refer to below section:

• HW/SW Version Compatibility
• Building S10 Hardware Design with 10Gbe QSE
• Building Linux with QSE Driver

Introduction

IEEE 1588 v2 is a protocol that enables precise synchronization of all real time-of-day (ToD) clocks in a network to a master clock. Diagram below illustrates how PTP synchronization can be achieved by exchanging of timestamp packets between master and slave nodes. The master node will initiate the handshaking by sending a Sync packet. When all 4 t1, t2, t3, t4 timestamps are gathered, slave node can calculate and adjust its time-of-day to achieve synchronization with master node. This handshake process is repeated at specified interval to ensure that slave clock is always synchronized to master clock.

Pre-requisites

This design example is a derivative from Stratix 10 SoC GSRD and it is recommended to familiarize with the development flow of Golden System Reference Design prior attending to this design example.

Overview

This design example presents an example of IEEE 1588v2 2-step FPGA implementation in Quartus Pro v21.2 using Stratix 10 SoC, Low Latency Ethernet 10G MAC with Multi-rate PHY and Linux kernel v5.4 software stack.

This design supports ordinary clock, both PTP Master and Slave mode. PTP timestamping is handled by the hardware while the HPS and software stack handles packet generation and slave ToD adjustment.

Hardware Architecture

Main design components of the hardware design are Low Latency Ethernet 10G MAC, Multi-rate Ethernet PHY, ToD and synchronizer, DMA and HPS system:

• MAC and PHY transmit and receive ethernet packets including PTP packets with timestamp;

• ToD module generates local time-of-day for TX and RX MAC;

• HPS will run Ethernet and PTP stack, include updating ToD. HPS sends and receives packets via DMA.

When PTP is initiated (see section Verify PTP capability using ptp4l tool), the HPS will be sending PTP packets (Sync, Follow-up, etc.). DMA modules will act as data mover between HPS and Ethernet MAC IP. All TX egress timestamps and RX ingress timestamps will be collected.

This design can support both PTP master and PTP slave. When it is configured as PTP master, the HPS will initiate timestamp packet exchange at a fixed interval e.g. once every second. When the design is configured as PTP slave, the HPS will calculate the offset between master and slave based on t1, t2, t3, t4 timestamps collected. It will then perform time-of-day adjustments by writing to its ToD module. The ToD module can support seconds, nanoseconds and fractional nanoseconds adjustment.

Note: In v21.2, standard MSGDMA Prefetcher IP is adopted instead of modified Prefetcher IP used in earlier design version.

Clocking Scheme

Reset Scheme

IP Settings

Tables below show the IP settings for Low Latency Ethernet 10G MAC IP and 1G/2.5G/5G/10G Multi-rate Ethernet PHY IP.

Hardware Component	IP Parameter Description
LL 10Gbe MAC	Speed: 1G/2.5G/10G Datapath options: TX & RX Enable ECC on memory blocks: Not selected Enable supplementary address: Selected Enable statistics collection: Selected Statistics counters: Memory-based TX and RX datapath Reset/Default To Enable: Selected Enable time stamping: Selected Enable PTP one-step clock support: Not selected Timestamp fingerprint width: 8 Time Of Day format*: Enable both 96b Time of Day Format only Use legacy Ethernet 10G MAC XGMII interface: selected Use legacy Ethernet 10G MAC Avalon Memory-Mapped interface: Not selected Use legacy Ethernet 10G MAC Avalon Streaming interface: selected
Multi-rate Ethernet PHY	Connect to MGBASE-T PHY: Selected Connect to NBASE-T PHY: Not selected Speed: 1G/2.5G/10G Enable SGMII bridge: Not selected Enabled IEEE 1588 Precision Time Protocol: Selected PHY ID (32 bit): 0x00000000 VCCR_GXB and VCC_GXB supply voltage for the Transceiver: 1_0V Reference clock frequency for 10GbE (MHz): 644.53125 Selected TX PMA local clock division factor for 1 GbE : 1 Selected TX PMA local clock division factor for 2.5 GbE : 1 Enable Native PHY Debug Master Endpoint: Not selected Enable capability registers: Not selected Enable control and status registers: Not selected Enable PRBS soft accumulators: Not selected

Software Architecture

Intel SoCFPGA Ethernet driver

1. QSE (MAC driver): intel_fpga_qse_ll_main.c

• Initialization/uninitialization of PHY and MAC, setup of Tx & Rx operations, interrupt routing
• Dynamic SFP module is used to support 10GbE
• PTP support - IOCTLs from application software to enable/disable/configure supported modes of timestamp operations

2. mSGDMA Prefetcher: altera_msgdma_prefetcher.c

• Implements mSGDMA Prefetcher IP core features such as setting up Tx & Rx descriptors, handles DMA transfers

3. ToD: intel_fpga_tod.c

• Registers PTP clock driver to kernel
• Calculates the ToD of clock offset adjustments

Release Content

HW/SW Version Compatibility

GHRD	Recommended Kernel/U-boot	Prebuild Binaries	Change List
21.2	linux-socfpga v5.4.114-lts u-boot-socfpga_v2021.07	link	Design based on v21.2 GHRD with ATF flow
21.2	linux-socfpga v5.10.60-lts u-boot-socfpga_v2021.07	Portable from v21.2 No changes required from 21.2 HW design Require to update kernel to 5.10.60-lts using with s10_1588_devkit_defconfig	Updated QSE driver and DTS to 10gbase-r to accomodate changes in phylink driver https://github.com/altera-opensource/linux-socfpga/commit/66e56cf745ca822fc1c051fce3a610d6656bc707 https://github.com/altera-opensource/linux-socfpga/commit/7ee168fb6f6ac0ded3a71c96aaa0036543fe08f8
21.4	linux-socfpga v5.10.100-lts u-boot-socfpga_v2021.07	link	Based on 21.4 GHRD DTS DMA register update to reflect new register mapping for Tx and Rx DMA CSR https://github.com/altera-opensource/linux-socfpga/commit/4f0ce4a842763a90d89a8650f03285720ac9cb71 https://github.com/altera-opensource/linux-socfpga/commit/89affd122a189259bbc51d34d7144108c055fd95 https://github.com/altera-opensource/linux-socfpga/commit/e6ee8aa50bee0d3683885159e5e4b30d23a2ac
22.2	linux-socfpga v5.15.50-lts u-boot-socfpga_v2022.01	link	Based on 22.2 GHRD

Release File Description

General description of release files.

File	Description
ghrd_1sx280lu2f50e2vg_hps_debug.sof	SOF file with Debug FSBL added
ghrd_1sx280lu2f50e2vg_hps.sof	SOF file with U-Boot SPL added
gsrd-console-image-stratix10.tar.gz	Compiled hardwre project (GHRD)
sdimage.img	SD card image
sdimage.tar.gz	SD card image in tar format
sdimage.tar.gz.md5sum	SD card image checksum
ghrd.core.rbf	Core fabric configuration bitstream
ghrd.hps.rbf	HPS configuration bitstream
u-boot.itb	U-Boot FIT image
u-boot-spl-dtb.hex	U-Boot SPL hex file
boot.scr.uimg	Image of U-Boot distro boot script
Image	Uncompressed kernel image
fit_kernel_stratix10.its	Kernel image FIT description file
kernel.itb	Kernel FIT image
socfpga_stratix10_socdk.dtb	Linux device tree binary

Release Branches

Component	Location
GHRD	https://github.com/altera-opensource/ghrd-socfpga/tree/master/s10_soc_devkit_ghrd
Linux	https://github.com/altera-opensource/linux-socfpga
Arm Trusted Firmware	https://github.com/arm-trusted-firmware
U-Boot	https://github.com/altera-opensource/u-boot-socfpga

System Requirements

Hardware Requirements

• PTP Master/Slave hardware:
  • 2 Stratix 10 SoC Development Kits (L-Tile) with:
    • HiLo DDR4 memory card (4 GB)
    • HPS peripheral daughter cards (OOBE)
• 2 SFP+ module and fiber optic cable
• 2 USB type A to mini USB type B cables

Software Requirements

This design example is build based on Stratix 10 SoC GSRD (Golden System Reference Design) and tested with Quartus Prime Pro (This reference design will not work on Quartus Prime Standard). It is recommended for user to setup and go through below materials before getting started with this design example.

• Quartus Prime Pro Release Notes*
• Stratix 10 GSRD*
• Linux host PC with Ubuntu 16.04 LTS (or any newer version)
• Compatible hardware/software version - refer to HW/SW version compatibility section

Note*

• Refer to desired version

Quick Start Guide

This design example leverage on default S10 SoC GSRD, with changes to Kernel Image and Device tree to enable QSE driver. There is no changes in U-boot image.

• "Release Content" section provide prebuild binary on specific build for quick testing.
• For other build version, you may refer to "Build/Update S10 10G Design" section for more details.

Flash SD Card with Prebuild Binary

1. Download and untar sdimage.tar.gz from "Release Content".

2. Flash the SD imagecontent into SD card.

Follow the instructions under "Creating SD Card on Linux/Windows" to create a boot-able SD card with this image file.

Setup Hardware Connection

1. Connect two Stratix 10 SoC Development Kits using a fiber optic cable via SFP+ port (J7 port). One of the boards will act as PTP Master while the other one as PTP Slave.

Optionally, the development kit can also be paired with X710 network adapter card to connect to a Linux machine.

2. The mini-USB port (J52) on the Stratix 10 SoC Development Kits are connected to a host machine with USB type A to mini-USB type B cables.

Hardware Bring Up

1. Configure the S10 SoC Development Kit as follows

• SW1: OFF-OFF-ON-ON-ON-ON-ON-ON
• SW2: ON-ON-ON (JTAG Configuration Mode, SW2.4 is not connected)
• SW3: All OFF
• SW4: ON-OFF-OFF-ON

2. After powering up the development kit, launch Clock Controller application, which is part of development kit package installer, to reconfigure FPGA Transceiver Reference Clocks U33. In the application, click on Si5341 tab and change the frequency of OUT2 from 135MHz to 125MHz.Then click on "Set" to configure the clock.

Note: you have to repeat this step for every board power cycle

3. Configure FPGA on the development kit with ghrd.hps.rbf.

Executing the Reference Design

Verify Link is Up

1. Linux operating system running on the Stratix 10 Soc Development Kit can be accessed using Serial Communication program such as Putty. Modify the serial line ID based on the COM port connected to the host.

2. Perform step 2-3 of Hardware bring up section (if you haven't already).

3. Type root as the login name when requested.

4. Repeat steps 1-3 for the other Stratix10 SoC Development Kit.

5. When both development kits are powered up and connected, link up status will be shown at console printout:

root@stratix10:~# [36.006812] altera_qse_ll f9020000.ethernet eth0: Link is Up - 10Gbps/Full - flow control off

Assign IP Address and Verify Connection with Ping

1. Assign IP address for both boards:

S10 #1> SOCFPGA_STRATIX10 # ifconfig eth0 192.168.0.1
S10 #2> SOCFPGA_STRATIX10 # ifconfig eth0 192.168.0.2

2. Run ifconfig command to verify if the IP address is successfully assigned for each board.

3. Use Ping command to verify connection between the two boards

S10 #1> SOCFPGA_STRATIX10 # ping 192.168.0.2

4. Press Ctrl + C to stop ping packet transmission.

Verify PTP Capability using PTP4L Tool

In this reference design, ptp4l in LinuxPTP package is used to implement PTP master and slave clock.

1. Enable hardware flow control for both boards with the following command:

SOCFPGA_STRATIX10 # ethtool -A eth0 rx on tx on

Flow control is used to prevent packet drop during high traffic.

2. Setup S10 #1 as master and S10 #2 as slave.

S10 #1> SOCFPGA_STRATIX10 # ptp4l -i eth0 -m
S10 #2> SOCFPGA_STRATIX10 # ptp4l -i eth0 -m -s

3. As the ToD of PTP slave is synchronized to the ToD of PTP master, a convergence of master offset from large to smaller number can be observed in the slave console as shown in the figure below.

PTP master:

PTP slave:

4. Press Ctrl + C on PTP slave board to stop the PTP measurements. Press Ctrl + C on PTP master board to terminate the PTP master.

Verify Bandwidth using iperf3 Tool

1. Setup S10 #1 as iperf3 master and S10 #2 as iperf3 slave.

S10 #1> SOCFPGA_STRATIX10 # iperf3 -s
S10 #2> SOCFPGA_STRATIX10 # iperf3 -c 192.168.0.1

S10#1 as iperf master

S10#2 as iperf slave

Note:

• -A is used in iperf3 to specify the CPU affinity to improve performance
• Ethernet bandwidth has dependency on CPU maximum performance and NAPI (New API) driver on Ethernet packet handling
• For performance test reference:
  • Ethernet Design Example Demo Testing
  • iperf3 command
  • NAPI performance

Build/Update S10 10G Design

Stratix 10 SoC Design Example for 10Gbe with IEEE1588 PTP Capability is a derivative from Stratix 10 GSRD. You need to follow hardwre/software build flow to build for specific GHRD/GSRD version. You are encourage to build based on recommended Stratix 10 GHRD in relative to supported GSRD branch. Please refer to release branches for more information.

Related information: HW/SW Version Compatibility

Building S10 Hardware Design with 10Gbe QSE

1. Download the desired GHRD version through Github repo.

cd $TOP_FOLDER
rm -rf  ghrd-socfpga-QPDS-22.2pro-21.1std QPDS-22.2pro-21.1std.zip s10_soc_devkit_ghrd
wget https://github.com/altera-opensource/ghrd-socfpga/archive/refs/tags/QPDS-22.2pro-21.1std.zip
unzip QPDS-22.2pro-21.1std.zip
mv ghrd-socfpga-QPDS-22.2pro-21.1std/s10_soc_devkit_ghrd .
rm -rf  ghrd-socfpga-QPDS-22.2pro-21.1std QPDS-22.2pro-21.1std.zip
cd s10_soc_devkit_ghrd

• There are multiple ways to download from Github repo
  • Download through latest GHRD master archive shows as above
  • Checkout Github GHRD repo
  • Download ZIP through Web
• Above steps downloaded every device family's GHRD. You can remove others and keep s10_soc_devkit_ghrd folder.

2. Modify ../s10_soc_devkit_ghrd/Makefile to include 10Gbe design custom settings:

• Keep default settings
• Disable SGMII, Partial Reconfiguration
• Enable 10Gbe 1588 Design

QUARTUS_DEVICE_FAMILY := Stratix\ 10
# For Silicon Rev B0, use:
#QUARTUS_DEVICE ?= 1SX280LU2F50E2VGS1
# REV Silicon Rev C1, use:
#QUARTUS_DEVICE ?= 1SX280LU2F50E2VGS3
# REV Final Production Silicon, use:
QUARTUS_DEVICE ?= 1SX280LU2F50E2VG
# FGI board device
#QUARTUS_DEVICE ?= 1SX280LU2F50E2VGS2
# Htile S10 Devkit
# QUARTUS_DEVICE ?= 1SX280HU2F50E1VGAS
# QUARTUS_DEVICE ?= 1SX280HU1F50E2VG

BOARD_TYPE ?= devkit
# Publicly Available DC: devkit_dc_oobe, devkit_dc_nand, devkit_dc_emmc
DAUGHTER_CARD ?= devkit_dc_oobe
# System initilization mode: fpga, hps
BOOTS_FIRST ?= hps
# HPS JTAG MODE: combined, separate
HPS_JTAG_MODE ?= combined
# Enable HPS EMIF ECC: 0, 1
ENABLE_HPS_EMIF_ECC ?= 1
# Enable Watchdog Reset: 0, 1
ENABLE_WATCHDOG_RST ?= 1
# Supported watchdog reset action: cold, warm, remote_update
WATCHDOG_RST_ACTION ?= cold

# Enable SGMII (1GbE, 100MbE, 10MbE) design (HPS EMAC + 1G/2.5G/5G/10G Multi-Rate Ethernet PHY Intel FPGA IP).
HPS_ENABLE_SGMII ?= 0
# Enable 10GbE 1588 Design. (Low Latency Ethernet 10G MAC Intel FPGA IP + 1G/2.5G/5G/10G Multi-Rate Ethernet PHY Intel FPGA IP). PTPv2 2-Step enabled.
HPS_ENABLE_10GbE ?= 1
# Enable PCIE Design.
ENABLE_PCIE ?= 0
# Enable Partial Reconfiguration Design.
ENABLE_PARTIAL_RECONFIGURATION ?= 0
# Enable FPGA I2C.
ENABLE_FPGA_I2C ?= 0

3. Generate and compile project with below command:

~/intelFPGA_pro/22.2/nios2eds/nios2_command_shell.sh make scrub_clean_all
~/intelFPGA_pro/22.2/nios2eds/nios2_command_shell.sh make generate_from_tcl
~/intelFPGA_pro/22.2/nios2eds/nios2_command_shell.sh make all

4. Hardware .sof file is genearte in directory: ../output/ghrd_1sx280lu2f50e2vg.sof.

Building Software Binaries

This design example leverage on default S10 SoC GSRD, with changes to Kernel Image and Device tree to enable QSE driver. There is no changes in U-boot. For a detail flow of how to generate software binaries, please refer to Stratix 10 SoC GSRD.

Building ATF/U-Boot

For a detail flow of ATF and U-boot binaries generation, please refer to Building Bootloader for Stratix 10 and Agilex.

Integrating U-boot FSBL to .sof file

1. During full compilation, the Intel Quartus Prime Pro Edition generates the data in the HPS FSBL handoff binary based on the settings inside the Platform Designer. The file contains information on:

• HPS pin multiplexing configuration, including HPS dedicated I/O settings and export to FPGA settings
• HPS clock configuration
• HPS SSBL boot source

2. Run below command to merge both .sof and FSBL:

###example: quartus_pfg -c -o hps=on -o hps_path=<hex_file> <input_sof> <output_rbf>###
$ quartus_pfg -c -o hps=on -o hps_path=software/hps_debug/u-boot-spl-dtb.hex output_files/ghrd_1sx280lu2f50e2vg.sof output_files/ghrd_1sx280lu2f50e2vg_hps.rbf

The following files are generated:

• ghrd_1sx280lu2f50e2vg.hps.rbf - HPS First configuration bitstream, phase 1: HPS and DDR
• ghrd_1sx280lu2f50e2vg.core.rbf - HPS First configuration bitstream, phase 2: FPGA fabric

Building Linux with QSE Driver

1. Setup build toolchain - compilation of the software relies on downloading the build toolchain and setting up the CROSS_COMPILE and ARCH environment variables accordingly.

$ cd $TOP_FOLDER
$ wget https://developer.arm.com/-/media/Files/downloads/gnu/11.2-2022.02/binrel/ $ gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz
$ tar xf gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz
$ rm -f gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu.tar.xz
$ export PATH=`pwd`/gcc-arm-11.2-2022.02-x86_64-aarch64-none-linux-gnu/bin:$PATH
$ export ARCH=arm64
$ export CROSS_COMPILE=aarch64-none-linux-gnu-

2. The following instructions are used to build Linux.

$ cd $TOP_FOLDER
$ rm -rf linux-socfpga
$ git clone https://github.com/altera-opensource/linux-socfpga 
$ cd linux-socfpga

3. Checkout linux-socfpga to desired branch after git clone.

###example: git checkout -b <branch_name> -t origin/<branch>###
git checkout -b s10_10g -t origin/socfpga-5.15.50-lts

4. You may obtain s10_1588_devkit_defconfig file in "Release Content" section for socfpga-5.10.100-lts and below. For socfpga-5.15.50-lts and above, please refer to steps below:

• Copy default linux-socfpga defconfig file and rename to s10_1588_devkit_defconfig . The default defconfig file is located in ../linux-socfpga/arch/arm64/configs/defconfig.
• Update the s10_1588_devkit_defconfig with the following configs to enable QSE driver.

CONFIG_INTEL_FPGA_QSE_LL=y
CONFIG_ALTERA_QSE_LL=y
CONFIG_PHYLINK=y
CONFIG_NET_ALTERA_ETH=y
CONFIG_SFP=y
CONFIG_ARCH_INTEL_SOCFPGA=y
###Remove all other CONFIG_ARCH_###

Note:

• CONFIG_ARCH_STRATIX10=y was renamed to CONFIG_ARCH_INTEL_SOCFPGA=y in linux-socfpga-5.15.50-lts and above
• Remove all CONFIG_ARCH_* and keep only CONFIG_ARCH_INTEL_SOCFPGA=y to avoid SFP failure
• You are recommended to enable QSE driver through defconfig file instead of using menuconfig GUI method. There is a known issue where menuconfig GUI will introduce additional config file which breaks the GPIO driver initialization.

5. Rebuild kernel image and DTB with the following command:

$ make clean && make mrproper
$ make s10_1588_devkit_defconfig 
$ make -j8

6.The following files are generated:

File name	Location	Description
Image	arch/arm64/boot/	Linux Image
socfpga_stratix10_socdk.dtb	arch/arm64/boot/dts/altera/	Device Tree Blob

7. Replace the default S10 GSRD image content in the SD card with the newly build kernel image (Image) and DTB (socfpga_stratix10_socdk.dtb).

8. Use below command to generate kernel.itb. You can use the customized fit_kernel_stratix10.its here which consist of FIT image description configuration for single Image MMC boot.

The following files are required to build kernel.itb. You must copy all the required content into the same folder prior to run mkimage command:

• Image
• socfpga_agilex_socdk.dtb
• ghrd.core.rbf
• fit_kernel_stratix10.its

$ mkdir kernel_image
$ cd kernel_image
$ cp <user directory>/arch/arm64/boot/dts/altera/socfpga_stratix10_socdk.dtb socfpga_stratix10_socdk.dtb
$ cp <user directory>/arch/arm64/boot/Image Image
$ xz --format=lzma Image
$ cp <user directory>/ghrd_1sx280lu2f50e2vg.core.rbf ghrd.core.rbf
$ cp <user directory>/fit_kernel_stratix10.its fit_kernel_stratix10.its
$ mkimage -f fit_kernel_stratix10.its kernel.itb

9. Replace the default Stratix 10 SoC GSRD Fit image content in the SD card with the newly build kernel Fit image - kernel.itb.

Debug Checklist/Known Issues

1. GPIO initialization failure could be related to config file issue.

• You are recommended to enable QSE driver through defconfig file instead of using menuconfig GUI methods. There is a known issue where menuconfig GUI will introduce additional config file which breaks the GPIO driver initialization.

2. SFP initialization failure could be related to CONFIG_ARCH_ issue.

• CONFIG_ARCH_STRATIX10=y was renamed to CONFIG_ARCH_INTEL_SOCFPGA=y in linux-socfpga-5.15.50-lts and above
• You must remove all CONFIG_ARCH_* and keep only CONFIG_ARCH_INTEL_SOCFPGA=y to avoid SFP failure

3. You are recommended to follow hardware update flow for proper S10 GHRD 10Gbe build. You may encouter software migration error if directly update Quartus hardware design using a newer QPDS version. Please follow recommended GHRD vs GSRD version in HW/SW version compatibility.