Introduction

The IEEE 1588 v2 protocol facilitates precise synchronization of clocks in a network to a master reference clock. The diagram below illustrates the process of PTP synchronization between master and subordinate nodes.

In this process, the master node initiates handshaking by sending a Sync packet. Once 't1', 't2', 't3', and 't4' timestamps are gathered, the subordinate node can calculate and adjust its time-of-day clock to achieve synchronization with the network's master node. This handshake process is repeated at regular intervals to ensure that the subordinate clock remains consistently synchronized with the master clock.

Figure 1. High-level visualization of PTP packet exchange between master and subordinate.

Pre-requisites

This system example design is based on the Golden System Reference Design (GSRD) for Intel Agilex 7 I-Series Transceiver-SoC Development Kit (4x F-Tile). Intel recommends you familiarize with the GSRD development flow before moving ahead with this document.

Overview

This document introduces a system example design for a 2-Port, 25GbE or 10GE, IEEE 1588v2 FPGA implementation. This design incorporates IEEE 1588 PTP 2-Step timestamping with 'Advanced Accuracy Mode' in hardware. It is compiled using the Intel® Quartus® Prime Pro Edition version 23.2 and targets the Intel Agilex® 7 FPGA I-Series Transceiver-SoC Development Kit (4x F-Tile) (DK-SI-AGI027FA or DK-SI-AGI027FB). The system example design leverages the Ethernet Subsystem Intel® FPGA IP and operates on a Linux OS with Kernel v5.15.

This system example design supports both ordinary and boundary clock configurations. Hardware manages PTP timestamping, while the HPS and PTP software stack handle packet generation and local Time of Day (ToD) adjustment.

Key differences between the 25GE and 10GE implementation are highlighted trought this document.

A conformance test report is available for both implementations. Contact your field representative to gain access to these documents.

Hardware Architecture

System Architecture

The Intel Agilex 7 I-Series Transceiver-SoC Development Kit (4x F-Tile) system example design incorporates two Ethernet ports with hardware PTP timestamping support and a PTP software stack that runs within the Agilex 7 Hard Processor System (HPS). Key components of the system example design are:

• HPS Subsystem
• Channelized MSGDMA Subsystem
• Main ToD Subsystem
• Channelized Subordinate ToD Subsystem
• High-Speed Serial Interface (HSSI) Subsystem

The HPS Subsystem includes an Agilex HPS and all logic required to interface with the rest of the system. The HPS is running a PTP software stack that is used to start PTP traffic on both Ethernet interfaces and perform adjustments to the system main ToD. All Ethernet packets received and transmitted by the system have as origin and destination the HPS Subsystem.

The HPS subsystem has connection to other components in the board. It connects to the QSFPDD cage through an I2C bus for status and configuration. It also connects through I2C to the onboard Microchip ZL30733 PTP & SyncE Network Synchronizer for the same purpose.

A Modular Scatter-Gather DMA (mSGDMA) engine is used to move information between the HPS Subsystem and the Ethernet interfaces. The DMA Subsystem instantiates two independent modules, called 'DMA Ports', that provide DMA services to each of the Ethernet interfaces in the system. Each 'DMA Port' has two DMA Channels, each one handle the TX or RX traffic from a single Ethernet interface. The DMA Channels have been adapted to process PTP time information and interact seamlessly with the HSSI Subsystem.

Due to the modular nature of the DMA Subsystem, it is possible to add additional DMA Ports to support extra Ethernet interfaces in the system. At the same time, it is possible to customize a DMA Port to instantiate a single channel if the application may require such configuration.

Besides moving data around, the DMA Subsystem provides translation services between its native Avalon® Streaming Interfaces and the AXI-Stream (AXI-ST) interfaces of the Ethernet Subsystem. A clock-domain-crossing between the Ethernet Subsystem clock domain and the DMA clock domain occurs inside this module.

The TX-DMA Channel generates a unique fingerprint ID that is assigned to all outgoing packet payload. The HSSI Subsystem then receives the Ethernet packet payload along with its fingerprint and timestamps the packet as it exits the FPGA through one of the Ethernet interfaces. After an Ethernet packet has been released to the network, the HSSI Subsystem returns the egress timestamp with its corresponding fingerprint. Upon receiving a known fingerprint, the TX-DMA promptly relays the associated timestamp to the HPS. This timestamp is used by the PTP software stack running in the HPS for further processing. The TX-DMA Channel remains in a state of readiness, continuously anticipating fingerprints from the HSSI Subsystem.

The RX-DMA Channel works similar to its TX counter part, listens for all Ethernet packets arriving to the interface and their corresponding PTP timestamp. The HSSI Subsystem supplies ingress timestamps for all incoming packets. The RX-DMA Channel engine combines these timestamps with the packet payload and conveys this information to the HPS Subsystem for further processing.

The Main ToD Subsystem serves as the local ToD reference clock for the system. The HPS is responsible for making frequency and phase adjustments exclusively to the Main ToD Subsystem, ensuring its synchronization with the network time.

The Channelized Subordinate ToD Subsystem instantiates a Subordinate ToD for each Ethernet interface. Each Subordinate ToD works in the same clock domain as its designated Ethernet interface and provides timestamp services to the TX and RX Datapath. It's important to note that the Subordinate ToD solely conveys timestamps provided by the Main ToD Subsystem, without undergoing any frequency or phase adjustments.

The HSSI Subsystem is an instance of the Ethernet Subsystem Intel® FPGA IP. It plays a crucial role in configuring and implementing the system's ethernet interfaces. The Ethernet Subsystem configuration changes between the 10GbE and 25GbE implementations to reflect the desired link speeds.

Figure 2. System example design high-level block diagram.

F-Tile Channel Mapping

The system example design uses the Bank 13C F-Tile of the onboard Agilex device. Port 8 and Port 9 from the Ethernet Subsystem Intel® FPGA IP are utilized, and both are mapped to FGT channels on QUAD 2. Please refer to the table below for more details regarding the F-Tile interface mapping. There are no changes in the channel mapping between the 10GbE and the 25GbE designs.

Signal	Port	PMA	QUAD	Pin	Description
ftile_rx_serial[0]	p8	FGT	2	M1	RX FGT channel 0 from F-Tile QUAD2
ftile_tx_serial[0]	^	^	2	T7	TX FGT channel 0 from F-Tile QUAD2
ftile_rx_serial[1]	p9	^	2	W4	RX FGT channel 1 from F-Tile QUAD2
ftile_tx_serial[1]	^	^	2	W10	TX FGT channel 1 from F-Tile QUAD2
hssi_cdr_clk_out[0]	NA	NA	2	AA14	Recovery clock output pin for F-Tile QUAD2 Port 8
hssi_cdr_clk_out[1]	^	^	3	AT9	Recovery clock output pin for F-Tile QUAD3. Not used in the system example design

Keeping both Ethernet interfaces in the same F-Tile QUAD allows connecting 2 Intel Agilex® 7 FPGA I-Series Transceiver-SoC Development Kit with a single QSFP28 cable.

Using a single QUAD for both Ethernet interfaces have the limitation that only a single recovery clock output pin is usable for both ethernet ports (hssi_cdr_clk_out[0] recovered clock). This limitation prevents the full exercise of dual network reference clock for SyncE in the development kit, as the second ethernet recovered clock needs to be provided by a F-Tile channel on QUAD3.

To use the dual network reference clock for SyncE in the Intel Agilex® 7 FPGA I-Series Transceiver-SoC Development Kit, you need to make the following changes:

1. Instead of using Ethernet Subsystem port 9, use port 12 for the second Ethernet interface.
2. Route the recovered clock from port 12 to pin AT9.
3. Use a QSFPDD breakout cable to expose both Ethernet interfaces on different QSFP connectors.

By making these changes, you can utilize the dual network reference clock for SyncE in the development kit. This allows you to overcome the limitation of using a single recovery clock output pin for both Ethernet ports. A QSFPDD breakout cable is required for this change.

Custom FPGA IP Description

Block	Entity Name	Description
AVST to AXI Bridge	avst_axist_bridge	AVST to AXI bridge to cross information between the DMA channels and the Ethernet Subsystem. The bridge works on both ways, providing the AXI to AVST translation for information flowing from the Ethernet Subsystem to the DMA channels. The bridge is a single block servicing both, TX and RX DMA channels.
tx DMA fifo	tx_dma_fifo	Top level wrapper for custom blocks in the TX DMA Datapath.
TX DMA TS Adapter	hssi_ets_ts_adapter	Adapt the egress timestamp and fingerprint from the Ethernet Subsystem for a format manageable by the TX DMA channel.
TX DMA PKT FIFO	cdc_packet_fifo	Dual clock FIFO for clock domain crossing the packet information from the DMA channel to the Ethernet Subsystem.
TX FP Generator	tx_dma_fifo	Pseudo-random fingerprint generator. A fingerprint will be generated for all packet going out of the system. This is combinational logic inside the 'tx DMA fifo' module
TX REQ FP FIFO	fp_req_fifo	FIFO to store a copy of each fingerprint used to request a egress timestamp to the Ethernet subsystem.
TX TS/FP RESP FIFO	fp_resp_fifo/ts_fifo	Two independent FIFOs to store the returned egress timestamp and its corresponding fingerprint.
TX DMA FP Compare	tx_dma_fifo	Combinational logic that compares the output of 'TX REQ FP FIFO' and 'TX FP RESP FIFO'. The logic will indicate when the Ethernet Subsystem has returned the timestamp for the last outgoing packet. This is combinational logic inside the 'tx DMA fifo' module.
TX DMA TX Insert	tx_dma_fifo	Logic that insert egress timestamp into the MSGDMA prefetcher. This is combinational logic inside the 'tx DMA fifo' module.
rx DMA fifo	rx_dma_fifo	Top level wrapper for custom blocks in the RX DMA Datapath.
RX DMA PKT FIFO	cdc_packet_fifo	Dual clock FIFO for clock domain crossing the packet information from the Ethernet Subsystem to the DMA channel.
RX TS FIFO	ts_fifo	Single clock FIFO to store the ingress timestamps from the Ethernet Subsystem.
RX DMA TS Insert	rx_dma_fifo	Logic that insert ingress timestamp into the MSGDMA prefetcher. This is combinational logic inside the 'rx DMA fifo' module.
ToD Valid Generator	master_tod_top	State machine that flags when the Main ToD Subsystem output is valid for the ToD Subordinate Subsystem to consume.

Board Clocking Architecture

Component and signal identifiers used in this section follows the naming convention from the Intel Agilex® 7 FPGA I-Series Transceiver-SoC Development Kit Board Schematic file.

At board level, the PTP1588 clocking architecture includes the following components:

• Microchip ZL30733 PTP & SyncE Network Synchronizer - U123B
• Oven Controlled Crystal Oscillator (OCXO) – X2
• Intel Agilex® 7 AGIB027R31B - U1
• Si5332 Low-Jitter Clock Generator - U6

Figure 3. Board high-level clocking diagram.

The onboard Microchip ZL30733 PTP & SyncE Network Synchronizer takes charge of frequency synchronization, monitors reference clock quality, manages reference clock switching, and provides holdover functionality for SyncE and PTP1588 support. Five reference clocks can be used to generate the outputs in the ZL30733. The device is constantly monitoring the quality of the reference clocks and it switches to an alternative clock source when the measured period of the current reference signal is incorrect, or if it has excessive jitter. The following table shows the reference clock sources connected to the ZL30733.

Clock	Frequency	Description
OCXO_CLK20M	20 MHz	Onboard Oven Controlled Crystal Oscillator (X2) providing a stable high quality 20 MHz clock source. This reference clock is used for holdover periods when the ethernet recovered clocks are not available.
QSFP_RCVD_CLK1_[P/N]	39.062 MHz (25GbE) 26.041 MHz (10GbE)	F-Tile Ethernet recovered clock from Ethernet port 1.
QSFP_RCVD_CLK2_[P/N]	39.062 MHz (25GbE) 26.041 MHz (10GbE)	F-Tile Ethernet recovered clock from Ethernet port 2. This clock input is not utilize in the current system example design revision.
1PPS_SMA_IN	1 Hz	Onboard SMA connector. Must be configured as input through software. This input is not used in this system example design.
10MHz_SMA_IN	10 MHz	Onboard SMA connector. Must be configured as input through software. This input is not used in this system example design.

The ZL30733 generates 4 clock signals that are synchronized to the active reference clock. The device continues to generate all its outputs even if the active reference clock becomes unavailable, it has excessive jitter, or the period is incorrect. When the reference input change, the output clocks are synchronized to the new reference clock. As of now, the ZL30733 can only switch between a single network-recovered clock and the onboard OCXO. The next table shows the output clocks generated by the ZL30733 devices.

Clock	Frequency	Description
ToD_CLK_100M_[P/N]	156.25 MHz	Reference clock for the ToD Master and Slave Subsystems.
1PPS_SMA_OUT	1 Hz	Onboard SMA connector. Must be configured as output through software. This signal can be used to compare the accuracy of the 1 PPS signal generated by the FPGA for debug purposes. The PPS pulse width is defined by the ‘Pulse Width’ parameter in the Ethernet IEEE 1588 Time of Day Clock Intel® FPGA IP.
10MHz_SMA_OUT	10 MHz	Onboard SMA connector. Must be configured as output through software.
ETH_REFCLK_156.25M_[P/N]	156.25 MHz	Reference clock for the Ethernet Subsystem.

The Intel Agilex® 7 AGIB027R31B FPGA relies on the ZL30733 device for all IEEE 1588 PTP related reference clocks. A Si5332 Low-Jitter Clock Generator generates a 100 MHz that is used for support logic in the FPGA. Following Table shows the input clocks for the system FPGA.

Clock	Frequency	Description
ToD_CLK_100M_[P/N]	156.25 MHz	Reference clock for the ToD Master and Slave Subsystems. Signal connected to bank 2E, pin DB53 in the FPGA.
ETH_REFCLK_156.25M_[P/N]	156.25 MHz	Reference clock for the Ethernet Subsystem. Signal connected to Bank 13C, pin R14/U14 in the FPGA.
CLK_GPIO_[P/N]_4	100.00 MHz	General purpose clock for core fabric. Signal connected to Bank 2D, pin CM29/CL30 in the FPGA.

Three output clocks are generated by the AGIB027R31B device. Two clocks derived by the Clock Data Recovery (CDR) circuitry from each ethernet channel and a Pulse Per Minute (PPM) signal that is connected to the onboard FMC J7A connector. The following table shows the output clocks for the same device.

Clock	Frequency	Description
QSFP_RCVD_CLK1_[P/N]	39.062 MHz (25GbE) 26.041 MHz (10GbE)	F-Tile Ethernet recovered clock from port 1. Signal connected to Bank 13C, pin AA14/AB13 in the FPGA.
QSFP_RCVD_CLK2_[P/N]	39.062 MHz (25GbE) 26.041 MHz (10GbE)	F-Tile Ethernet recovered clock from port 2. Signal connected to Bank 13C, pin AT9/AR10 in the FPGA.
FMC_A_LA_P30	1.00 Hz	PPM signal generated by FPGA ToD. Signal connected to Bank 3B, pin J56 in the FPGA.

The output frequency from the recovered clocks ports depends on the system example design speed configuration.

Clock Tree Bring-Up Sequence

The Intel Agilex® 7 FPGA I-Series Transceiver-SoC Development Kit is set to operate in HPS Boot First Mode. Upon power-up, all I/O allocated to the FPGA remains in a tri-state (high-impedance) state while the HPS undergoes booting. Once the HPS completes booting, it initiates the configuration of the FPGA.

The Microchip ZL30733 PTP & SyncE Network Synchronizer is designed to load a configuration profile from its internal non-volatile memory during power-up. This profile specifies the characteristics of the reference clocks at its inputs and the parameters required to generate the profile output clocks.

However, it's important to note that the ZL30733's internal non-volatile memory may not initially contain the correct profile for the PTP1588 system example design at power-up. At boot time, the HPS establishes access to the ZL30733 via an I2C interface and proceed to load the correct profile for the specific application. This action configures the input reference clocks and output clocks as outlined in the following table.

Clock	Port	Frequency
QSFP_RCVD_CLK1	REFIN0	39.062 MHz (25GbE) 26.041 MHz (10GbE)
QSFP_RCVD_CLK2	REFIN1	39.062 MHz (25GbE) 26.041 MHz (10GbE)
OCXO_CLK20M	REFIN4	20 MHz
ToD_CLK_100M	OUT0	156.25 MHz
1PPS_SMA_OUT	OUT1n	1 Hz
ETH_REFCLK_156.25M	OUT3	156.25 MHz

The HPS configures the ZL30733 device before proceeding to configure the FPGA. This is because the transceivers and PLL in the FPGA require their reference clock signals to be stable before they can be successfully configured and power up. The HPS sets the appropriate REFIN0 and REFIN1 frequency values depending on the Ethernet link speed of the design.

At board level, the connection between the HPS and the ZL30733 device needs to be enabled, as by defaul the Development Kit Board Test System has control over the ZL30733 device. To grand access to the HPS to the ZL30733 I2C configuration interface, the system example design comes with a custom bitstream for the onboard Intel MAX10 device that enables the bridge between the FPGA and the ZL30733. Refer to signal ZL_I2C_EN in the Intel Agilex® 7 FPGA I-Series Transceiver-SoC Development Kit schematic files for more information regarding the connection.

FPGA Clocking Architecture

The component and signal identifiers used in this section follows the naming convention from the Agilex 7 I-Series Transceiver-SoC DevKit (4x F-Tile) system example design With IEEE1588 PTP Intel® Quartus® Prime project.

A high-level FPGA system-level clock architecture is shown in the figure below.

Figure 4. Fpga high-level clocking diagram.

For the clock frequencies associated with the Ethernet Subsystem IP ports i_p[8/9]_clk_[tx/rx]_tod (Time-of-Day associated input clock) and i_p[8/9]_clk_ptp_sample (Sample Clock for PTP Measurement Input Clock), the system example design follow the guidelines provided in the Ethernet Subsystem Intel® FPGA IP User Guide.

The clock signal generated by the ftile_iopll_todsync_sampling IOPLL Intel® FPGA IP is utilized by both instances of the Ethernet IEEE 1588 TOD Synchronizer Intel® FPGA IP to manage the clock domain crossing between the system's main ToD and the subordinate TX ToD and RX ToD. The sampling frequency set by the ftile_iopll_todsync_sampling IOPLL Intel® FPGA IP follows the guidelines outlined in the Ethernet Design Example Components User Guide.

The system's main ToD has 'Advanced Accuracy Mode' enabled and employs the 'IOPLL and TOD Setup using IOPLL Reconfig IP' scheme for its hardware implementation.

The following table shows an overview of all clock signals that are related to the PTP Datapath.

Clock	Frequency	Reference Clock	Description
clk_100_out_clk	100 MHz	fpga_clk_100 (100 MHz)	Reference clock is provided by CLK_GPIO_P_4 signal at board level. This clock is used by all modules in the design to drive the Intel® Avalon® Memory-Mapped Interface and the Advanced eXtensible Interface (AXI) – lite Interface used for control and status registers on the system modules. clk_100_out_clk is an alias for the clock signal as it is distributed by the System Manager Module.
eth_f_tod_clk_156	156.25 MHz	ftile_master_todclk_ref (156.25 MHz)	Reference clock is provided by ToD_CLK_100M signal at board level. eth_f_tod_clk_156 is the main reference clock for the Ethernet IEEE 1588 Time of Day Clock Intel® FPGA IP instantiated in the Master ToD subsystem. Refer to the Ethernet Design Example Components User Guide for more information on clocking requirements. eth_f_tod_clk_156 is an alias for the clock signal as it is distributed by the System Manager Module.
ftile_iopll_ptp_sampling.outclk0	114.285 MHz	fpga_clk_100 (100 MHz)	Reference clock is provided by CLK_GPIO_P_4 signal at board level. Output frequency is defined by the Ethernet Subsystem Intel® FPGA IP and is generated by ftile_iopll_ptp_sampling IOPLL Intel® FPGA IP. The clock signal is used as sample clock for the PTP measurement in the Ethernet Subsystem Intel® FPGA IP.
ftile_iopll_todsync.sampling_outclk0	266.666 MHz	eth_f_tod_clk_156_outclk0 (156.25 MHz)	Reference clock is provided by ToD_CLK_100M signal at board level. The output frequency is defined by the Sampling Clock Frequency described in the Ethernet IEEE 1588 TOD Synchronizer Intel® FPGA IP. This clock signal is generated by ftile_iopll_ptp_sampling IOPLL Intel® FPGA IP. This clock signal is be used in the process of timestamp synchronization between the system master ToD and the subordinate ToD blocks assigned to each ethernet ports.

The table below offers an overview of all generated clocks signals used in the Main ToD Subsystem to implement the 'Advance Accuracy Mode'.

Clock	Frequency	Reference Clock	Description
ghrd_f_mtod_master_pll.outclk_0	78.125 MHz	ftile_master_todclk_ref (156.25 MHz)	Reference clock is provided by ToD_CLK_100M signal at board level. Output frequency is defined by the Ethernet IEEE 1588 Time of Day Clock Intel® FPGA IP and is generated by ghdr_mtod_master_pll IOPLL Intel® FPGA IP. This clock signal drives the phase shift adjustment to the ghdr_mtod_pss_pll IOPLL Intel® FPGA IP instantiated to support ‘Advance Accuracy Mode’ for the Ethernet IEEE 1588 Time of Day Clock Intel® FPGA IP. This clock signal defines the frequency of the phase shift adjustments done to the ghdr_mtod_pss_pll IOPLL.
ghrd_f_mtod_pss_pll.outclk_0	108.17 MHz	ftile_master_todclk_ref (156.25 MHz)	Reference clock is provided by ToD_CLK_100M signal at board level. Output frequency is defined by the Ethernet IEEE 1588 Time of Day Clock Intel® FPGA IP to implement ‘Advance Accuracy Mode’. This clock signal is generated by ghdr_mtod_pss_pll IOPLL Intel® FPGA IP.
ghrd_f_mtod_pss_pll.outclk_1	156.25 MHz	ftile_master_todclk_ref (156.25 MHz)	Reference clock is provided by ToD_CLK_100M signal at board level. Output frequency is defined by the Ethernet IEEE 1588 Time of Day Clock Intel® FPGA IP and is generated by ghdr_mtod_pss_pll IOPLL Intel® FPGA IP. This clock signal is used to generate the PPS signal for the system

The next table presents generated clock signals that play an active role in the Ethernet Datapath.

Clock	Frequency	Reference Clock	Description
o_[p8/p9]_clk_rec_div	390.625 MHz (25G) 156.25 MHz (10G)	Reference clock is provided by ETH_REFCLK_156.25M signal at board level.	Ethernet Subsystem RX recovered clock. This clock is used to synchronize the RX ToD Subordinate Subsystem with the RX Ethernet subsystem channel
o_[p8/p9]_clk_tx_div	390.625 MHz (25G) 156.25 MHz (10G)	Reference clock is provided by ETH_REFCLK_156.25M signal at board level.	Ethernet Subsystem TX recovered clock. This clock is used to synchronize the TX ToD Subordinate Subsystem with the RX Ethernet subsystem channel
o_[p8/p9]_clk_pll	415.041 MHz	Reference clock is provided by ETH_REFCLK_156.25M signal at board level.	Ethernet Subsystem core logic clock. This clock drives the Ethernet Subsystem AXI-ST interfaces
iopll_clk_avst_div2.outclk_0	207.52 MHz	Reference clock is provided by o_[p8/p9]_clk_pll	Half-rate derived clock from Ethernet Subsystem core logic clock. It is used to drive the AVST logic from the DMA Subsystem

ToD Architecture

For more information about the Ethernet IEEE 1588 Time of Day Clock Intel® FPGA IP and the Ethernet IEEE 1588 TOD Synchronizer Intel® FPGA IP, please refer to the Ethernet Design Example Components User Guide.

Figure 5. Time-of-Day high-level block diagram.

The configurations of the Ethernet IEEE 1588 Time of Day Clock Intel® FPGA IP and the Ethernet IEEE 1588 TOD Synchronizer Intel FPGA IP instances inside 'TX ToD' and 'RX ToD' vary between the 25G and 10G system example design configurations. The differences in parameters are summarized in the following tables:

Ethernet IEEE 1588 Time of Day Clock Intel® FPGA IP

IP Parameter	25G	10G
PERIOD_CLOCK_FREQUENCY	1	0
DEFAULT_NSEC_PERIOD	2	6
DEFAULT_FNSEC_PERIOD	36700	26214
DEFAULT_NSEC_ADJPERIOD	2	6
DEFAULT_FNSEC_ADJPERIOD	36700	26214

Ethernet IEEE 1588 TOD Synchronizer Intel FPGA IP

IP Parameter	25G	10G
SYNC_MODE	9	2
PERIOD_NSEC	2	6
PERIOD_FNSEC	36700	26214

Register Map

Name	Component	H2F AXI Master	Register Description
hssi_ss_1.axi4_lite_interface	Ethernet Subsystem CSR	0x0000_0000 - 0x03ff_ffff	Link
ocm.s1	On-Chip Memory	0x0400_0000 - 0x0403_ffff
mtod_subsys.master_tod_top_0_csr	Main ToD Subsystem	0x0404_0000 - 0x0404_003f	Link
sys_ctrl_pio_0.s1	QSDP-DD Control PIO	0x0404_0040 - 0x0404_004f	Link
qsfpdd_status_pio.s1	QSFP-DD Status PIO	0x0404_0050 - 0x0404_005f	Link
ftile_debug_status_pio_0.s1	F-Tile Status Register	0x0404_0060 - 0x0404_006f	Link
mtod_subsys.mtod_subsys_pps_load_tod_0_csr	Reserved	0x0404_0100 - 0x0404_01ff
dma_subsys.dma_subsys_port8_csr	DMA Subsystem Port 8	0x0448_0000 - 0x0448_00ff	Link
dma_subsys.dma_subsys_port12_csr	DMA Subsystem Port 12	0x044c_0000 - 0x044c_00ff	Link

QSDPDD SPI Control PIO Register Description

Address	Name	Bit	Reset Value	Attribute	Description
0x0	qsfpdd_ctrl_pio	[0]	1'b1	RW	QSFPDD resetn
^	qsfpdd_ctrl_pio	[1]	1'b1	RW	QSFPDD initmode/lpmode
^	qsfpdd_ctrl_pio	[2]	1'b0	RW	QSFPDD modseln
0x1 - 0xF	Reserved

QSDPDD Status PIO Register Description

Address	Name	Bit	Reset Value	Attribute	Description
0x0	qsfpdd_status_pio	[0]		RO	QSFPDD modprsn
^	qsfpdd_status_pio	[1]		RO	QSFPDD intn
0x1 - 0xF	Reserved

F-Tile Status Register Description

Bit	Ethernet Subsystem Port	Description
0	Port 8	Ethernet Subsystem 'o_p8_tx_lanes_stable' signal. For more information consult the following link
1	^	Ethernet Subsystem 'o_p8_tx_pll_locked' signal. For more information consult the following link
2	^	Ethernet Subsystem 'o_p8_rx_pcs_ready' signal. For more information consult the following link
3	^	Ethernet Subsystem 'o_p8_tx_ptp_ready' signal. For more information consult the following link
4	^	Ethernet Subsystem 'o_p8_rx_ptp_ready' signal. For more information consult the following link
5	^	Ethernet Subsystem 'o_p8_rx_ptp_offset_data_valid' signal. For more information consult the following link
6	^	Ethernet Subsystem 'o_p8_tx_ptp_offset_data_valid' signal. For more information consult the following link
10	Port 9	Ethernet Subsystem 'o_p9_tx_lanes_stable' signal. For more information consult the following link
11	^	Ethernet Subsystem 'o_p9_tx_pll_locked' signal. For more information consult the following link
12	^	Ethernet Subsystem 'o_p9_rx_pcs_ready' signal. For more information consult the following link
13	^	Ethernet Subsystem 'o_p9_tx_ptp_ready' signal. For more information consult the following link
14	^	Ethernet Subsystem 'o_p9_rx_ptp_ready' signal. For more information consult the following link
15	^	Ethernet Subsystem 'o_p9_rx_ptp_offset_data_valid' signal. For more information consult the following link
16	^	Ethernet Subsystem 'o_p9_tx_ptp_offset_data_valid' signal. For more information consult the following link

DMA Subsystem Memory Map

Slave	Component	subsys_ftile_25gbe_1588_csr	Register Description
ftile_25gbe_rx_dma_ch1	RX DMA channel	0x0080 - 0x00bf	Link
ftile_25gbe_tx_dma_ch1	TX DMA channel	0x0000 - 0x003f	Link

DMA TX and RX Channels Memory Map

Slave	Component	[rx/tx]_dma_csr	Register Description
[tx/rx]_dma_dispatcher	DMA dispatcher	0x0020 - 0x003f	Link
[tx/rx]_dma_prefetcher	DMA prefetcher	0x0000 - 0x001f	Link

Interrupt Map

Interrupt	F2H IRQ 0
dipsw_pio_irq	0
button_pio_irq	1
mtod_subsys_pps_load_tod_0_pps_irq	2
qsfpdd_status_pio	5
ftile_debug_status_pio	6
dma_subsystem_port8_tx_dma_ch1_irq	7
dma_subsys_port8_rx_dma_ch1_irq	8
dma_subsystem_port12_tx_dma_ch1_irq	9
dma_subsys_port12_rx_dma_ch1_irq	10

Software Architecture

Architecture Overview

The system example design software architecture adopts a layered approach, emphasizing modularity and reusability. The system's drivers are designed to be modular, offering specific functionality that can be leveraged by higher layers of the software stack. The software architecture for the system example design is independent of the hardware boot-up sequence (whether is FPGA first or HPS first) or the method used to access the HPS. The architecture outlined below specifically focuses on the components related to the PTP software stack.

Figure 6. High-level Precision Time Protocol software stack diagram.

The Intel Agilex® 7 FPGA I-Series Transceiver-SoC Development Kit (4x F-Tile) (DK-SI-AGI027FA) follows a HPS-first design approach. This means that the HPS initializes first and then configures the FPGA with its corresponding bitstream. The HPS loads the uBoot image from SPI flash in this system example design. The secondary boot loader image contains the final kernel and FPGA configuration bitstream. The uBoot secondary boot loader activates the HPS bridges and programs the FPGA through its connection to the SDM. Once the FPGA is programmed, the HPS proceeds to boot the Linux operating system.

In this system example design, the Ethernet Subsystem IP is controlled by the HPS as the primary system CPU. The Quad Small Form-factor Pluggable module is connected to the HPS through an I2C connection, enabling the HPS to communicate with the QSFP module via this bus. The control pins of the QSFP, including MODSEL, Presence, interrupt, and LP_MODE, are connected to the HPS using General Purpose Input/Output (GPIO) pins that are available for use by the HPS.

Intel Agilex 7 SoC-FPGA F-Tile Drivers

HSSI Subsystem Drivers

The HSSI Subsystem driver acts as a bridge between the software operating in the HPS and the HSSI Subsystem within the FPGA. It provides various levels of abstraction to simplify communication with the underlying Ethernet Subsystem IP. The HSSI Subsystem driver exposes Ethernet netdev driver APIs that higher-level software layers can utilize to interact with the Ethernet Subsystem IP. Some of the abstractions offered by the HSSI Subsystem driver include:

1. Get Link state.
2. Get MAC stats. These abstractions are used by the HSSI ethernet netdev driver to provide ethernet functionality to the above layers.

HSSI Ethernet and associated driver

The HSSI Ethernet netdev driver offers a network device interface (Linux netdev interface) to the Linux kernel. It registers all the necessary interfaces to enable the corresponding functionalities provided by the system like:

1. mSGDMA support for data movement.
2. PTP functionality support.
3. ToD driver functionality support.

After successful registration, the driver awaits configuration-related requests from the user. To handle the configurations request, the driver offers an interface for the user-space application ethtool to communicate with the driver and configure various settings.

QSFP Driver interface

The QSFP driver is responsible for interacting with the onboard QSFP module. It reads the QSFP Serial Electrically Erasable Programmable Read-Only Memory (SEEP) and controls the power and interrupt pins of the QSFP. In the Intel Agilex 7 I-Series Transceiver-SoC Development Kit, the QSFP module is connected directly to the HPS using an I2C bus. The kernel QSFP driver communicates with the QSFP by reading the registers directly through the I2C bus and provides the information to applications such as ethtool.

TOD Driver interface

The TOD driver collaborates with the Master ToD Subsystem to configure the system's Time of the Day (ToD) value. The Master ToD Subsystem incorporates the Ethernet IEEE 1588 Time of Day Clock Intel® FPGA IP, and utilizes a stable clock to record the time of the day with nanosecond resolution. The driver interface provides read and write functionality to the Ethernet IEEE 1588 Time of Day Clock IP. By comparing the incoming value with the calculated ToD drift values, the ToD clock can be adjusted to either count faster or slower, enabling faster synchronization with the master time.

User space applications

Ptp4l

The main utility responsible for running the PTP servo is called ptp4l. This utility is included in the HPS final image and is also an open-source utility. ptp4l offers multiple configuration options to set up the system correctly. For more information on the various configuration options, please refer to the following link: ptp4l man page.

Phc2sys

phc2sys is an open-source utility program that synchronizes two clocks in the system. It is commonly used to synchronize the system clock to a PTP hardware clock (PHC), which is itself synchronized by the ptp4l program. For more information on phc2sys, please refer to the following link: phc2sys man page.

Ethtool

ethtool is a well-known open-source utility used to configure or query network driver and hardware settings. For more information on ethtool, please refer to the following link: ethtool man page.

Release Contents

25GE 2-Port System Example Design Release Link

Release Package for DK-SI-AGI027FB development kit: dk-si-agi027fb_release_package_25G.zip

10GE 2-Port System Example Design Release Link

Release Package for DK-SI-AGI027FB development kit: dk-si-agi027fb_release_package_10G.zip

Release Files Description

File	Description
hardware/xxG_ghrd_agib027r31b1e2v_23_2_0_94.qar	Intel Quartus Prime system example design archive
bmc/max10_system_0002aa4F.pof	MAX10 configuration bitstream for bridging the FPGA HPS and the onboard ZL30733 device
fim/ghrd.core.rbf	HPS first configuration bitstream, phase 2 : FPGA
fim/ghrd.hps.rbf	HPS first configuration bitstream, phase 1: HPS and DDR
fim/ghrd_agib027r31b1e2v.sof	FPGA configuration file
linux/socfpga_fm87_ftile_25g_ptp.dtb	Device tree binary
linux/fit_kernel_fm87_ftile.its	FIT image description file for kernel.itb
linux/kernel_fm87.itb	FIT Image file which is a combination of kernel Image, DTB and ghrd_core.rbf
linux/Image_fm87.lzma
yocto/boot.scr.uimg
yocto/u-boot.itb	U-boot image source file which is a combination of ATF and SSBL u-boot.img
yocto/u-boot-spl-dtb.hex	HPS FSBL hex file
yocto/cfg/master.cfg	ptp4l reference configuration file for a master interface
yocto/cfg/slave.cfg	ptp4l reference configuration file for a slave interface
yocto/cfg/boundary.cfg	ptp4l reference configuration file for setting a development kit as a boundary clock
yocto/socfpga_fm87_ftile_25g_ptp.zip	Yocto project source files
sdimage.tar.gz	Prebuilt SD card image for this system example design

System Requirements

Hardware Requirements

• PTP Master/Subordinate hardware
  • 2 x DK-SI-AGI027FB or DK-SI-AGI027FA - Intel Agilex® 7 FPGA I-Series Transceiver-SoC Development Kit (4 F-Tiles)
  • Micro SD card
  • HPS IO48 OOBE daughter card
  • QSPI flash daughter card
  • Mini USB cable for OOBE daughter card serial port
  • USB Type B cable for on-board Intel FPGA Download Cable II

QSFP Cable Compatibility

The design was tested with the following cables

• FS (Q28-PC01) - 1m (3ft) 100G QSFP28 Passive Direct Attach Copper Twinax Cable (System reference design is tuned to work with this cable by default).
• FS (Q28-AO05) - 5m (16ft) 100G QSFP28 Active Optical Cable.

The System Example Design includes the following QSF assignments for 1m DAC cables and AOC cables.

set_instance_assignment -name HSSI_PARAMETER "vsr_mode=VSR_MODE_LOW_LOSS" -to ftile_rx_serial[0] -entity ghrd_agilex_top
set_instance_assignment -name HSSI_PARAMETER "vsr_mode=VSR_MODE_LOW_LOSS" -to ftile_rx_serial[1] -entity ghrd_agilex_top

Software Requirements

• Intel® Quartus® Prime Pro Edition Design Software Version 23.2
• Linux Host PC with Ubuntu 22.04 LTS
  • Minicom 2.7.1

Quick Start Guide

Flash the SD card

The SD card image file is provided in "sdimage.tar.gz" package, you may refer to Release Content for more details.

Follow the instructions under "Creating SD Card on Linux/Windows" from the Intel Agilex 7 SoC GSRD to create a boot-able SD card with this image file.

Hardware Setup

• Start by connecting the two Intel Agilex I-series Transceiver-SoC Development Kit using a QSFP-DD/28 cable via the QSFP-DD port J27 (Highlighted in yellow in the image below). One of the boards acts as PTP Master while the other as PTP Subordinate.

Figure 7. Board level connection between development kits.

• Connect the mini-USB ports (J7) from each of the OOBE Daughter Card to your host machine. Both development kits OOBE Daughter Cards are connected to the same host.
• Connect the Type B USB cable from each development kit to the host for JTAG access.
• Set all configuration switches in the Agilex 7 I-Series Transceiver-SoC Development Kit to their default value.

Switch	Default Position
S19 [1:4]	OFF / OFF / ON / ON
S20 [1:4]	ON / ON / ON / ON
S9 [1:4]	ON / OFF / OFF X
S10 [1:4]	ON / ON / ON / ON
S15 [1:4]	ON / ON / ON / OFF
S1 [1:4]	OFF / OFF / OFF / OFF
S6 [1:4]	OFF / OFF / OFF / OFF
S22 [1:4]	ON / ON / ON / ON
S23 [1:4]	ON / ON / ON / ON
S4 [1:4]	ON / ON / ON / ON

• As mentioned in the Board Clocking Architecture section, the system example design requires a custom MAX10 bitstream to provide the HPS access to the onboard ZL30733 devices. Before proceeding further you will need to load max10_system_0002aa4F.pof to the onboard MAX10. You can use the Intel Quartus Programmer to perform the operation or follow the next steps to perform the action through command line:

Verify that all devices from the development kit are recognized and check the Jtag cable number assigned to the development kit with jtagconfig.

/home/hectorca$ jtagconfig
1) Simulator [fmev0132.fm.intel.com:1127]
  Unable to lock chain - Hardware not attached

2) Agilex I-Series SOC Dev Kit on 10.122.105.156 [1-3]
  6BA00477   S10HPS/AGILEX_HPS/N5X_HPS
  0343B0DD   AGFB027R24C(.|B|R2|R0)/..
  031830DD   10M16S(A|C|L)

3) Agilex I-Series SOC Dev Kit on 10.122.105.156 [1-4]
  6BA00477   S10HPS/AGILEX_HPS/N5X_HPS
  0343B0DD   AGFB027R24C(.|B|R2|R0)/..
  031830DD   10M16S(A|C|L)

From the previous output, you can see that two Intel Agilex 7 Transceiver-SoC Development Kits are visible, both of them have all their devices identified correctly and that they have been assigned to cable 2) and 3).

Now you can configure the development kits from your host with the following command:

/home/hectorca$ quartus_pgm -c 2 -m jtag -o "p;max10_system_0002aa4F.pof@3"

Repeat the same operation on the second development kit.

Exercising the System Example Design

Boot Linux and verify the link is up

The Embedded Linux operating system running on the Intel Agilex 7 Transceiver-SoC Development Kit can be accessed using a Serial Communication program such as Mincom or Putty. Start by identifying the assigned ID for each of your serial connections between the host and the development kits. You can run the following command in your Ubuntu host to display the last USB-to-Serial devices connected to the machine:

admin@10.1.23.255:~$ dmesg | grep "ttyUSB*"
[    6.231169] usb 1-2: FTDI USB Serial Device converter now attached to ttyUSB0
[    6.251435] usb 1-1.2: FTDI USB Serial Device converter now attached to ttyUSB1
[    6.255400] usb 1-1.3: FTDI USB Serial Device converter now attached to ttyUSB2

In this example, the last two devices represent the connection to the Intel Agilex 7 Transceiver-SoC Development Kits.

Start a serial session to each development kit using minicom. Start a minicom session on different terminals to have both of them open at the same time.

admin@10.1.23.255:~$ minicom -D /dev/ttyUSB1

Configure each serial session with the following parameters:

• Bps/Par/Bits: 115200 8N1
• Hardware Flow Control: No
• Software Flow Control: No

You can access Minicom configuration screen with the following key combinations:

• 'CTRL + a then z' for the Command Summary menu
• 'SHIFT + o' for the configuration menu
• Then select 'Serial port setup'

Your 'Serial port setup' screen looks as follows after adjusting the configuration parameters:

Welcome to minicom 2.7.1                                                                  

OPTI+-----------------------------------------------------------------------+             
Comp| A -    Serial Device      : /dev/ttyUSB1                              |             
Port| B - Lockfile Location     : /var/lock                                 |             
    | C -   Callin Program      :                                           |             
Pres| D -  Callout Program      :                                           |             
    | E -    Bps/Par/Bits       : 115200 8N1                                |             
    | F - Hardware Flow Control : No                                        |             
    | G - Software Flow Control : No                                        |             
    |                                                                       |             
    |    Change which setting?                                              |             
    +-----------------------------------------------------------------------+             
            | Screen and keyboard      |                                                  
            | Save setup as dfl        |                                                  
            | Save setup as..          |                                                  
            | Exit                     |                                                  
            +--------------------------+

For more information about the serial session bring up please refer to 'Configuring Serial Connection' from the GSRD for Agilex 7 I-Series Transceiver-SoC DevKit (4x F-Tile) website.

Using the Intel® Quartus® Programmer Version 23.2, configure the onboard Intel Agilex device with ghrd_agib027r31b1e2v.hps.rbf. Alternatively, you can achieve the same goal through command line with the following steps:

Verify that all devices from the development kit are recognized and check the JTAG cable number assigned to the development kit with jtagconfig.

/home/hectorca$ jtagconfig
1) Simulator [fmev0132.fm.intel.com:1127]
  Unable to lock chain - Hardware not attached

2) Agilex I-Series SOC Dev Kit on 10.122.105.156 [1-3]
  6BA00477   S10HPS/AGILEX_HPS/N5X_HPS
  0343B0DD   AGFB027R24C(.|B|R2|R0)/..
  031830DD   10M16S(A|C|L)

3) Agilex I-Series SOC Dev Kit on 10.122.105.156 [1-4]
  6BA00477   S10HPS/AGILEX_HPS/N5X_HPS
  0343B0DD   AGFB027R24C(.|B|R2|R0)/..
  031830DD   10M16S(A|C|L)

From the previous output, you can see that 2 Agilex I-Series SOC Dev Kit are visible, both of them have all their devices identified correctly and that they have been assigned to cable 2) and 3).

Now you can configure the development kits from your host with the following command:

/home/hectorca$ quartus_pgm -c 2 -m jtag -o "p;ghrd_agib027r31b1e2v.hps.rbf@2"
Info: *******************************************************************
Info: Running Quartus Prime Programmer
    Info: Version 23.2.0 Build 94 06/14/2023 SC Pro Edition
    Info: Copyright (C) 2023  Intel Corporation. All rights reserved.
    Info: Your use of Intel Corporation's design tools, logic functions 
    Info: and other software and tools, and any partner logic 
    Info: functions, and any output files from any of the foregoing 
    Info: (including device programming or simulation files), and any 
    Info: associated documentation or information are expressly subject 
    Info: to the terms and conditions of the Intel Program License 
    Info: Subscription Agreement, the Intel Quartus Prime License Agreement,
    Info: the Intel FPGA IP License Agreement, or other applicable license
    Info: agreement, including, without limitation, that your use is for
    Info: the sole purpose of programming logic devices manufactured by
    Info: Intel and sold by Intel or its authorized distributors.  Please
    Info: refer to the applicable agreement for further details, at
    Info: https://fpgasoftware.intel.com/eula.
    Info: Processing started: Fri Oct 20 08:27:08 2023
    Info: System process ID: 24918
Info: Command: quartus_pgm -c 4 -m jtag -o p;ghrd_agib027r31b1e2v.hps.rbf@2
Info (213046): Using programming cable "Agilex I-Series SOC Dev Kit on 10.122.105.156 [1-3]"
Info (213011): Using programming file ghrd_agib027r31b1e2v.hps.rbf with checksum 0x76817BF9 for device AGIB027R31B@2
Info (209060): Started Programmer operation at Fri Oct 20 08:27:09 2023
Info (18942): Configuring device index 2
Info (18943): Configuration succeeded at device index 2
Info (209011): Successfully performed operation(s)
Info (209061): Ended Programmer operation at Fri Oct 20 08:27:15 2023
Info: Quartus Prime Programmer was successful. 0 errors, 0 warnings
    Info: Peak virtual memory: 1927 megabytes
    Info: Processing ended: Fri Oct 20 08:27:15 2023
    Info: Elapsed time: 00:00:07
    Info: System process ID: 24918
/home/hectorca$

Repeat the the previous command for the second development kit. Keep in mind that you may need to update the cable numbers in your quartus_pgm command base on your host.

If everything went as expected, each Minicom terminal shows the messages from the HPS booting Linux OS. Here you can download a transcript of a successful OS booting.

To login into the system use 'root' as your credentials:

agilexfm87 login: root
Last login: Mon Mar 13 20:59:13 +0000 2023 on /dev/ttyS0.
root@agilexfm87:~#

Repeat the same steps for the second Intel Agilex 7 I-Series Transceiver-SoC Development Kit.

Assign IP address and verify connection between boards

Start by checking the network status on each Intel Agilex 7 I-Series Transceiver-SoC Development Kit with the 'ip' command:

root@agilexfm87:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether a2:33:a4:54:0b:1c brd ff:ff:ff:ff:ff:ff
    inet 10.122.105.70/24 brd 10.122.105.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::a033:a4ff:fe54:b1c/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 6a:19:3c:de:09:07 brd ff:ff:ff:ff:ff:ff
    inet 169.254.83.72/16 brd 169.254.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::6819:3cff:fede:907/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 3e:09:ef:f7:a6:40 brd ff:ff:ff:ff:ff:ff
    inet 169.254.209.81/16 brd 169.254.255.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::3c09:efff:fef7:a640/64 scope link 
       valid_lft forever preferred_lft forever
root@agilexfm87:~#

eth1 and eth2 are the interface names assigned to both ethernet ports. Both ethernet interfaces need to be in 'UP' state as shown in the previous transcript. The interfaces also have an assigned IP4 and IP6 address assigned to them.

eth1
inet 169.254.83.72/16 brd 169.254.255.255 scope global eth1
inet6 fe80::6819:3cff:fede:907/64 scope link 

eth2
inet 169.254.209.81/16 brd 169.254.255.255 scope global eth2
inet6 fe80::3c09:efff:fef7:a640/64 scope link

Use the 'ping' command to verify the connectivity between both development kits. Start by getting the IP address of eth1 from both development kits:

Development kit 1, eth1 IP address: 169.254.83.72
Development kit 2, eth1 IP address: 169.254.60.108

Both IP addresses must belong to the same sub network in order to communicate between each other. Execute the following command to text the connectivity:

Development Kit 1

root@agilexfm87:~# ping -c 3 169.254.60.108                                                                      
PING 169.254.60.108 (169.254.60.108): 56 data bytes
64 bytes from 169.254.60.108: seq=0 ttl=64 time=0.314 ms
64 bytes from 169.254.60.108: seq=1 ttl=64 time=0.092 ms
64 bytes from 169.254.60.108: seq=2 ttl=64 time=0.081 ms

--- 169.254.60.108 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.081/0.162/0.314 ms
root@agilexfm87:~#

Development Kit 2

root@agilexfm87:~# ping -c 3 169.254.83.72
PING 169.254.83.72 (169.254.83.72): 56 data bytes
64 bytes from 169.254.83.72: seq=0 ttl=64 time=0.348 ms
64 bytes from 169.254.83.72: seq=1 ttl=64 time=0.103 ms
64 bytes from 169.254.83.72: seq=2 ttl=64 time=0.075 ms

--- 169.254.83.72 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.075/0.175/0.348 ms
root@agilexfm87:~#

Repeat the same procedure for the eth2 interface in both development kits.

If you want to assign a different IP address to one interface, you can use the following commands:

root@agilexfm87:~# ip addr add 169.254.83.111/16 dev eth1

After adding a new IP address, the interface will have 2 addresses assigned to it, use the following command to remove the first IP address assigned to the interface:

root@agilexfm87:~# ip addr del 169.254.83.72/16 dev eth1

Verify PTP capability using ptp4l

To run a basic test we you must configure one development kit as a PTP master and the second development kit as a PTP slave. Both of them work as ordinary clocks.

The system example design include ptp4l configuration files in the Linux file system, you can access them in the following path:

root@agilexfm87:~# ls  /root/cfg/
boundary.cfg  
master.cfg    
slave.cfg

Start by configuring the development kit 1 as a master:

root@agilexfm87:~# ptp4l -i eth1 -m -f /root/cfg/master.cfg
ptp4l[6243.984]: selected /dev/ptp0 as PTP clock
[ 6244.065983] intel_fpga_eth 84480000.hssi_0_eth eth1: xtile_set_hwtstamp_config config flags:0x0, tx_type:0x1,c
ptp4l[6244.067]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[6244.067]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[6244.443]: port 1: LISTENING to MASTER on ANNOUNCE_RECEIPT_TIMEOUT_EXPIRES
ptp4l[6244.443]: selected local clock 6a193c.fffe.de0907 as best master
ptp4l[6244.443]: port 1: assuming the grand master role

Continue configuring the second development kit as a slave (press CTRL + c to stop the PTP measurements):

root@agilexfm87:~# ptp4l -i eth1 -m -s -f /root/cfg/slave.cfg
ptp4l[6432.463]: selected /dev/ptp0 as PTP clock
[ 6432.518054] intel_fpga_eth 84480000.hssi_0_eth eth1: xtile_set_hwtstamp_config config flags:c
ptp4l[6432.519]: port 1: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[6432.519]: port 0: INITIALIZING to LISTENING on INIT_COMPLETE
ptp4l[6432.614]: port 1: new foreign master 6a193c.fffe.de0907-1
ptp4l[6432.865]: selected best master clock 6a193c.fffe.de0907
ptp4l[6432.865]: port 1: LISTENING to UNCALIBRATED on RS_SLAVE
ptp4l[6432.895]: master offset          1 s0 freq      +0 path delay         0
ptp4l[6432.957]: master offset          1 s0 freq      +0 path delay        27
ptp4l[6433.020]: master offset          1 s0 freq      +0 path delay        27
ptp4l[6433.082]: master offset          2 s2 freq      +5 path delay        27
ptp4l[6433.082]: port 1: UNCALIBRATED to SLAVE on MASTER_CLOCK_SELECTED
ptp4l[6433.145]: master offset          1 s2 freq      +7 path delay        28
ptp4l[6433.207]: master offset          1 s2 freq      +7 path delay        28
ptp4l[6433.270]: master offset          0 s2 freq      +6 path delay        28
ptp4l[6433.332]: master offset          0 s2 freq      +6 path delay        27
ptp4l[6433.395]: master offset          0 s2 freq      +6 path delay        28
ptp4l[6433.457]: master offset         -1 s2 freq      +4 path delay        28
ptp4l[6433.520]: master offset         -1 s2 freq      +4 path delay        28
ptp4l[6433.583]: master offset         -1 s2 freq      +4 path delay        28
ptp4l[6433.645]: master offset         -1 s2 freq      +4 path delay        27
ptp4l[6433.708]: master offset         -2 s2 freq      +2 path delay        28
ptp4l[6433.770]: master offset         -1 s2 freq      +3 path delay        27
ptp4l[6433.833]: master offset         -2 s2 freq      +1 path delay        27
ptp4l[6433.895]: master offset         -2 s2 freq      +1 path delay        27
ptp4l[6433.958]: master offset         -2 s2 freq      +1 path delay        27
ptp4l[6434.020]: master offset         -2 s2 freq      +1 path delay        27
ptp4l[6434.083]: master offset         -2 s2 freq      +1 path delay        28
ptp4l[6434.145]: master offset         -2 s2 freq      +0 path delay        28
ptp4l[6434.208]: master offset         -2 s2 freq      +0 path delay        27
ptp4l[6434.270]: master offset         -2 s2 freq      +0 path delay        28
ptp4l[6434.333]: master offset         -2 s2 freq      -0 path delay        27
ptp4l[6434.395]: master offset         -2 s2 freq      -0 path delay        27
ptp4l[6434.458]: master offset         -2 s2 freq      -1 path delay        28
ptp4l[6434.520]: master offset         -2 s2 freq      -1 path delay        27
ptp4l[6434.583]: master offset         -1 s2 freq      +1 path delay        27
ptp4l[6434.645]: master offset         -2 s2 freq      -1 path delay        28
ptp4l[6434.708]: master offset         -2 s2 freq      -1 path delay        28
ptp4l[6434.770]: master offset         -1 s2 freq      +0 path delay        27
ptp4l[6434.833]: master offset         -1 s2 freq      +0 path delay        27
ptp4l[6434.895]: master offset         -1 s2 freq      +0 path delay        27
ptp4l[6434.958]: master offset         -1 s2 freq      -0 path delay        27
ptp4l[6435.021]: master offset         -1 s2 freq      -0 path delay        27
ptp4l[6435.083]: master offset         -2 s2 freq      -2 path delay        28
ptp4l[6435.146]: master offset         -1 s2 freq      -0 path delay        27
ptp4l[6435.208]: master offset         -1 s2 freq      -1 path delay        27
ptp4l[6435.271]: master offset         -1 s2 freq      -1 path delay        27
ptp4l[6435.333]: master offset         -1 s2 freq      -1 path delay        27
ptp4l[6435.396]: master offset         -2 s2 freq      -3 path delay        28
ptp4l[6435.458]: master offset         -1 s2 freq      -1 path delay        27
ptp4l[6435.521]: master offset         -1 s2 freq      -1 path delay        27
ptp4l[6435.583]: master offset         -1 s2 freq      -1 path delay        28
ptp4l[6435.646]: master offset         -1 s2 freq      -1 path delay        28
ptp4l[6435.708]: master offset         -1 s2 freq      -1 path delay        27
ptp4l[6435.771]: master offset         -1 s2 freq      -2 path delay        28
ptp4l[6435.833]: master offset         -1 s2 freq      -2 path delay        27
ptp4l[6435.896]: master offset         -1 s2 freq      -2 path delay        27
ptp4l[6435.958]: master offset         -1 s2 freq      -2 path delay        27
ptp4l[6436.021]: master offset         -1 s2 freq      -2 path delay        28
ptp4l[6436.083]: master offset          0 s2 freq      -0 path delay        27
ptp4l[6436.146]: master offset         -1 s2 freq      -2 path delay        28
ptp4l[6436.208]: master offset          0 s2 freq      -0 path delay        27
ptp4l[6436.271]: master offset          0 s2 freq      -0 path delay        27
ptp4l[6436.333]: master offset          0 s2 freq      -0 path delay        28
ptp4l[6436.396]: master offset          0 s2 freq      -0 path delay        27
ptp4l[6436.459]: master offset          0 s2 freq      -0 path delay        27
ptp4l[6436.521]: master offset          0 s2 freq      -0 path delay        27
ptp4l[6436.584]: master offset          0 s2 freq      -0 path delay        28
ptp4l[6436.646]: master offset          0 s2 freq      -0 path delay        27
ptp4l[6436.709]: master offset          0 s2 freq      -0 path delay        27
ptp4l[6436.771]: master offset          0 s2 freq      -0 path delay        27
ptp4l[6436.834]: master offset          0 s2 freq      -0 path delay        28
ptp4l[6436.896]: master offset          0 s2 freq      -0 path delay        28
ptp4l[6436.959]: master offset          0 s2 freq      -0 path delay        27
ptp4l[6437.021]: master offset          0 s2 freq      -0 path delay        27

Please refer to the ptp4l man page for more information on its use.

Running an iPerf3 test

You can run synthetic ethernet traffic between the development kits with iPerf3. The use of this tool is for testing purposes. To run a basic iPerf3 test between development kits, start configuring the first development kit as a server:

root@agilexfm87:~# iperf3 -s

To continue with the second development kit configuration, you must provide the IP address of the first development kit that is running as a server:

root@agilexfm87:~# iperf3 -M 1460 -c 169.254.83.111 -t 5 -R -b 650M

If everything runs properly with the test, the output from the first development kit will look like:

root@agilexfm87:~# iperf3 -s
-----------------------------------------------------------
Server listening on 5201 (test #1)
-----------------------------------------------------------
Accepted connection from 169.254.60.108, port 35802
[  5] local 169.254.83.111 port 5201 connected to 169.254.60.108 port 35808
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  77.5 MBytes   650 Mbits/sec    0    242 KBytes       
[  5]   1.00-2.00   sec  77.5 MBytes   650 Mbits/sec    0    329 KBytes       
[  5]   2.00-3.00   sec  77.5 MBytes   650 Mbits/sec    0    382 KBytes       
[  5]   3.00-4.00   sec  77.4 MBytes   649 Mbits/sec    0    474 KBytes       
[  5]   4.00-5.00   sec  77.5 MBytes   650 Mbits/sec    0    502 KBytes       
[  5]   5.00-5.00   sec   128 KBytes  1.21 Gbits/sec    0    502 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec   388 MBytes   650 Mbits/sec    0             sender

The output from the second development kit will be similar to:

root@agilexfm87:~# iperf3 -M 1460 -c 169.254.83.111 -t 5 -R -b 650M                             
Connecting to host 169.254.83.111, port 5201
Reverse mode, remote host 169.254.83.111 is sending
[  5] local 169.254.60.108 port 35808 connected to 169.254.83.111 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  77.5 MBytes   650 Mbits/sec                  
[  5]   1.00-2.00   sec  77.5 MBytes   650 Mbits/sec                  
[  5]   2.00-3.00   sec  77.5 MBytes   650 Mbits/sec                  
[  5]   3.00-4.00   sec  77.4 MBytes   649 Mbits/sec                  
[  5]   4.00-5.00   sec  77.5 MBytes   650 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-5.00   sec   388 MBytes   650 Mbits/sec    0             sender
[  5]   0.00-5.00   sec   387 MBytes   650 Mbits/sec                  receiver

iperf Done.

Packet loss can occur when running UDP traffic with iperf3. The system's HPS may not be able to handle high bandwidth traffic and could drop packets, which are not resent due to the nature of the protocol. Lowering the bitrate of the iperf3 UDP test can help reduce the number of dropped packets throughout the test.

Basic Configurations

The system example design is configured to take different PTP network roles. A set of example ptp4l configuration files are provided as part of the system example design. this configuration files can be found under the following path:

/root/cfg

Ordinary Clock

The system example design Ethernet ports is configured as a network ordinary clock that assume the role of master clock or subordinate clock. The Ethernet ports are independent between each other, and is configured with opposite roles at the same time.

Master Ordinary Clock

Under this configuration, the Ethernet port provides the timing reference to the downstream attached device which adjusts its local clock to synchronize with the master clock running in the development kit. Use the following syntax to configure one of the Ethernet interface as a master ordinary clock:

ptp4l -i <ethernet interface name>  -f /root/cfg/master.cfg

e.g.

ptp4l -i eth1 -m -f /root/cfg/master.cfg

Figure 8. Master ordinary clock topology.

Subordinate Ordinary Clock

Also known as slave ordinary clock. In this configuration the system example design connects to an upstream port and synchronize to its master clock time. The PTP software stack running in the system example design HPS uses the incoming PTP messages to continuously adjust its local clock to keep the synchronization between it and the network master. Use the following syntax to configure one of the Ethernet interface as a master ordinary clock:

ptp4l -i <ethernet interface name> -m -s -f /root/cfg/slave.cfg

e.g.

ptp4l -i eth2 -m -s -f /root/cfg/slave.cfg

Figure 9. Subordinate ordinary clock topology.

Boundary Clock

The system example design is configured as a network boundary clock. Under this configuration, one of the ethernet interfaces works as a subordinate PTP port while the second interface is configured as a master PTP port. The boundary clock configuration make sure that the time information provided by the subordinate port has been adjusted for the resident time in the system.

To exercise the boundary clock configuration you need one Development Kit and two other network partners. Follow the next steps to set the Development kit as a boundary clock:

Development kit ethernet interface 2 plays the role of master interface for the boundary clock. Start by configuring interface eth2 as a master clock using the following the next command:

ptp4l -i eth2 -f /root/cfg/master.cfg &

Keep the last process running in the background to keep control over the command line session. Continue by configuring interface eth1 as a subordinate clock. It is important that for boundary clock configuration the subordinate interface is eth1, as it is linked to Port 8 from the Ethernet Subsystem which feeds its RX recovered clock for SyncE back to the onboard ZL30733 device. Use the following command to configure eth1 interface as a subordinate clock:

ptp4l -i eth1 -m -s -f /root/cfg/slave.cfg

Connect the network grand master to the development kit ethernet interface number 1. Connect the subordinate ordinary clock device to the development kit ethernet interface 2. After the connections, the development kit will propagate the time information from the network grand master to the ordinary clock connected to the development kit.

The configuration of a development kit can be achieved with a single ptp4l configuration file. You can use boundary.cfg, included in the system example design release package, as a reference to create your own boundary configuration file. boundary.cfg may need adjustment to work under your specific network setup.

Figure 10. Boundary clock topology.

Build/Update the F-Tile System Example Design for the DK-SI-AGI027FB Development Kit

Building the Quartus System Example Design

Get a copy of ghrd_agib027r31b1e2v_23_2_0_94.qar from the 25G 2-Port System Example Design Release package or the 10G 2-Port System Example Design Release package.

The system example design was created and tested with the Intel® Quartus® Prime Pro Edition Design Software Version 23.2.

Restore the project archive with Intel Quartus Prime GUI or using the following command:

quartus_sh --restore ghrd_agib027r31b1e2v_23_2_0_94.qar -output ghrd_agib027r31b1e2v_23_2_0_94_restore

Compile the restored project with the Intel Quartus Prime GUI or through the command line with the following commands:

cd ghrd_agib027r31b1e2v_23_2_0_94_restore
quartus_sh --flow compile ghrd_agib027r31b1e2v.qpf

Building the System Example Design Software Binaries

Integrating U-Boot FSBL hex file to RBF file

The configuration bitstream generated after an Intel Quartus Prime compilation contains both the FPGA core and I/O sections, as well as the HPS First-Stage Bootloader (FSBL). Once the system example design is recompiled, you must integrate the '.hex' file containing the U-Boot FSBL into the new bitstream.

To integrate the '.hex' file into the new bitstream execute the following command:

quartus_pfg -c -o hps=on -o hps_path=<path_to_fslb>/<fslb_file_name>.hex ./<sof_file_name>).sof ./<output_file>.rbf

The command looks like:

quartus_pfg -c -o hps=on -o hps_path=software/hps_debug/u-boot-spl-dtb.hex ./output_files/ghrd_agib027r31b1e2v.sof ./output_files/ghrd_agib027r31b1e2v.rbf

The following files are generated:

• ghrd_agib027r31b1e2v.hps.rbf - HPS First configuration bitstream, phase 1 (HPS and DDR)
• ghrd_agib027r31b1e2v.core.rbf - HPS First configuration bitstream, phase 2 (FPGA fabric)

You can use the U-Boot FSBL hex file included in this system example design (yocto/u-boot-spl-dtb.hex).

For more information on how to build the U-Boot from scratch refer to the following link.

Building the Flattened uImage Tree from Source

The next section present the steps to build the software component for the system example design from its source files. The instructions in this section require a Host PC running Ubuntu 22.04 LTS. The steps may require adjustments while using a different OS on the host.

Setting up the Environment

Install the following dependencies in your building host:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install fakeroot ncurses-dev libssl-dev bc flex libelf-dev bison gawk wget git diffstat unzip texinfo gcc build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev python3-subunit mesa-common-dev zstd liblz4-tool file locales 
export LC_ALL="en_US.UTF-8"
export LC_CTYPE="en_US.UTF-8"
export LC_NUMERIC="en_US.UTF-8"
export LANG=en_US.UTF-8
export LANGUAGE=en_US.UTF-8

Make bash your default command line interpreter:

sudo ln -sf /bin/bash /bin/sh

For more information on the building environment and support for other OS refer to:

• Yocto Project Reference Manual
• GSRD for Agilex 7 I-Series Transceiver-SoC DevKit (4x F-Tile)

Building u-boot.itb

Get a copy of socfpga_fm87_ftile_25g_ptp.zip or socfpga_fm87_ftile_10g_ptp.zip from the appropiate release package. Restore the ZIP file with the following command for the 25G system example design:

unzip socfpga_fm87_ftile_25g_ptp.zip

Or the dollowing commandfor the 10G system example design:

unzip socfpga_fm87_ftile_10g_ptp.zip

After restoring the package, go to the building folder and start the compile process. For the 25G system example design execute:

cd yocto_ftile 
. agilex_fm87-gsrd-25G2P_build.sh
build_default 

For the 10G example design execute the following commands:

cd yocto_ftile 
. agilex_fm87-gsrd-10G2P_build.sh
build_default 

The build process time depends on the resource specifications of the Host been used to build the software. After a successful compilation process for the 25G system example design, the resulting binary files can be find in the next locations:

• socfpga_fm87_ftile_25g_2port_ptp.dtb → yocto_ftile/agilex_fm87-gsrd- rootfs/tmp/deploy/images/agilex_fm87/devicetree/socfpga_fm87_ftile_25g_2port_ptp.dtb
• kernel_fm87.itb → yocto_ftile/agilex_fm87-gsrd-rootfs/tmp/deploy/images/agilex_fm87/
• u-boot.itb → yocto_ftile/agilex_fm87-gsrd-rootfs/tmp/deploy/images/agilex_fm87/
• u-boot.txt → yocto_ftile/agilex_fm87-gsrd-rootfs/tmp/deploy/images/agilex_fm87/
• u-boot-spl-dtb.hex → yocto_ftile/agilex_fm87-gsrd-rootfs/tmp/deploy/images/agilex_fm87/
• sdimage.tar.gz → yocto_ftile/agilex_fm87-gsrd-images/

After a successful compilation process for the 10G system example design, the resulting binary files can be find in the next locations:

• socfpga_fm87_ftile_10g_2port_ptp.dtb → yocto_ftile/agilex_fm87-gsrd- rootfs/tmp/deploy/images/agilex_fm87/devicetree/socfpga_fm87_ftile_10g_2port_ptp.dtb
• kernel_fm87.itb → yocto_ftile/agilex_fm87-gsrd-rootfs/tmp/deploy/images/agilex_fm87/
• u-boot.itb → yocto_ftile/agilex_fm87-gsrd-rootfs/tmp/deploy/images/agilex_fm87/
• u-boot.txt → yocto_ftile/agilex_fm87-gsrd-rootfs/tmp/deploy/images/agilex_fm87/
• u-boot-spl-dtb.hex → yocto_ftile/agilex_fm87-gsrd-rootfs/tmp/deploy/images/agilex_fm87/
• sdimage.tar.gz → yocto_ftile/agilex_fm87-gsrd-images/

Rebuild u-boot.itb with a new FPGA Bitstream

If changes are made to the hardware project, adding Signal Tap for example, you must rebuild the HPS software. The HPS second stage bootloader have the FPGA core bitstream SHA signature embedded in the compile process, with an bitstream update the SHA calculation change and needs to be updated in the second stage bootloader.

Follow the next steps to update the FPGA core bitstream used in the HPS second stage bootloader:

Step 1

Rename your 'core.rbf' to agilex_fm87_gsrd_ghrd_25G2P.core.rbf or agilex_fm87_gsrd_ghrd_10G2P.core.rbf as appropriate. The build process looks for a 'core.rbf' named agilex_fm87_gsrd_ghrd_25G2P.core.rbf or agilex_fm87_gsrd_ghrd_10G2P.core.rbf, if any other name is used the building process will fail.

Replace agilex_fm87_gsrd_ghrd_25G2P.core.rbf or agilex_fm87_gsrd_ghrd_10G2P.core.rbf from the release package with the new 'core.rbf' file from your compile. RBF files can be found in the following path of the release package:

yocto_ftile/meta-fm87-ftile-ptp/recipes-bsp/ghrd/files/

Step 2

Generate the SHA signature for the new 'core.rbf' and update hw-ref-design.bbappend

To generate the SHA signature of the new 'core.rbf' with the next command (use agilex_fm87_gsrd_ghrd_10G2P.core.rbf for the 10GE system example design):

 sha256sum agilex_fm87_gsrd_ghrd_25G2P.core.rbf | cut -f1 -d" " 

The previous command will provide a 64 byte SHA signature, use the following as an example:

4566d998d949ba33fc44c0fdb8beecacf06d5d901b706fe4ae0c75f15fc5e014

Step 3

Update the SHA signature in hw-ref-design.bbappend with your preferred test editor. You can find hw-ref-design.bbappend in the following path:

yocto_ftile/meta-fm87-ftile-ptp/recipes-bsp/ghrd/hw-ref-design.bbappend

Open hw-ref-design.bbappend and look for the following line for the 25G system example design:

sha256sum_25G2P = "ea9f20f20734515ee90e85cb87c0b1d2f51204049c31cf6fdc60bf8fc4e958ee"

For the 10G system example design, look for the following line:

sha256sum_10G2P = "d70631e1aae4715057cc394d77a7fa6e0f41fca8c393bb552c005bd7f556ea04"

Update the 64 byte SHA value with the calculation from Step 2.

Step 4

Start the software building process.

Refer to section Building the Flattened uImage Tree from Source for setting the building environment before start the build process.

For the 25G system example design execute:

. agilex_fm87-gsrd-25G2P_build.sh
build_default 

For the 10G system example design execute:

. agilex_fm87-gsrd-10G2P_build.sh
build_default 

Build/Update the F-Tile System Example Design for the DK-SI-AGI027FA Development Kit

Building the Quartus System Example Design

Get a copy of ghrd_agib027r31b1e2v_23_2_0_94.qar from the 25G 2-Port System Example Design Release package or the 10G 2-Port System Example Design Release package.

The system example design was created and tested with the Intel® Quartus® Prime Pro Edition Design Software Version 23.2.

Restore the project archive with Intel Quartus Prime GUI or using the following command:

quartus_sh --restore ghrd_agib027r31b1e2v_23_2_0_94.qar -output ghrd_agib027r31b1e2v_23_2_0_94_restore
cd ghrd_agib027r31b1e2v_23_2_0_94_restore

On the restored project folder, open ghrd_agib027r31b1e2v.qsf file with you prefered text editor replace the following assigments:

Replace the target device from:

set_global_assignment -name DEVICE AGIB027R31B1E2V 

to:

set_global_assignment -name DEVICE AGIB027R31B1E1V

Update the power settings assigments from:

set_global_assignment -name PWRMGT_PAGE_COMMAND_ENABLE OFF
set_global_assignment -name PWRMGT_SLAVE_DEVICE_TYPE ED8401
set_global_assignment -name PWRMGT_SLAVE_DEVICE0_ADDRESS 62
set_global_assignment -name PWRMGT_SLAVE_DEVICE1_ADDRESS 00
set_global_assignment -name PWRMGT_SLAVE_DEVICE2_ADDRESS 00
set_global_assignment -name PWRMGT_VOLTAGE_OUTPUT_FORMAT "LINEAR FORMAT"
set_global_assignment -name PWRMGT_LINEAR_FORMAT_N "-13"

to:

set_global_assignment -name PWRMGT_SLAVE_DEVICE_TYPE LTC3888
set_global_assignment -name PWRMGT_SLAVE_DEVICE0_ADDRESS 62
set_global_assignment -name PWRMGT_VOLTAGE_OUTPUT_FORMAT "LINEAR FORMAT"
set_global_assignment -name PWRMGT_LINEAR_FORMAT_N "-12"

Compile the restored project with the Intel Quartus Prime GUI or through the command line with the following commands:

quartus_sh --flow compile ghrd_agib027r31b1e2v.qpf

Building the System Example Design Software Binaries

Integrating U-Boot FSBL hex file to RBF file

The configuration bitstream generated after an Intel Quartus Prime compilation contains both the FPGA core and I/O sections, as well as the HPS First-Stage Bootloader (FSBL). Once the system example design is recompiled, you must integrate the '.hex' file containing the U-Boot FSBL into the new bitstream.

To integrate the '.hex' file into the new bitstream execute the following command:

quartus_pfg -c -o hps=on -o hps_path=/.hex ./).sof ./.rbf

The command looks like:

quartus_pfg -c -o hps=on -o hps_path=software/hps_debug/u-boot-spl-dtb.hex ./output_files/ghrd_agib027r31b1e2v.sof ./output_files/ghrd_agib027r31b1e2v.rbf

The following files are generated:

• ghrd_agib027r31b1e2v.hps.rbf - HPS First configuration bitstream, phase 1 (HPS and DDR)
• ghrd_agib027r31b1e2v.core.rbf - HPS First configuration bitstream, phase 2 (FPGA fabric)

You can use the U-Boot FSBL hex file included in this system example design (yocto/u-boot-spl-dtb.hex).

For more information on how to build the U-Boot from scratch refer to the following link.

Rebuild u-boot.itb

Follow the next steps to update the FPGA core bitstream used in the HPS second stage bootloader:

Step 1

Rename your new 'core.rbf' to agilex_fm87_gsrd_ghrd_25G2P.core.rbf or agilex_fm87_gsrd_ghrd_10G2P.core.rbf as appropriate. The build process looks for a 'core.rbf' named agilex_fm87_gsrd_ghrd_25G2P.core.rbf or agilex_fm87_gsrd_ghrd_10G2P.core.rbf, if any other name is used the building process will fail.

Replace agilex_fm87_gsrd_ghrd_25G2P.core.rbf or agilex_fm87_gsrd_ghrd_10G2P.core.rbf from the release package with the new 'core.rbf' file generated with quartus_pfg. RBF files can be found in the following path of the release package:

yocto_ftile/meta-fm87-ftile-ptp/recipes-bsp/ghrd/files/

Step 2

Generate the SHA signature for the new 'core.rbf' and update hw-ref-design.bbappend

To generate the SHA signature of the new 'core.rbf' with the next command (use agilex_fm87_gsrd_ghrd_10G2P.core.rbf for the 10GE system example design):

 sha256sum agilex_fm87_gsrd_ghrd_25G2P.core.rbf | cut -f1 -d" " 

The previous command will provide a 64 byte SHA signature, use the following as an example:

4566d998d949ba33fc44c0fdb8beecacf06d5d901b706fe4ae0c75f15fc5e014

Step 3

Update the SHA signature in hw-ref-design.bbappend with your preferred test editor. You can find hw-ref-design.bbappend in the following path:

yocto_ftile/meta-fm87-ftile-ptp/recipes-bsp/ghrd/hw-ref-design.bbappend

Open hw-ref-design.bbappend and look for the following line for the 25G system example design:

sha256sum_25G2P = "ea9f20f20734515ee90e85cb87c0b1d2f51204049c31cf6fdc60bf8fc4e958ee"

For the 10G system example design, look for the following line:

sha256sum_10G2P = "d70631e1aae4715057cc394d77a7fa6e0f41fca8c393bb552c005bd7f556ea04"

Update the 64 byte SHA value with the calculation from Step 2.

Step 4

Start the software building process.

Refer to section Building the Flattened uImage Tree from Source for setting the building environment before start the build process.

For the 25G system example design execute:

. agilex_fm87-gsrd-25G2P_build.sh
build_default 

For the 10G system example design execute:

. agilex_fm87-gsrd-10G2P_build.sh
build_default  

Debug Checklist

This section helps you identify and resolve common problems that may occur when bringing up this design.

eth1 and eth2 are not detected by Linux

After login into the HPS, the Ethernet ports are not listed by Linux:

root@agilexfm87:~# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
 inet 127.0.0.1/8 scope host lo
 valid_lft forever preferred_lft forever
 inet6 ::1/128 scope host 
 valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
 link/ether 52:0e:ec:7b:99:7e brd ff:ff:ff:ff:ff:ff
 inet 10.122.105.175/24 brd 10.122.105.255 scope global eth0
 valid_lft forever preferred_lft forever
 inet6 fe80::500e:ecff:fe7b:997e/64 scope link 
 valid_lft forever preferred_lft forever

There are multiples causes for this error.

Incorrect onboard MAX10 bitstream

The custom Intel MAX10 bitstream is not loaded, causing the ZL30733 configuration fail. Review the Linux boot up transcript for the next message:

Loading fpga from 0x02840620 to 0x0a000000
.......FPGA reconfiguration OK!
 Initializing Clock Cleaner (ZL30733) via I2C 
zl30733_i2c_init: Could not identify ZL30733 chip on I2C bus 0, address 0x70
.......FPGA reconfiguration OK!
Enable FPGA bridges

To workaround this issue, load the max10_system_0002aa4F.pof following the steps described in Hardware Setup section of this guide. Once the correct Intel MAX10 bitstream is loaded, the transcript looks like:

Loading fpga from 0x02840620 to 0x0a000000
.......FPGA reconfiguration OK!
 Initializing Clock Cleaner (ZL30733) via I2C 
...Configuring PLL Done!
.......FPGA reconfiguration OK!
Enable FPGA bridges

The Quality of Ethernet Link is not Optimal

The System Example Design doesn't implement auto-Negotiation and Link Training, and comes with a fixed set of analog setting tuned for a FS (Q28-PC01) - 1m (3ft) 100G QSFP28 Passive Direct Attach Copper Twinax Cable. Using different cables (lenght, optical, vendor) may required manual tunning of the analog setting for the Ethernet interfaces.

Before moving forward with any debug activity, make sure that the cables are properly connected to both development kits.

Start getting a snapshot of the Ethernet links health with the procedure described in 'Reading the Ethernet Subsystem IP Configuration and Status Registers with the HPS' section from this document. If the the status of a port is not optimal, you may be required to adjust the analog settings for the Ethernet port if a DAC cable is being use. Please refer to section Enabling Transceiver Tool Kit for the Ethernet Subsystem of this document and execute a BER test and an Eye Viewer test. If the results are not optimal, follow the recommendations described in 7.2.7. Running Link Optimization Tests from the F-Tile Architecture and PMA and FEC Direct PHY IP User Guide.

If you are using an optical cable, you may need to add the QSF assignments listed in QSFP Cable Compatibility and recompile the Intel Quartus Prime project for this system reference design.

System Debug Tools

Reading the Ethernet Subsystem IP Configuration and Status Registers with the HPS

The HPS exposes the Configuration and Status Registers (CSR) from the Ethernet Subsystem IP through the Linux file system. The Ethernet Subsystem IP CSR provides a snapshot of the status of the Ethernet interfaces. You may use this information to debug potential issues with the Ethernet interfaces.

To generate a snapshot of the Ethernet Subsystem IP CSR, login into your system HPS through a serial session and execute the following command:

root@agilexfm87:~# cat /sys/kernel/debug/hssiss_dbg/dumpcsr

The output is similar to the following transcript:

Dumping device feature registers
 0: 10003015
 4: 30000000
 8: 18418b9d
 c: 99a078ad
 10: d9db4a9b
 14: 4118a7cb
 18: c0
 1c: 0
 20: 0
 24: 31c
Dumping other CSR registers
HSSISS_CSR_VER: 20000
HSSISS_CSR_COMMON_FEATURE_LIST: c005
Dumping port attributes
 0: 0
 1: 0
 2: 0
 3: 0
 4: 0
 5: 0
 6: 0
 7: 0
 8: a40415
 9: a40415
 a: 0
 b: 0
 c: 0
 d: 0
 e: 0
 10: 0
 11: 0
 12: 0
 13: 0
HSSISS_CSR_CMDSTS: 5
HSSISS_CSR_CTRLADDR: 50020c06
HSSISS_CSR_RD_DATA: f
HSSISS_CSR_WR_DATA: 9ee077
HSSISS_CSR_GMII_TX_LATENCY: 0
HSSISS_CSR_GMII_RX_LATENCY: 0
Dumping port status
 0: 0
 1: 0
 2: 0
 3: 0
 4: 0
 5: 0
 6: 0
 7: 0
 8: a00180
 9: e00194
 a: 0
 b: 0
 c: 0
 d: 0
 e: 0
 10: 0
 11: 0
 12: 0
 13: 0
HSSISS_CSR_TSE_CTRL: 0
HSSISS_CSR_DBG_CTRL: 62000
HSSISS_CSR_HOTPLUG_DBG_CTRL: f000000
HSSISS_CSR_HOTPLUG_DBG_STS: 0

To decode the information dumped by 'dumpcsr' you can refer to section '7. Register Descriptions' from the Ethernet Subsystem Intel® FPGA IP User Guide. For example, offsets shown below 'Dumping device feature registers' correspond to the register description from section '7.1 Subsystem Registers'.

For debug purposes, the register contents shown after 'Dumping port status' are the most useful. Offsets with a read value of '0' represent Ethernet Subsystem ports that are not being used by the system example design.

Dumping port status
.
.
8: a00180
9: e00194
.
.

To decode the information for port 8 and port 9 you must use the information from section '7.1.14 HSSI Ethernet Port X Status'.

Port 8 status decode (0xA00180):

Bit	'dumpcsr' Value	Description
31:27	0	Reserved
26	0	No parity errors detected
25:24	0	Not applicable for F-Tile
23	1	tx_pll_locked status bit
22	0	rx_pcs_ready status bit
21	1	tx_lanes_stable status bit
20	0	Not applicable for F-Tile
19	0	Not applicable for F-Tile
18	0	Reserved
17	0	Reserved
16	0	Reserved
15	0	Reserved
14:13	0	Reserved
12:11	0	Reserved
10	0	tx_unidir_control register bit 1 status - unidirectional remote fault disable
9	0	tx_unidir_control register bit 2 status - unidirectional force remote fault
8	1	Remote Fault status bit
7	1	Local Fault status bit
6	0	Unidiectional enable status bit (Clause 66)
5	0	Link Fault Generation Enable status bit (Clause 66)
4	0	rx_block_lock status bit
3	0	rx_am_lock status bit
2	0	o_cdr_lock status bit
1	0	o_rx_hi_ber status bit
0	0	Reserved

Port 9 status decode (0xE00194):

Bit	'dumpcsr' Value	Description
31:27	0	Reserved
26	0	No parity errors detected
25:24	0	Not applicable for F-Tile
23	1	tx_pll_locked status bit
22	1	rx_pcs_ready status bit
21	1	tx_lanes_stable status bit
20	0	Not applicable for F-Tile
19	0	Not applicable for F-Tile
18	0	Reserved
17	0	Reserved
16	0	Reserved
15	0	Reserved
14:13	0	Reserved
12:11	0	Reserved
10	0	tx_unidir_control register bit 1 status - unidirectional remote fault disable
9	0	tx_unidir_control register bit 2 status - unidirectional force remote fault
8	1	Remote Fault status bit (Only functional if feature is enabled)
7	1	Local Fault status bit (Only functional if feature is enabled)
6	0	Unidiectional enable status bit (Clause 66)
5	0	Link Fault Generation Enable status bit (Clause 66)
4	1	rx_block_lock status bit
3	0	rx_am_lock status bit
2	1	o_cdr_lock status bit
1	0	o_rx_hi_ber status bit
0	0	Reserved

From the previous decoding, you may conclude that something is wrong with Port 8. Port 8 TX side seems to be working properly as tx_pll_locked and tx_lanes_stable are asserted. however, the RX side is not ready as its CDR haven't been able to recover a clock signal from the data provided by the link partner.

Port 9 is not flagging any issues for its TX and RX paths. It is worth noting that bit 8 (Remote Fault status) and bit 9 (Local Fault status) are asserted, both bits latch any occurrence of a fault status and must be cleared manually. A remote fault have happened when the system was initializing and it never got cleared, so those signals are not a concern if the link is up.

Bits 23, 22, 21, 4, 5, 3,2, and 1 show the status of the TX and RX Ethernet interfaces for a single port. This information can be used to narrow down if the problem with the Ethernet link is a hardware or software issue, and if it is affecting the TX, RX or both sides of the interface.

For more information about the status signals mentioned in this section, refer to section 7.1 Status Interface from the F-Tile Ethernet Intel® FPGA Hard IP User Guide.

Access to the Ethernet Subsystem IP Configuration and Status Registers with the HPS

Access to the Ethernet Subsystem CSR is possible throught the HPS. The following syntax is used to read and write the CSR registers, it is worth noting that the instructions described below need to be executed under the following path:

/sys/kernel/debug/hssiss_dbg

Syntax for write operations:

echo "<base> <offset> <direct> <wr_value>" > direct_reg

Syntax for read operations:

echo "<base> <offset> <direct> <wr_value>" > direct_reg
cat direct_reg

where:

Parameter	Description
base	Base address from the Ethernet Subsystem IP. The Address map can be found in section 7.3. F-Tile Address Maps from the Ethernet Subsystem Intel® FPGA IP User Guide
offset	Register offset to access. Refer to the F-Tile Ethernet Intel® FPGA Hard IP Register for an address and description of register belonging to an Ethernet Port from the Ethernet Subsystem IP
direct	Set this parameter to '1' for access to registers from one of the Ethernet Ports of the Ethernet Subsystem IP, or '0' for access to a Subsystem CSR
wr_value	Data used for write operations

Executing a soft global reset on Port 8 from the Ethernet Subsystem

Follow the next steps to execute a soft global reset Port 8 (eth1 Ethernet interface).

Set bit [0] of eth_reset(0x108) register from Port 8 to start a soft global reset on the interface:

root@agilexfm87:/sys/kernel/debug/hssiss_dbg# /sys/kernel/debug/hssiss_dbg# echo "0x1200000 0x108 1 1" > direct_reg
[ 2348.950123] intel_fpga_eth 84480000.hssi_0_eth eth1: Link is Down

Interface eth1 goes down after the write:

root@agilexfm87:/sys/kernel/debug/hssiss_dbg# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d2:e6:90:96:1e:b4 brd ff:ff:ff:ff:ff:ff
    inet 10.122.105.199/24 metric 10 brd 10.122.105.255 scope global dynamic eth0
       valid_lft 42683sec preferred_lft 42683sec
    inet6 fe80::d0e6:90ff:fe96:1eb4/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast state DOWN group default qlen 1000
    link/ether f6:12:8e:6c:b7:a0 brd ff:ff:ff:ff:ff:ff
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 46:0c:68:ed:21:46 brd ff:ff:ff:ff:ff:ff
    inet 169.254.69.173/16 brd 169.254.255.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::440c:68ff:feed:2146/64 scope link 
       valid_lft forever preferred_lft forever

Read register eth_reset_status(0x10C) for reset acknowledge from the reset controller:

root@agilexfm87:/sys/kernel/debug/hssiss_dbg# /sys/kernel/debug/hssiss_dbg# echo "0x1200000 0x10C 1" > direct_reg
root@agilexfm87:/sys/kernel/debug/hssiss_dbg# cat direct_reg
1005500

Register eth_reset_status returned value is in decimal format. It confirms that the Ethernet interface is going through a reset cycle.

Clear bit [0] of eth_reset(0x108) register from Port 8 to take the ethernet interface out of global reset:

root@agilexfm87:/sys/kernel/debug/hssiss_dbg# /sys/kernel/debug/hssiss_dbg# echo "0x1200000 0x108 1 0" > direct_reg
[ 2485.718184] intel_fpga_eth 84480000.hssi_0_eth: DBG: eth_ftile_tx_rx_user_flow speed=25000 num_vl=1 num_fl=1 num_pl=1
[ 2485.729538] intel_fpga_eth 84480000.hssi_0_eth: DBG: eth_ftile_tx_rx_user_flow ETH_TX_PTP_READY - tx_ref_pl:0 tx_extra_latency:0x00031072 tx_tam_adjust:-1640482
[ 2485.751985] intel_fpga_eth 84480000.hssi_0_eth: DBG: eth_ftile_tx_rx_user_flow ETH_RX_PTP_READY - rx_ref_pl:0 rx_extra_latency:0x800369d0 rx_tam_adjust:-1706917
[ 2485.766323] intel_fpga_eth 84480000.hssi_0_eth eth1: Link is UP
[ 2485.772369] IPv6: ADDRCONF(NETDEV_CHANGE): eth1: link becomes ready

Read register eth_reset_status(0x10C) to confirm that the reset cycle has finished:

root@agilexfm87:/sys/kernel/debug/hssiss_dbg# /sys/kernel/debug/hssiss_dbg# echo "0x1200000 0x10C 1" > direct_reg
root@agilexfm87:/sys/kernel/debug/hssiss_dbg# cat direct_reg
1006607

Confirm that interface eth1 is UP in the system:

root@agilexfm87:/sys/kernel/debug/hssiss_dbg# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether d2:e6:90:96:1e:b4 brd ff:ff:ff:ff:ff:ff
    inet 10.122.105.199/24 metric 10 brd 10.122.105.255 scope global dynamic eth0
       valid_lft 42545sec preferred_lft 42545sec
    inet6 fe80::d0e6:90ff:fe96:1eb4/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether f6:12:8e:6c:b7:a0 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::f412:8eff:fe6c:b7a0/64 scope link 
       valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 46:0c:68:ed:21:46 brd ff:ff:ff:ff:ff:ff
    inet 169.254.69.173/16 brd 169.254.255.255 scope global eth2
       valid_lft forever preferred_lft forever
    inet6 fe80::440c:68ff:feed:2146/64 scope link 
       valid_lft forever preferred_lft forever

Enabling Transceiver Tool Kit for the Ethernet Subsystem

The system example design supports the F-Tile Transceiver Toolkit for debugging potential link quality issues. Follow the next steps to enable the Transceiver Toolkit:

1. With Intel Quartus Prime Pro version 23.2, open the system example design project. If you haven't restore the Intel Quartus Prime project, follow the steps described in Building the Quartus System Example Design without compiling the project.
2. In 'Project Navigator' click on 'IP Components' Tab.
3. Double click on the entity 'hssi_ss_1', the IP Parameter Editor will open the Ethernet Subsystem IP instance.
4. In the 'HSSI Subsystem' >> 'Device 0 Configuration' >> 'Main Configuration' tab, set to 'Enable' the 'Enable JTAG to Avalon Master Bridge' parameter. Refer to the screen shot below.
5. In the 'HSSI Subsystem' >> 'Device 0 Configuration' >> 'F-Tile IP Configuration' >> 'Port 8 Configuration' >> 'P8 IP' tab, click on the 'Enable debug endpoint for transceiver toolkit' parameter.
6. repeat step 5 for 'Port 9 Configuration' >> 'P9 IP' tab. Refer to the screen shot below.
7. Save and regenerate the IP.
8. Recompile the Intel Quartus Prime project.
9. Regenerate the software with the new generated 'core.rbf' file.

Refer to section '7.2. F-Tile Transceiver Debugging Flow Walkthrough' from the 'F-Tile Architecture and PMA and FEC Direct PHY IP User Guide' for more information on link quality related issues and their resolution. Sections '7.2.5. Running BER Tests' and '7.2.6. Running Eye Viewer Tests' are essential to qualify the Ethernet link health.

Known Issues

Issue	Workaround	Notes
Platform Designer may fail while performing a 'Sync System Infos' on qsys_top.qsys	There is no workaround available at this time. Avoid running 'Sync System Infos' on qsys_top.qsys	Error tracking number: 14021462395

Revision History

Version	Date	Comment
1.0		Initial release