Zynq UltraScale+ (Kria K26) firmware for the PL
This repo is a collection of firmware modules used for the DAPHNE_MEZZ board. This board uses the Zynq UltraScale+ (Kria) device.
- Front End Alignment and Synchronization Logic (RTL)
- Input and Output Spy Buffers (RTL)
- Timing Endpoint (RTL)
- SPI master for stand alone DAC chips (RTL)
- SPI master for AFE chips and associated DAC chips (RTL)
- SPI master for the current monitor (RTL)
- I2C master for a bunch of devices (Xilinx IP)
- Misc stuff (RTL)
- Streaming Core Logic (RTL)
- Self-Triggered Core Logic (RTL)
These functional blocks tie together using AXI-LITE interconnect buses so that the processor side of the Zynq (PS) can access them.
The front end de-serialization and alignment logic has changed significantly since DAPHNE2. In that version I had complex state machines to do the "automatic" alignment. These state machines have been removed and replaced with an AXI-LITE interface which provides access to 13 registers that control the front end logic. The idea here is that we will write a program (or script) that runs in user-land and this program controls the alignment process. This program will work closely with the front end and spy buffers to capture, readout, and evaluate the alignment.
There are 13 32-bit registers that control the front end. Most registers are read-write and only a few of the 32 bits in each register are used. The address of these registers is relative to the base address assigned to the front end AXI-LITE instance.
base+0 = Control Register is R/W
bits 31..3 = don't care
bit 2 = idelay_en_vtc
bit 1 = iserdes_reset
bit 0 = idelayctrl_reset
base+4 = Status Register is R/O
bits 31..1 = zero
bit 0 = idelayctrl_ready
base+8 = Trigger Register W/O
Write anything to the Trigger Register to force a momentary pulse
on the TRIG output. This will force the SPY BUFFERS to capture the
raw input data. This register is write only and the data you write
here doesn't matter.
base+12 = AFE0 Delay Tap Register all are R/W
base+16 = AFE1 Delay Tap Register
base+20 = AFE2 Delay Tap Register
base+24 = AFE3 Delay Tap Register
base+28 = AFE4 Delay Tap Register
bits 31..9 = don't care
bits 8..0 = IDELAY delay tap value, range 0-511, used for "fine" bit alignment
base+32 = AFE0 Bitslip Register all are R/W
base+36 = AFE1 Bitslip Register
base+40 = AFE2 Bitslip Register
base+44 = AFE3 Bitslip Register
base+48 = AFE4 Bitslip Register
bits 31..4 = don't care
bits 3..0 = ISERDES "bitslip" value, used for "coarse" word alignment
The input spy buffer has 4x9 (45) 16-bit inputs. There is a trigger input and an AXI-LITE interface for communication. There are only two registers defined for this module. The control/status register (addr base+0) is used to ARM the module and select which input bus to capture. The value to write here has the AFE number (0-4) in the upper nibble and the AFE channel number (0-8) in the lower nibble. Remember, AFE channel 8 is the special "frame marker which is always seven 1s and seven 0s.
Reading from the control/status register returns the current state of the spy buffer. Once it is ARMED, it must be triggered by asserting the TRIGGER input. It then captures 4096 samples and waits to be armed again. Read the data register at address base+4 repeatedly to read the captured data. AXI-LITE data is always 32 bits and each 32 bit data word contains ONE sample to keep things simple. The capture depth can be changed with a VHDL generic at build time.
The status of the spy buffer is reported in the uppermost nibble of the status register.
"0001" when (state=rst) else -- in reset
"0010" when (state=wait4arm) else -- idle; waiting to be armed by the user
"0011" when (state=fifo_clear) else -- flushing the FIFO
"0100" when (state=fifo_wait) else -- pause after FIFO flush
"0101" when (state=wait4trig) else -- waiting to be triggered
"0110" when (state=store) else -- capturing data but the FIFO is not yet FULL
Normally, the input spy buffer will be idle in state "wait4arm" (2). Once armed, it will move to state "wait4trig" (5). Once it has been triggered it will move back to idle "wait4arm" (2) again. Now you can read the captured data by reading the data register (base+4) repeatedly.
The output spy buffer has eight 64-bit inputs. There is an AXI-LITE interface for communication. There are only two registers defined for this module. The control/status register (address base+0) is used to ARM the module and select which output stream to capture. Reading from the control/status register returns the current state of the spy buffer. Once it is ARMED, the spy buffer looks for the LAST word of a frame, then it then captures the next N VALID samples and waits to be armed again. Read the data register (address base+4) repeatedly to read the captured data. AXI-LITE data is always 32 bits. The captured 64-bit data is read LOW word first, then HIGH word, etc. The capture depth N default is 1024 samples but can be changed at build time with a VHDL generic.
The status of the spy buffer is reported in the uppermost nibble of the status register.
"0001" when (state=rst) else -- in reset
"0010" when (state=wait4arm) else -- idle; waiting to be armed by the user
"0011" when (state=fifo_clear) else -- flushing the FIFO
"0100" when (state=fifo_wait) else -- pause after FIFO flush
"0101" when (state=wait4last) else -- waiting for the LAST marker
"0110" when (state=store) else -- capturing data but the FIFO is not yet FULL
Normally, the input spy buffer will be idle in state "wait4arm" (2). Once armed, it will move to state "wait4last" (5). If output records are flowing, it should very quickly move to state "store" and then back to idle "wait4arm" (2) again. Now you can read the captured data by reading the data register (base+4) repeatedly.
The timing endpoint firmware is largely unchanged since DAPHNEv2. The output clocks have changed to include 125MHz and the high speed clock changes from 437.5MHz to 500MHz. Endpoint registers are now accessed through an AXI-LITE interface. Note that the AXI registers have changed, there are now fewer registers: clock control register, clock status register, endpoint control register, and endpoint status register. The address of these registers is relative to the base address of the AXI-LITE instance in the timing endpoint module.
base+0 = clock control register R/W
bits 31..3: don't care
bit 2: clock source (0=local, 1=use endpoint)
bit 1: MMCM1 reset
bit 0: general "soft" reset from user
base+4 = clock status register R/O
bits 31..2: zero
bit 1: MMCM1 locked
bit 0: MMCM0 locked
base+8 = endpoint control register R/W
bits 31..17: don't care
bit 16: endpoint reset
bits 15..0: endpoint address
base+12 = endpoint status register R/O
bits 31..5: zero
bit 4: endpoint timestamp ok
bits 3..0: endpoint state machine status "good to go!"
There are three serial DAC chips daisy chained on a single SPI interface. This block has a single AXI-LITE interface. There is no readback capability on these DACs, they are write only, and all three DAC chips must be written at once. The DAC chips are AD5327BRUZ-REEL7.
There are five AFE chips on the board and each AFE has four serial DAC chips (2 offset, 2 trim). AFE0 (and 4 DAC chips) is on it's own SPI interface. AFE1 and AFE2 share an SPI interface, as does AFE2 and AFE3. There is a total of 3 SPI masters in this module. This block has a single AXI-LITE interface with several registers through which all 5 AFEs and all 20 DAC chips can be programmed. AFE registers can be read back but the DAC chips are write only. Also, DAC chips are daisy chained in pairs and must be written together. The AFE chips are AFE5808AZCF and the DAC chips are AD5327BRUZ-REEL7.
The Current Monitor is ADS1261IRHBT and is on a dedicated SPI interface.
The I2C master communicates with lots of devices all over the board. This block is Xilinx IP and has a single AXI-LITE interface.
This module is a "catch all" place where various minor functions are grouped together and accessed through a single AXI-LITE interface. This includes:
- fan PWM speed control and monitoring
- vbias enable signal
- analog mux enables and select lines
- user LEDs
There are eight streaming mode senders, each sender handles four input streams. There is a programmable mux located between this core and the front end module. This programmable mux has an AXI-LITE interface and allows the user to connect any input channel (0-39) to the 8x4 streaming sender inputs.
The input mux has 45 inputs (5 AFE x 9 ch x 16 bit) inputs and 32 outputs (8 senders x 4 ch x 14 bits) which feed into the streaming senders. It is fully programmable and supports a few test modes. Each output as a corresponding 8 bit control register (muxctrl) which is also output to the streaming senders.
There are 40 independent self-triggered sender modules (STC3.VHD). Each sender computes an average baseline, and the each sender has a unique threshold value (10 bits). A simple algo is used for a trigger condition: one sample below (baseline+theshold) followed by one sample above (baseline+theshold).
Two HERMES channels are used for this sender. Each HERMES input is fed by a 20 self-triggered senders using a round robin selection logic block.
The thresholds can be changed per channel, by reading and writing to this module via the AXI interface. The thresholds are 10 bits, default is 0x3FF, which disables the corresponding self-triggered sender.
This package file contains some constants and user defined data types.
Constraints and other build related files will go in the xilinx directory.
Procedure TBD...
The front end alignment primitives have changed quite a bit since the DAPHNEv2 (Artix 7) firmware was designed. But fundamentally the logic works the same way. Each AFE device has 8 data outputs and one "frame" marker output. The frame marker has the same timing as the data outputs, but it always outputs a fixed pattern "1111111100000000". The frame marker is considered to be a 9th data bit for each AFE. This frame pattern is the key for making the front end alignment work: whatever delays and bitslips are done to make the frame pattern look correct are automatically applied to the other 8 data bits from the same AFE chip. Since the routing delays on all 9 LVDS pairs (within an AFE) are tightly controlled on the layout, the data will then be properly aligned as well.
Many of these registers are tightly packed with control bits, which means that oftentimes one will need to set or clear a bit in these registers without disturbing any other bits in the register. To make this work, you'll need to read-modify-write and use AND and OR operations with bit masks to do what you want.
For example, suppose you need to SET bit 2 in a particular register. Do it like this:
x = peek(my_reg)
x = x | 0x00000004
poke(my_reg,x)
Now suppose you need to CLEAR bit 3 in a particular register. Do it like this:
x = peek(my_reg)
x = x & 0xFFFFFFF7
poke(my_reg,x)
This is how one would do it in an OLD SCHOOL language like C. There is probably some very fancy simpler and much less error prone way to do it in more modern languages!
-
Configure the timing endpoint and verify that is in a good state. Whenever the timing endpoint or timing master is reset, reconfigured, etc. this procedure will need to be repeated.
-
Initialize the IDELAYCONTROL module. This module is responsible for making sure that the IDELAY delay tap values are calibrated in very small steps (a few picoseconds) and that these delays are constant over chip voltage and temperature changes. First set the "idelayctrl_reset" bit, then clear it. Now read the "idelayctrl_ready" status bit; it should be 1. That means that the IDELAY delay tap values are now calibrated.
-
Configure all AFE chips using the SPI interface. There are several control bits that will need to be changed from the power on default values. From the AFE5808A datasheet:
- write 0xE000 to address 2, TEST_PATTERN_MODES=111(RAMP)
- write 0x2000 to address 3, SERIALIZED_DATA_RATE=10(16X)
- write 0x0008 to address 4, ADC_RESOLUTION_SELECT=14 bit, ADC_OUTPUT_FORMAT=offset binary, LSB_MSB_FIRST=LSB first
- write 0x0100 to address 10, SYNC_PATTERN=YES
4a. Clear the "idelay_en_vtc" bit. This temporarily disables the circuitry that enables the IDELAY delay tap values to track changes in chip temperature and voltage.
4b. At this time it's a good idea to reset the ISERDES as well. Set and then clear iserdes_reset control bit.
- Now sweep the IDELAY tap values and look for the bit edges. There are 512 IDELAY tap values in 7ps steps. The AFE is operating at 62.5MHz and transmitting serial data at 16x, so the data rate is 1Gbps per LVDS pair, or a bit period of 1.0ns. Therefore the width of each data bit is equal to about 142 IDELAY taps.
To do this delay sweep, run this loop:
for (i=0; i<512; i++)
{
write i to AFE0 Delay Tap Register
arm the input spy buffer (write the value 8 here, which selects AFE0 channel 8 "frame marker")
trigger the input spy buffer
x = the first sample of spybuffer
print i, x
}
Now you don't really care WHAT raw value is that was captured by the spy buffer. That's not the point. You're looking for the IDELAY tap value that causes it to CHANGE or JUMP. That's the important information. That's how we locate the bit "edges". And you don't need to read out the WHOLE spy buffer for AFE0 channel 8, just the first sample of that spy buffer. That's all we need to see. So the output of that software loop above might look something like this:
0x000 0xFF00
0x001 0xFF00
0x002 0xFF00
0x003 0xFF00
0x004 0xFF00
0x005 0xFE01 ah, starting to change!
0x006 0xFF00 some "dithering" going on as we're right on the edge
0x007 0xFE01 now we've changed! ok this is one edge of the bit!
0x008 0xFE01
0x009 0xFE01
0x00A 0xFE01
0x00B 0xFE01
...
0x092 0xFE01
0x093 0xFE01
0x094 0xFE01
0x095 0xFC03 another change, this is the other edge of the bit!
0x096 0xFC03
0x097 0xFC03
0x098 0xFC03
0x099 0xFC03
...
What we're looking for here is to find pair of bit edges. The first edge in this example happens around tap value 0x007 and the other edge happens around tap value 0x095. Note that these edges are separated by about 142 taps, just like we would expect. Good, now let's pick the ideal tap value by selecting something smack dab in the middle of the bit, say, tap value 78 or 0x04E. Write this value to the AFE Delay Tap register. OK now we're done with the fine adjustment for AFE0.
- After completing the prior step we know that we are sampling the high speed data serial right in the center of the data bit, at the most reliable point. But we're not done yet. Now we will need to do the word alignment of the serial data. A high speed shift register is used for this operation. The high speed serial bits are flying down this shift register and we must grab 16 bits at just the right time to form the 16 bit parallel word properly. If we grab the bits one bit too early or one bit too late we'll accidentally grab a bit that belongs to a previous or next AFE sample.
This is where the BITSLIP operation comes in. We know that the frame maker always sends the same pattern "1111111100000000". So that's what we're looking for, using this simple loop:
for(b=0;b<15;b++)
{
write b to AFE0 bitslip control register
arm the input spy buffer (write the value 8 here, which selects AFE0 channel 8 "frame marker")
trigger the input spy buffer
x = the first sample of spybuffer
print b, x
}
And the output of this loop might look something like this:
0 0x1FE0
1 0x3FC0
2 0x7F80
3 0xFF00 this is it!!!
4 0xFE01
5 0xFC03
6 0xF807
7 0xF00F
8 0xE01F
9 0xC03F
A 0x807F
B 0x00FF
C 0x01FE
D 0x03FC
E 0x07F8
F 0x0FF0
3 is the correct BITSLIP value, write 3 to the AFE0 bitslip control register.
So for this AFE0 example the ideal tap value is 0x04E and the bitslip value is 3. These settings will be automatically applied to all data channels for AFE0.
- Repeat steps 5 and 6 for the remaining AFE four chips.
- Set the "idelay_en_vtc" control bit. This will ensure that the IDELAY calibrated tap values stay in calibration over voltage and temperature changes.
- Now the AFEs are still in "count up" test pattern mode. Let's check it all. Trigger the spy buffers, and dump them all out. AFE channels 0 to 7 should show values counting up by one, and AFE channel 8 should always be 0xFF00 for every sample. Remember, when you read a spy buffer you're reading 32 bits and there are two samples contained in each 32 bit word. The lower 16 bits is the older sample and the upper 16 bits is the newer sample.
-
If everything looks good, let's put the AFEs into normal data taking mode by writing to the SPI interface. From the AFE5808A datasheet:
- write 0x0000 to address 2, TEST_PATTERN_MODES=000(normal data)
- write 0x2000 to address 3, SERIALIZED_DATA_RATE=10(16X)
- write 0x0008 to address 4, ADC_RESOLUTION_SELECT=14 bit, ADC_OUTPUT_FORMAT=offset binary, LSB_MSB_FIRST=LSB first
- write 0x0100 to address 10, SYNC_PATTERN=YES
-
At any time you can arm/trigger the input and output spy buffers and readout without impacting any physics operations on DAPHNE. Looking at AFE channel 8 frame marker is a good periodic check to verify that the front end alignment is working properly.
If there is down time and DAPHNE is not taking physics data, it should be possible to put the AFEs into one of the test modes (writing to AFE SPI address 2) and triggering the spy buffers to verify all data channels are properly aligned. Switching the AFE chips into and out of test pattern modes should not impact any timing so it is not likely this alignment procedure would need to be repeated each time the test pattern modes change in the AFE chips.