Converting VeEX PTP TE Recordings to a Basic Format Compatible with CAT Analyzer

PC applications are used to perform advanced PTP Time Error analysis and mask validation. This guide explains how convert VeEX's simplified CSV files, so they can be opened, analyzed and compared by third-party analyzer apps, like Calnex's CAT.

Related Products:  TX340s, RXT-3400, RXT-6200 and MTTplus-340

Related PC Software: VeEX Wander Analysis, Calnex Analysis Tool (CAT)

VeEX test sets, running IEEE1588v2/PTP Timing Measurements, in slave mode, can log packet timing or Time Error (TE) measurement information in real time, and then save it into a USB memory stick for further analysis.

To record TE data from a running PTP measurement, insert a USB memory stick, let the test set recognize it, press the Rec->USB button on the 1588v2/PTP application, select the type of parameter to record, enter the desired File Name and press OK. The button then turns red and will display Recording, until pressed again to stop it.

The resulting file is an open CSV format, which users can easily modify manually, using macros or scripts (recommended), to convert them to other simple CSV formats supported by third-party post-analysis applications.

PTP parameter recordings supported by VeEX products:

• 2Way TE
• TE1 (Forward TE)
• TE4 (Reverse TE)
• Sync PDV
• Follow Up PDV
• Delay Request PDV

For more details on how to configure and use VeEX's PTP emulation and packet-based Time Error measurement features, refer to the IEEE 1588v2/PTP Testing - Quick Reference Guide.

People who have been using CAT in their labs or offices, and have historical or reference traces already on file, could use a conversion procedure to take recordings from VeEX products, convert them and be able to compare, perform further analysis and/or generate reports compatible with what they already have on file.

1. Understanding CSV File Formats

1.1 VeEX's PTP Time Error (TE) CSV Format

VeEX's simplified packet-based timing and Time Error recording file format consists of an informational header for traceability, the TE data log in X,Y format (X=Time, Y=TE), and a footer with the end-of-file marker and extra information (summary) about the recording. 

VeEX Inc.,VePAL TX340S
S/N,TT7B00RO810181B
SW Version,tx300s-Release-4.0.10
Test Type,PTP Timing
Reference Clock,ATOMIC 1PPS
Test Signal,2Way TE
Start Time, 2022/08/03 16:11:37

Time(s), TIE(ns)
0.000000, 68.451
0.062473, 68.444
  ¦         ¦
64471.715580, 75.697
64471.778020, 75.704
64471.840494, 75.711
End TIE Data,
End Time, 2022/08/04 10:06:08
Primary-ET, 64471 s
Primary-Total Sampling, 1031553
Primary-Sampling Interval,16/s

1.1.1 Header

• Vendor (VeEX inc.) and product type (e.g., VePAL TX340S).
• S/N is the serial number of the instrument (test set) used to take the measurements, for traceability purposes.
• SW Version loaded in the test set, for traceability purposes.
• Test Type is the particular test application used to record the data.
• Reference Clock used to make the timing measurements (e.g., GPS 1PPS, Atomic 1PPS, External 1PPS).
• Test Signal is the type of measurement being recorded: 2Way TE, TE1 (Forward TE), TE4 (Reverse TE), Sync PDV, Flwup PDV (Follow Up PDV), or DelReq PDV (Delay Request PDV).
• Start Time indicates the beginning of the test, in the universal YYYY/MM/DD hh:mm:ss format.

1.1.2 Data

• Time(s), TIE(ns) mark the beginning of the data set.
• Individual timestamps (in seconds) and measurement data samples (in nanoseconds).
  • The Time is relative to the Start Time, so it always starts at zero.
  • Although named TIE, this column may contain TIE (Wander), TE, or Phase Error data.

1.2.3 Footer

• End TIE Data marks the end of the data recording.
• End Time is the timestamp for when the test was stopped, in YYYY/MM/DD hh:mm:ss format.
• Primary-ET indicates the total measurement duration, in seconds (ET = Elapsed Time).
• Primary-Total Sampling indicates the total number of data samples collected.
• Primary-Sampling Interval indicates the data sampling rate (e.g., 1s, 1/8s, 1/16s, 1/32s) 

1.2 Simple VER:1 Compatible CSV Format

Here is an example of the simplest semi-colon separated value structure that post-analysis applications, like Calnex's CAT software, can import. (No one can guarantee that third-party vendors would continue to support older or simple dataset formats in their newer releases. This particular format is provided just as an instructional example, which could be adapted to different text-based formats from other vendors.)

TIE Data File Format

VER:;1;
DataType:;TIEDATA; Format:;CSV;
MeasType:;1PPS Absolute;
Port:;C;
START:;31/12/2022 23:59:59;
PERIOD:;1;
value;
0
0
-1
0
1
0
¦

TE and PDV Data File Format

VER:;1;
DataType:;TIMEERRORDATA; Format:;CSV;
MeasType:;Sync;
START:;31/12/2022 23:59:59;
timestamp;value;
121116195;-7;
237117443;-5;
345120283;-2;
453124883;0;
485122691;-1;
    ¦      ¦ 

To the best of our knowledge, these are the different elements:

1.2.1 Header

• VER: Is the format version. It should be left as 1.
• DataType: Describes the type of data recorded below the value label. Also referred as "Instrument type". (Although the CAT version used to test the instructions in this article was able to export to all the data types below, it was only able to open the files it exported with TIEDATA. So, the TE and PDV formats could not be verified. They may have addressed this issue in newer releases).
  • TIMEERRORDATA (TE) - Enables Time Error measurements: TE, TE (Filtered), cTE, dTE analysis. This format requires a timestamp field (column) in front of the value, as in timestamp;value;. The timestamps and values in nanoseconds.
  • PDVDATA (PDV) - Enables packet-oriented PDV, TIE, MTIE, TDEV, MATIE, MAFE, FFO, FPC, FPR, FPP). This format also requires the timestamp field (column).
  • TIEDATA (TIE and Packet TIE) - Provides clock-oriented TIE, MTIE, TDEV, MAFE, FFO analysis (Metrics). This format does not require the Timestamp field (column), just value;
• Format: Describes the file data structure. It should be left as CSV.
• MeasType: Defines the type of parameter measured in the recording.
• Port: Identifies the port used (on a Calnex unit) to make the measurements. It can be left as C (or use VX as a reminder that it was a conversion from a VeEX source). For Time Error data, the Port: line can be omitted.
• START: Indicates the beginning of the test, in DD/MM/YYYY hh:mm:ss format.
• PERIOD: Is the sampling rate, which is 1/Packet_Rate (e.g., 0.0625 for 16 packet/s).
• value indicates the start of the list of measured values.

1.2.2 Data

• Individual consecutive measurement data samples (in nanoseconds). For TIEDATA type, there is no explicit timestamp information, but it is assumed as starting from zero and incrementing by a sampling Period for every line. 

1.2.3 Footer

• There is no explicit End of File. The file ends when the values stop. There is no explicit measurement duration either (if required, it could need to be calculated by counting the number of samples collected and multiplying it by the Period). 

2. Converting VeEX Format to VER:1

For simplicity, the TIEDATA format will be used in this example. Note that the VeEX TIE(ns) column, without the Time(s) column, is exactly the same as the single value column in the VER:1 file. For illustration purposes (if the VeEX CSV file is small enough to be opened with a spreadsheet program), one could follow these simple instructions to manually convert the TE files, in five minutes or less:

1. Open de VeEX .csv file using Excel or any other spreadsheet program.
2. Save the file with a new .cat.csv extension or any customized name. (It is recommended to keep the same file names and just change the extension, so the files can be correlated in the future.)
3. Delete the VeEX header.
4. Scroll to the bottom of the file and delete the footer summary information rows (End File Data and below).
5. For TIEDATA type, select the whole Column A (timestamps) and delete it.
   1. If using TIMEERRORDATA or PDVDATA types: Keep column A, write 1000000000 on an empty cell, Copy that cell, select column A, go to >Home >Paste >Paste Special, select Multiply, and OK, to convert it to ns. Delete the 1000000000. Later, open the resulting CSV file with a text editor, and use Find-and-Replace to change the commas to semi-colons.
6. Copy the VER:1 header and insert it at the top of the file. Then, delete any extra rows left between the new header and the data samples.
7. Edit the new VER:1 header to reflect the original data from the VeEX file (MeasType, START and PERIOD).
   1. We don't have the official list of labels defined for the MeasType parameter. Nonetheless, the CAT program seems to take most of the typical labels and identifies them as Packet data, but it may not perform the right timing analysis (which is ok if only performing MTIE and TDEV analysis). At the moment, we recommend using the 1pps TE 2WayTE Absolute label, because it also allows cTE and dTE post-analysis. (This article will be updated once more information about the packet-oriented Time Error and PDV data formats is found).
   2. Manually overwrite the start date and time, keeping in mind that the date formats are different (convert YYYY/MM/DD from VeEX to DD/MM/YYYY).
   3. Period = 1/SamplingRate (e.g., 1/16s = 0.0625 or 1/64s = 0.015625). With regards to VeEX's packet-oriented timing measurements, the sampling rate is the same as the PTP packet rates (e.g., use Sync packet/s rate for 2WayTE samples).
8. Save the modified file.
9. If necessary, open the resulting CSV file with a text editor (e.g., NotePad++), use Find-and-Replace to change the commas to semi-colons, make any other adjustments, and Save.
10. Test the new file with the target analyzer application (e.g., CAT) and correct any typos or extra spaces, using a text editor.

The following example shows a side-by-side comparison between the original 2WayTE.csv from a VeEX product and the converted 2WayTE.cat.csv files.

The extra "2WayTE" label was appended to the MeasType "1pps TE Absolute" for future reference, and to help identify the data type when the file is loaded into the post-analysis application. This extra label doesn't seem to affect the data import process.

The PERIOD value is calculated from the Sampling Rate, available in VeEX's footer. However, if the file gets truncated or it is too large to search, the sampling rate can be calculated by looking at the top of the Time(s) column, identify when the 0.x turn into 1.x, to identify the first second. In this example, those are rows 10 to 25, a total of 16 rows, or 16 samples/s, which is equivalent to PERIOD = 0.0625s.

Cell colors added for illustration (correlation) purposes only.

MeasType Values as Exported by CAT

Signal Under Test (VeEX)	DataType, MeasType (CAT)
1PPS TE (Absolute)	TIEDATA, 1pps TE Absolute
1PPS TE (Relative)	TIEDATA, 1pps TE Relative
1588v2 TE (TE, cTE, dTE, MTIE, TDEV, etc.)	TIEDATA, 1pps TE 2WayTE Absolute
1588v2 TE (TE, cTE, dTE, MTIE, TDEV, etc.)	TIMEERRORDATA, 2Way TE
Forward TE (TE1, cTE, dTE, MTIE, TDEV, etc.)	TIMEERRORDATA, Sync
Reverse TE (TE4, cTE, dTE, MTIE, TDEV, etc.)	TIMEERRORDATA, Delay Req
Sync PDV (TIE, MTIE, TDEV)	PDVDATA, Sync
Follow Up PDV (TIE, MTIE, TDEV)	PDVDATA, Follow Up
Delay Request PDV (TIE, MTIE, TDEV)	PDVDATA, Delay Req
Path Delay PDV (TIE, MTIE, TDEV)	PDVDATA, Path Delay
SyncE (Wander)	TIEDATA, SyncE
1.544 MHz (Wander)	TIEDATA, 1.5M TIE
2.048 MHz (Wander)	TIEDATA, 2M TIE
10 MHz (Wander)	TIEDATA, 10M TIE
1.544 Mbit/s (Wander)	TIEDATA, T1 TIE
2.048 Mbit/s (Wander)	TIEDATA, E1 TIE

3. Final Results & Comparison

Let's start with the TX340s 2Way TE measurement recording example from the previous section, which was performed at 16 samples/second, for about one hour.

Below is the TE data graph from the original 2WayTE.csv file, as seen with VeEX's Wander Analysis PC software:

The following is the information shown once the converted 2WayTE.cat.csv TE file is loaded into the CAT analyzer:

Here is a screenshot of TE graph from Calnex's CAT (0.1Hz low-pass filter applied), with cTE and dTE (Dynamic Time Error) tabs enabled:

Dynamic Time Error (dTE) post-analysis example:

Side-by-side Comparisons

The 2Way TE files used in this article, as examples, can be downloaded here, to be used as references for experimentation.

4. Can PTP Timing VER:1 Files be Converted to VeEX CSV?

Technically yes, by using a similar process in reverse. However, there is a lot of non-explicit information "missing" from the VER:1 files, that needs to be recalculated. Such as:

• Timestamps - A new Column A needs to be added and filled by recalculated timestamps, starting from zero and cumulatively adding the sampling period to each line (assuming that the sampling can be considered constant, with no PDV).
• Total Sampling - Needs to be calculated by counting or calculating the total number of samples in the file.
• Elapsed Time (ET) - It is basically the last timestamp, or the total number of samples multiplied by the sampling PERIOD.
• End Time - Add the Elapsed Time to the Start Time and convert back to YYYY/MM/DD hh:mm:ss format.
• Sampling Interval = 1/PERIOD
• Test Type - Use "PTP Timing"
• Test Signal - Use "2Way TE", "TE1" for Forward TE, "TE4" for Reverse TE, "Sync PDV", "Flwup PDV" for Follow Up PDV, or "DelReq PDV", to match the original data source.

The rest of the values are for information and traceability purposes, but they are not available in the VER:1 file. Therefore, some experimentation may be required, if vendor, instrument model or S/N need to be included. Otherwise, copy and use VeEX's original values from a known CSV file. 

 

VeEX Wander Analysis PC version used: 2.2.0.19104

Calnex CAT version used: 23.11.19114.1547