from "The Persistence Of Memory" by Salvador Dali

KEK GPS Clock System

HGB, last edited 22 Aug. 2003

One major question for the K2K Long Baseline Neutrino Oscillation Experiment (E386) was: How do we synchronize the beam events generated at KEK, Tsukuba, with the Super-Kamiokande detector in Kamioka, approx. 250 km west of Tsukuba?
The answer is: With GPS (Global Position System)!

The currently operational 27 GPS satellites each contain an atomic clock system which is operated and monitored by the US Naval Observatory (USNO) Master Clock. These provide world-wide accessable accurate time info within a few nanoseconds. The realistically achieved accuracy of high-quality non-military GPS receivers (which are not configured to decode the "Selective Availability" scrambling available for military purposes only) is approx. 100 nsec (350 nsec worst case with poor satellite coverage).
For more general info about GPS, see the excellent Global Position System Overview website at the U.Texas.

The K2K GPS system at KEK is located in the North Counter Hall as part of the DAQ online system. It is almost identical in design and function as the GPS subsystem at SuperK in Kamioka. It consists of two GPS receivers of different manufacturers and design to insure independent time quality check:

  1. TrueTime XL-DC Time+Frequency Receiver as main unit, and
  2. Motorola UT Plus Oncore as backup receiver.
Each of the receivers is getting its GPS satellite data via separate antennas which are mounted on the North Counter Hall roof, just above the control room section (see photo below). The receivers are mounted in an electronics rack approx. in the control room of the North Counter Hall, 2nd floor, right next to the DAQ trigger electronics (see photo below).

Here is the block diagram of the system:

GPS Ant. XL-DC TAC32 FI-VME RS232VME TACGPS LTC5 Fifos LTC5 Bit3 nubl01 VMEcrate VMEcrate

KEK GPS system block diagram

The TrueTime receiver is a self-contained unit in its own 19-inch rack-mount housing and can be set up and calibrated without the need of additional external hardware or software. In December 1998 we upgraded it with a so-called "Time Interval & Event Timing" option with which an external trigger signal can be directly measured for the exact absolute arrival time (roughly +/- 50 nsec accuracy with approx. 30 nsec resolution according to the manufacturer, TrueTime, Inc.). This is basically the main function we need for accurately time-stamping K2K Spill Trigger events.

The backup receiver (Motorola UT Plus Oncore) is piggy-back mounted on a UW-made VME module (TACGPS = Totally Accurate Clock GPS module) and is located in the GPS VME crate (see photo below. The VME module provides as an interface and buffer for the receiver's serial port (for date, position, status, setup, etc.) and 1PPS synchronization pulse. It is monitored via a serial 9600 baud connection to a Windows PC running TAC32 software (see screen example).

The 1PPS (One Pulse Per Second) synchronization pulses of the two receivers are each separately logged via the UW-made Local Time Clock module. This way, the rollover of a new second of GPS time is time-stamped with "Local Time" - a free-running 50 MHz counter - to register buffers on the VMEbus, providing 20 nsec resolution. The GPS time and status infos of each receiver is available on the VMEbus via different paths with a few milliseconds delay after the second rollover:

Then, the arrival times of the K2K beam "spill" triggers are recorded with "Local Time" in a FIFO module (CAEN V533 32-bit Fifo). The spill trigger is also latched with the "Event Time" option of the TrueTime receiver which then provides the absolute time of arrival in format day_of_year:hour:minute:sec.nanosec (UTC) on a separate RS232 serial port.

Last-not-least, a pre-trigger signal ("-1.1 msec", provided by the Beam Channel system prior to the beam) is latched with the ICRR-made VME-TRG module for providing a spill event number count for the DAQ system.
The GPS VME crate is controlled by a Linux Pentium PC ("nubl01") with a single C program gpstrg via a Bit3 VME-PCI adaptor. After initializing the VME modules and both GPS receivers it runs continuously in an infinite loop of scanning the VME modules for arrivals of new LTC times of the 1PPS and spill trigger pulses, event numbers from the VME-TRG module, and the serial port data and IRIG-B data from both GPS receivers. If a new spill trigger pulse is detected, the 1PPS and LTC informations are used to calculate the absolute spill time with 20-nanosecond resolution and an accuracy of approx. +/- 50 nsec of UTC time in average, computed separately for each of the two GPS clocks. The algorithm for computing the high-accuracy time is briefly described in our K2K GPS Clock proposal (24 April 1998).

The result is transferred to the DAQ system and SuperK detector via ethernet approx. 120 msec after each spill trigger (see example of the GPS_TRIGGER control screen). In between spill trigger pulses (generated at a rate of approx. 1/2.2 Hz) gpstrg is doing extensive data quality check to compensate for occasional glitches in the GPS data and LTC oscillator drifts. Histogram tables for automated monitoring plots are periodically updated as well.

Hardware documentation and links:
  1. Main GPS receiver: TrueTime XL-DC description (177k pdf)
    This unit was enhanced with the "Time Interval / Event Time (TIET)" option in December 1998, recently offered by TrueTime, Inc. for their XL-DC time+frequency products. This option provides UTC time of the arrival time of an external trigger pulse with +/- 30 nsec accuracy to absolute UTC time.
  2. Backup GPS engine: Motorola UT Plus Oncore
    This module is piggy-backed on the TACGPS board which provides power fom the VME crate and drivers/receivers for the 1PPS and RS232 serial port signals.
  3. KEKGPS VME crate: (modified in June 2003)
    • Cabling diagram (gif or pdf), see also photo:
    • slot 1: Bit-3 model 616 [SBS-Bit3]:
      VME-to-PCI bus interface
    • slot 2: VME-TRG module [ICRR-made]:
      trigger module (developed for SuperK) with 16-bit event number counter + 50 MHz 48-bit clock counter
    • slot 4: V533 Fifo module [C.A.E.N.]:
      Pipeline fifo buffer 4096x32-bit, up to 66 MHz
    • slots 6 + 7: Local Time Clock Ver.5 modules [UW-made]:
      50 MHz 32-bit clock counter + 2-channel trigger register
    • slot 8: TACGPS module [UW-made]:
      Interface for Motorola UT Plus Oncore receiver + 50 MHz VCO circuit (test); now only used for NIM fanout of the 50 MHz clock signal distribution
    • slot 9: RS232VME module [UW-made]:
      2-channel RS232 fifo buffer VMEbus interface
    • slot 10: NEW!   Local Time Clock Ver.6 module [UW-made]:
      1:5 PLL frequency multiplier for 10 MHz input + 50 MHz output, 32-bit clock counter for 50 MHz clock, 5 TRG inputs, add-on module with 16+7-channel digital differential inputs, 4096x64-bit event buffer (fifo), new interface for Motorola UT Plus Oncore GPS receiver with RS232 serial data UART + buffer
    • slot 12: FI-VME module: [UW-made]:
      optical fiber receiver + IRIG-B decoder
    • slot 13-21 bay: NEW!   SRS model PR10: 10 MHz Rubidium Frequency Standard.
      Reference clock for Local Time Clock Ver.6 module, GPS synchronized via 1PPS from Motorola Oncore GPS.
  4. IRIG-B code infos
  5. SuperK GPS system infos

Operation and software help:

  1. KEK GPS System User Manual
  2. Main online program on nubl01: gpstrg.
  3. KEK GPS system online monitor, live, updated half-hourly.
  4. Monitoring software for the Motorola GPS receiver: TAC32 Software description or copy of the TAC32 manual (1MB pdf).

    TAC32 screen shot
  5. vmelib for Linux, PCI-VMEbus software for PCs running Linux provided by our KEK collaborators (infos in English and Japanese)
  6. VME news at NIKHEF (The Netherlands): Linux device driver for Bit3 models 616/617
  7. SK/K2K GPS Rollover Bug Repair [H.G.Berns + R.J.Wilkes, 8/31/99].

Calibration and setup measurements:

  1. GPS Survey Measurements at KEK and SuperK (1998):
    • Results of 10-day position survey at KEK GPS location:
      TrueTime XL-DC:
      540000 of 3D-fix samples      		 Latitude: North 36o 09' 08.23''
      Longitude: East 140o 04' 19.38''
      Altitude: +78.5 meters (WGS84)
      Motorola UT+ Oncore
      (with TAC32 software):
      916530 of 3D-fix samples      		 Latitude: North 36o 09' 08.261''
      Longitude: East 140o 04' 19.207''
      Altitude: +87.40 m (WGS84)
      +48.24 m (MSL)
      Estimated errors: +/- 2 meters horizontally, +/- 6 meters vertically.
    • Distance between SK and KEK GPS antennae:
      249,900.2 meters = 833.58 µsec TOF
  2. Oscilloscope screen dumps, recorded March 1999.
  3. Older KEK GPS system notes.
  4. KEK GPS setup status logs, Sept.1998...
    (sorry, not quite up to date; didn't have time for more documentation yet)
  5. Examples of online monitoring histograms.

KEK / Super-K reports about GPS

  1. "Alignment and Control of Neutrino Beam towards SuperKamiokande" (700 kB pdf or 5.5 MB gzipped ps) by H.Noumi et al., Draft, 31 Jan. 2000.
  2. "GPS clock calibration using an atomic clock" (240 kB pdf or 270 kB postscript) by S.Yamada et al, Draft, Dec. 1999.
  3. "GPS Time Synchronization System for K2K" by H.G.Berns & R.J.Wilkes, presented at RT99 conference, Santa Fe, New Mexico, June 14-18, 1999.
  4. "Precision Positioning of SuperKamiokande with GPS for a Long-Baseline Neutrino Oscillation Experiment" (6.3 MB pdf) by H.Noumi et al, KEK Internal 96-17, March 1997.

Other GPS-related infos

  1. GPS related links on the Web
    (GPS equipment manufacturers, general infos, NIST, USNO, etc.)
  2. Y2K and GPS-1024-week rollover infos.
{Click on photo(s) for full size}

KEK North Hall
KEK North Counter Hall (12/98)

KEK GPS Antennas
GPS Antenna location on North Hall roof (11/98)

TrueTime XL-DC Receiver
TrueTime XL-DC front panel (3/99)

TrueTime XL-DC rear side
TrueTime XL-DC rear side - cables (3/99)


GPS and DAQ racks in North Hall control room (3/99)


GPS system PCs and Monitors behind DAQ racks (3/99)


GPS electronics rack setup (5/99)


GPS VME crate front view (5/99)