SK/K2K GPS Rollover Bug Repair

H. G. Berns and R. J. Wilkes

8/31/99

 


Short Attention-Span Summary:

Events recorded from August 22 18:00 JST until end of run 7812 (August 31 11:55 JST) had invalid primary GPS timestamps in the OD header, due to GPS Week Rollover. For these events, the secondary timestamp must be used for any precise timing application. The primary GPS receiver has been temporarily replaced, and will have its firmware updated to properly handle post-rollover times. The secondary GPS receiver showed no problems at all, as expected.

Introduction

GPS specifies dates in terms of the number of weeks from 06 January 1980. Unfortunately, when the GPS system was designed in the late 70s, nobody imagined it would become widely used by civilians around the world, with millions of mass-market receivers playing a crucial role in the US and world economy. In the memory-tight computing environment of that era, it made perfect sense to limit the length of the week number to 10 bits, such that dates roll over after 1023 weeks. Surely by 1999 the US DOD would replace GPS with a newer, better system! Rollover occurred at 23:59:47 UTC on 21 August 1999, or  approximately 09:00 JST on 22 August 1999 (see http://tycho.usno.navy.mil/gps_week.html ).

Response of SK/K2K GPS Receivers

At each site, we have two receivers: TrueTime XL-DC (primary) and Motorola UT (secondary - see http://www.phys.washington.edu/~berns/SUPERK/GPS/ and http://www.phys.washington.edu/~berns/K2KGPS/ for details). Timestamps from both primary and secondary receivers are inserted in the Super-K and K2K data stream, as explained in previous notes (see http://www.phys.washington.edu/~superk/software/sorter/wordstructures.html#gps and http://www.phys.washington.edu/~superk/hardware/skutc.html). At K2K, the TrueTime receiver is a more recent version of the same model used at SK. The TrueTime receivers cost over $4000 each, because they are intended as stand-alone time and frequency standards, and contain a good quality internal clock which maintains time accuracy in the absence of satellite data. The Motorola UT receivers, which cost less than $200 each, are simply GPS receiver modules with no internal clock (we provide that on the UW-built LTC board). Guess which clocks managed the rollover without the slightest problem?

TrueTime warranted their newer version’s firmware (at KEK) to be completely prepared for the GPS week rollover, but made no warranty for the older version (at SK). On the other hand, their technical staff did not expect that the older version would fail. We therefore chose to wait and see rather than pay for an upgrade PROM which might have been unnecessary. Motorola’s UT modules were all purchased within the past 2 years and the company claimed them to be fully rollover-ready.

Hans Berns checked the log files immediately after the rollover, and at intervals during the following day. The Motorola receivers at both sites carried on operation without any errors. The new TrueTime receiver (at KEK) lost its time synch immediately, but recovered about 22 minutes later, and has performed perfectly ever since. The older TrueTime receiver, at SK, carried on without apparent error for approximately 9 hr after rollover, then began counting time backward to midnight again, and has repeated this forward-and-back cycle continuously ever since (we call this the Groundhog Day Effect). In spite of this bizarre effect, examination of the log data indicates that satellites are being tracked, and seconds rollovers are being synchronized properly. Thus in fact the SK TrueTime data are useful for subsecond timing, as long as the date and time of day are taken from another source (eg, CPU time). However, the date and time will be incorrect starting from 9 hours after the rollover, or 18:00 JST on 22 August 1999, until the receiver was replaced as described below.

Fixing the Rollover Problem

We notified TrueTime of the problems encountered (along with many other customers, apparently) and within a few days received updated PROMs for the old receiver and VME/fiber interface at SuperK, which are supposed to fix the firmware. Since K2K is not running, we adopted the following simple plan:

  1. Send the new TrueTime receiver from KEK to SK where it can replace the old receiver.
  2. Send the old TrueTime receiver from SK to Seattle, where Hans can install the update PROM and check its performance carefully in case further repairs or a trip to the factory are needed.
  3. Upon completion of upgrades, return the old receiver to SK and send the new receiver back to KEK in time for the October run.

Step 1 was accomplished on 8/31/99, thanks to Clark McGrew and Kate Scholberg. Primary timestamps were accurate within 300 nsec almost immediately, and reached full accuracy within 24 hr, when the receiver has had time to average 90,000 satellite fixes and locks its measured antenna coordinates.

Effects on data

At Super-K, for reasons of backward compatibility, the TrueTime timestamp is placed in the OD header, while higher precision data for K2K from both the TrueTime and Motorola GPS clocks are available in a separate data block, the GPS header (see web pages cited above), which was implemented in the online data on 19 Feb 1999. Until recently, very little SK analysis software actually used the GPS time, since for most applications, the CPU timestamp contained in the ID event header is sufficiently accurate.

Events recorded from August 22, 1999, 18:00 JST until end of run 7812, on August 31, 11:55 JST, had invalid primary GPS timestamps in the OD header. Therefore the secondary timestamp, from the GPS Header (integers gpstime2_hi, gpstime2_lo, and ltcgps2), should be used for any precise timing application.