For a long time, all machines in Apollo-NG's infrastructure use chrony as a replacement for the usual ntpd package. chrony can be compared to ntpd like nginx can be compared to Apache, newer, much more lightweight approach and some additional very nice features. While nginx has replaced most of the Apache installations these days, chrony still isn't adopted as a good alternative by most people yet.

During the NTPD reflection attack time and the mysql/kernel/ntpd leap second bug, it was nice to see how chrony really saved a lot of time and grief by not being affected. It has been working here - and in many other scenarios - absolutely flawlessly until today.

While setting up gentoo on an Odroid C1 quad-core ARMv7 as a replacement embedded system for picoprint, chronyd died as soon as it tried to sync the kernel time, like this:

2015-02-07T16:29:09Z chronyd version 1.31 starting
2015-02-07T16:31:20Z Selected source 129.70.132.35
2015-02-07T16:31:20Z Fatal error : adjtimex failed for set_frequency, freq_ppm=-1.6047e+00 required_freq=1.6047e+00 required_tick=10000

Deploying 2.0pre1 didn't fix the issue, baking new kernels with different dyn_ticks settings didn't help either but finally there was a break: https://bugzilla.redhat.com/show_bug.cgi?id=1188074#c3.

--- a/kernel/time/ntp.c
+++ b/kernel/time/ntp.c
@@ -634,9 +634,9 @@ int ntp_validate_timex(struct timex *txc)
                return -EPERM;
 
        if (txc->modes & ADJ_FREQUENCY) {
-               if (LONG_MIN / PPM_SCALE > txc->freq)
+               if (-MAXFREQ_SCALED / PPM_SCALE > txc->freq)
                        return -EINVAL;
-               if (LONG_MAX / PPM_SCALE < txc->freq)
+               if (MAXFREQ_SCALED / PPM_SCALE < txc->freq)
                        return -EINVAL;
        }

After patching the odroid sources with the recommended one from the bugtracker chronyd is working like a charm again. So if you're having issues with newer kernels (at least from an embedded perspective), that use dynamic or idle tickless clocks you might want to patch your kernel as well.

The Odroid C1 also comes with another not so obvious pitfall: The RTC power supply. The optional RTC battery, you can usually buy wherever you can buy the Odroid itself, is the only power supply for the RTC. Meaning: When you don't have this battery, the RTC will not get powered at all (not even when your system is powered and running) so in order to get the RTC working, you need to feed power to the RTC Header.

A quick glance in the schematics revealed the secret: U15 (XC6215B0927R-G) is a highly precise, low noise, positive voltage LDO regulator, to feed the RTC the voltage it needs. Since using a battery seemed kinda backwards and there was a 5.5V 1F goldcap fyling around here, this hack (second from the left minus the resistor) was used to supply 3.3V from Pin 1 of the Odroid C1 40 pin header through a 0.4V drop schottky diode (to get to about 2.9V which emulates a CR2032 Battery closely enough) and then fed to the goldcap to charge it and supply the RTC power header of the Odroid C1 board while running. GND is the pin towards the UART where it says RTC on the silkscreen, connected to (-) of the goldcap and also to GND of the 40 pin header. The PIN closer to the IR Receiver is connected to the goldcaps (+).