F-22A Raptor Date Lined

Lockheed’s F-22 Raptor Gets Zapped by International Date Line

Via a reader:

When the group of Raptors crossed over the IDL, multiple computer systems crashed on the planes. Everything from fuel subsystems, to navigation and partial communications were completely taken offline. Numerous attempts were made to “reboot” the systems to no avail.

Luckily for the Raptors, there were no weather issues that day so visibility was not a problem. Also, the Raptors had their refueling tankers as guide dogs to “carry” them back to safety. “They needed help. Had they gotten separated from their tankers or had the weather been bad, they had no attitude reference. They had no communications or navigation,” said Retired Air Force Major General Don Shepperd. “They would have turned around and probably could have found the Hawaiian Islands. But if the weather had been bad on approach, there could have been real trouble.”

“The tankers brought them back to Hawaii. This could have been real serious. It certainly could have been real serious if the weather had been bad,” Shepperd continued. “It turned out OK. It was fixed in 48 hours. It was a computer glitch in the millions of lines of code, somebody made an error in a couple lines of the code and everything goes.”

No attitude reference. No communications. No navigation.

TILT.

Comments

  1. Well, if the US DoD/Lockheed Martin ever need to get a programmer who isn’t a total idiot to write the software for their jets.. just drop me a line. Seriously, any programmer worth his salt knows that you perform all timing functions based on a FIXED TIME such as UTC, and all time diplays are based off this fixed time, plus your current time zone/daylight saving offset. That way if your local time changes, all it affects are your displays, since all internal calculations are based on a fixed time reference. It’s obvious that if you base calculations off local time, that local time can change either forwards or backwards, and stuff up your calculations. For example, I once changed my time zone from GMT to local time while I was downloading a file, and the file download progress went haywire, because it calculated the time remaining by looking at the current time and the time the download started and subtracting them. This method is fine except when you’re using local times. Here I am writing database apps, not multi-billion dollar defence code, and I know how to do it properly. If they paid me a few million dollars I’m sure I could write some software for them that doesn’t totally suck. Come on; what are these guys being paid for? Seriously? There should have been at least one seasoned person on the project who knew about potential problems like these and worked to avoid them. Simply inexcusable.

  2. Nicholas: I agree that this is ‘inexcusable’, however as someone who is also in the field of software engineering (testing, in my case), I can say that there’s no way to tell that it’s as simple a picture as you paint. It is completely possible for them to base their times off of a fixed time, and for a sudden change in locale to STILL cause problems like this. I wouldn’t assume it’s so straightforward. I could come up with an essentially limitless list of ways they could have implemented the system properly, only to put bad logic on top of that solid system and bring everything down. That being said, as someone in software testing, I ‘pretty sure’ someone didn’t cover something they should have in their release tests…

  3. Not the first time something like this has happened. During the 80’s when the F-16 came on line, everything was coded in Ada, and the ‘managment buzzword of the day’ was ‘reusable code’…. Code from the navigation system was ‘reused’ in the flight control system. As a result, when you flew an F-16 across the equator, the aircraft rolled inverted (latitude goes negative south of the equator)…. Sounds like a similar problem….

  4. FrontPorchPhilosopher: Is that true, or the military version of an urban legend? Sounds like it might have just been a simulation? Here’s what I saw on: http://www.csl.sri.com/users/neumann/illustrative.html#8 *f F-16 simulation: virtual plane flipped over whenever it crossed equator (S 5 2) More on the upside-down F-16 bug that was caught in simulation: the bug apparently led to a deadlock over whether to do a left or right roll to return to upright, and the software froze (S 9 5). ADDED NOTE: This is seemingly an example for Les Lamport’s Buridan’s Ass paper, where the ass halfway between two points could not decide which way to go. This case is one that still needs definitive resolution after all these years. Either (1) this was a flaw in the avionics software that was detected by the simulation, or (2) perhaps more likely it an error in the simulation program itself rather than the avionics software. Does anyone still alive know for sure?

  5. KTLA, yeah, I did think that there might be other reasons for this problem, but couldn’t think of any that would *reasonably* to cause a crash. Put it this way, let’s assume that if they hadn’t crossed the date line, but flown around the world the other way this problem wouldn’t have happened. That is, it wasn’t a bug due to being in the GMT+11 time zone, but due to crossing from the GMT-12 to GMT+11 (I could have those slightly off, but I think you know what I mean). This assumption could be wrong, but I would be surprised, since it would be an even worse bug if it simply didn’t work in the eastern hemisphere at all. So assuming that, this crash was caused by the local time going backwards, essentially. Now, why would *any* part of the software care about the progression of *local* time? Local time is essentially used only for display, right? Or perhaps if your software says ‘land at airport x at local time y’.. but local time y has a fixed mapping to GMT/UTC time z.. so you should convert it to GMT/UTC at the first available opportunity, and convert it back to local time for display, so that you can perform date calculations off it without having the opportunity for this sort of bug. So sure, I can think of other reasons than the one I described which could lead to a bug when you cross the date line, but I can’t think of any that would actually crash the system, unless the programmers did something *even worse*. Weird display values should NOT crash it. I mean really, I don’t know what language they used, but they should have redundancy checks in many places, especially checking for ‘invalid’ values going into functions and ignore them/return/assume a default. Basically this indicates a failure at multiple levels. Not only did they make some kind of a date/time bug but they also failed to check values in functions which it flowed through, allowing a bad internal value to crash the system. Now, I’ll admit, my programs have occasionally crashed. But I wasn’t writing software for the F-22, so I wasn’t being anywhere near as careful as I could have. Plus, I’ve learned from many of those mistakes and know how to avoid them in future. Something I would be trying very hard to do writing military Code. I would use a variety of practices I’ve picked up which greatly reduce the possibility of certain kinds of bugs. And, ideally, your bug protection overlaps in such a way that an error is possible, but it won’t bring down the system. Anyway.. yeah.. you’re right I don’t know exactly what happened, but it sounds suspiciously like it would be a problem similar to what I described.

  6. It wasn’t a problem with keeping time. The problem was that the longitude switches from W to E, and somewhere in the interplay between the GINS (GPS Inertial Navigation System) and the other avionics something didn’t like the switch. Possibly they would have had the same problem crossing the prime meridian going the other way. Also, it should be noted that they did test the code in a simulator, and it worked fine. But the simulator did not use the actual GINS hardware (because the simulator does not acutally move, so a GPS receiver would be useless to simulate movement). The GINS hardware was tested separately and also did not show an anomoly. Clearly this was some edge case that only appears when the two work together. My wild-ass guess about the issue is that the problem is how the actual 180 degree line is referred to. All of these are equivalent: 180 degrees west, 180 degrees east, 180 degrees. If the GINS reports 180 degrees, and the avionics is expecting an E or W, maybe it blows up (or something like that). That would be hard to catch in testing because the simulator guys would know to format 180 degrees the way the avionics needs to see it. The GINS guys would see any one of those three and think ‘OK, it works’, but no one necessarily checked to see if the GINS output and the avionics input, for that one case, work together. Again, that’s just a guess.

  7. Also, I’d be stunned if the F-16 story is true, mainly because I’d be surprised if the routines that control roll care about the planes lattitude/longitude at all. I would assume separate routines control navigation and merely exchange heading information with the avionics. ‘Which way are we going?’ ‘Now go that way.’ The equator story sounds like many other ‘upside down on the other side of the equator’ stories that run around.

  8. Ah good point.. I would have thought the safest thing would be for the navigation system to be tested with all of those possibilities, and be designed explicitly to handle any of the 180 degree east/west latitudes identically.. oh well. For some reason because it’s called the ‘date line’ I had it in my head that it was a time/date problem, but I forgot that it’s also the point where the co-ordinate system wraps. Still.. this is the kind of area where lots of attention is paid during development, I would have thought.

  9. Basically, it sounds like they relied too much on unit testing, without testing how modules interact at the edge cases. Also, the 180 E/W is just my wild guess as to the issue. Since both modules seemed to work independently, that struck me as one way in which communication between the modules could screw things up. That may not be the reason though. Regardless, it must have been some similar kind of edge case, that I agree should have been thought about previously. I’ll bet someone’s scrambling around to think about other edge cases right now (prime meridian, equator, the poles, below sea level in Death Valley, and others I may not have thought of).