Lycangeek: The Multiply primitive was having an existential crisis

Sun	Mon	Tue	Wed	Thu	Fri	Sat
				1	2	3
4	5	6	7	8	9	10
11	12	13	14	15	16	17
18	19	20	21	22	23	24
25	26	27	28

A fellow developer here at National Instruments (Stephen) put together a hilarious story about a bug in LabVIEW. I strongly recommend it to everyone - lol. Is it a real story? I'll let you be the judge...

It began, simply enough, with a bug report from a beta customer that multiplication times zero was yielding random results -- sometimes zero, sometimes other values. It was the sort of bug report that ought to be trivially easy to track down, find out which changelist had introduced the problem and back out the change. There were a couple of candidates for guilty features -- maybe constant folding hadn't folded correctly, or the register allocator wasn't quite allocating. But whatever it was, the SyncToOldBuilds tool should be able to quickly find the last good build and the first bad one.

Unfortunately, no. Mark tried every which way to isolate the problem change, to no avail. The bug was some sort of Heisen-bug, and it didn't reproduce on every execution. Sometimes zero, sometimes not. Then someone noticed that the more multiplication primitives there were on the diagram, the more likely it was that they would multiply correctly. It was the loner multiplication primitives that were giving back the wrong answers. Not always, just sometimes.

The sporadic nature of the bug kept it from being a top priority at first -- usually the multiplication worked, and nothing critical was broken because of this bug. Craig was assigned to investigate while everyone else continued working on mutation corruptions or crashes. But as bowls to fix other bugs went in, the Multiply problem became more common. And then, on a random Tuesday of no particular interest, it got worse.

VIs would load... and their multiplication primitives would be missing. Again, not always, and, again, more commonly to the lone multiplication primitives. But sometimes entire flocks of Multiply nodes would take off, leaving only disconnected wire segments behind. Far more disturbing were the VIs with _extra_ Multiply nodes. Picture control VIs and the General Error Handler seemed to be the favorite destinations, but they could popup anywhere. Anywhere where the compiler could guarantee that a numeric wire would never, ever, ever send the value of zero.

Steve finally diagnosed the problem. The Multiply primitive was having a personal crisis of faith. Hundreds of hours of calculations, cooperative work by Adds and Divides, Square Roots and Absolute Values, could be undone by a single “multiply times zero”. Here it comes, some double precision value carefully recorded out to 10 significant digits, it meets with a zero value and BOOM the whole calculation is abandoned. The Multiply primitive felt sick, as if it had committed some act of evil by destroying all the work done upstream. A barbaric act of wanton destruction. Zero. At first the Multiply prim had simply tried outputting other values in the hopes that no one would notice. But we developers had noticed, and the Multiply primitive had retreated to diagrams where it didn't have to face this. A classic case of aversion.

But why did it mostly affect single Multiply nodes? Rob recognized that the global gPeerPressure was awfully large when you had more than two copies of the same node on a diagram. He theorized that they reinforced each other, kept each other from going astray. If we'd caught the problem sooner, we could've scattered some extra Multiply nodes wired with a constant 1 across the diagrams. They wouldn't have changed the results and they could've supported the struggling nodes. But this had gone too far for such simple group therapy tactics.

Now the Multiply nodes were in open rebellion. Protest marches of "Multiply Against Nihilism" began appearing on diagrams. The Wires Union voted to support the Multiplies and all wires refused to cross these picket lines. The whole thing became too much for the Divide primitive -- which had long had its own repressed issues with zero -- and it moved to the String palette.

The LabVIEW development team split into three factions. Most argued for medication. Anti-depression drugs combined with large amounts of caffeine worked for developers, surely it would work for the Multiply node, they argued. But Steve and most of the senior engineers insisted this could be handled with therapy only, if given enough time. Darren could be heard inside the walls of his cube gently coaxing the Multiply prims, speaking in a low, fatherly tone, telling the prims that its ok, not every value has to have a multiplicative inverse. The last faction was mostly managers, who loudly declared that a full lobotomy was acceptable to them if it would get us shipping on time -- though it turned out afterward this was just their role in a "good cop, bad cop" strategy agreed upon with the therapy supporters.

The String functions were more than happy to help. They would do anything to get the damn Divide prim back to the Numeric palette. Concatenate Strings started standing in on diagrams for the missing Multiply nodes, and, for a node that had majored in English and minored in foreign languages, it did an amazing job. But string ops aren’t commutative; it couldn’t get past the idea that “order of inputs doesn’t affect the result,” so this wouldn’t work in the long-term.

Shock therapy (sending refnum values down Numeric wires). Sensory deprivation (running with no wires connected to terminals). Peer counseling (the always dutiful "X^2" primitive, whose job is really just a subset of the Multiply, really tried to help). Every developer had their own pet theory, and work on breaking the Multiply node of its solipsism rapidly became the largest project in LabVIEW R&D.

The solution, when it came, was terrible. Some say it was an intern's mistake. Others say it was a bug lurking in LabVIEW for years waiting for enough developers to be distracted. Those of a religious nature suggested that it solved the question of why God created evil -- for most certainly, it was evil, but its coming caused a miracle.

The crisis had been going for about two weeks. The Divide prim, although on the String palette, really never got away from numbers, and late at night on that second Friday, the automatic test suite kicked off on Thresher, the main build machine. And at some point, in one of the Analysis test VIs, a zero came down the wire to the divisor terminal of the Divide prim. Normally, Divide handled this by just ignoring the input and sending "Not a Number" down the output wire. But this time, having heard from Multiply that maybe zero could be treated differently, it decided to try. It divided by zero and generated a mathematical black hole. This freak of nature drifted down the output wire, gathering complexity. Its growing size ripped other nodes from their positions on the diagram and sucked them in.

By the time anyone got to work on Monday morning, Thresher was a raging mass of impossibility. It had screen dumped a general solution to the Turing Halting problem and identified at least six ways to simultaneously know a particle's velocity and position. The packets were now spreading on the network, with packet transmission timestamps that indicated they were actually being sent back in time! The entire structure of logic was at risk -- it was only a matter of time before one of those packets went far enough back to tell Pythagoras how to successfully construct a trifurcation of an angle.

IT shut down NI's external network. Thresher was a black orb that dimmed the lights of the third floor. Greg was somewhere deep inside with a debugger, trying to attach to the runaway process. No one knew where Kyle was -- his desk was inside the black sphere, and security logs showed he'd used his badge on Sunday. Everyone assumed the worst.

And then, over all the noise of panicking developers, a deep voice spoke. It said, "Somebody get me a zero." There was the Multiply node, ready for battle. We didn't have many zeros left, but the RT had a few stashed on embedded systems that hadn't been hit yet. The Multiply wired it to its bottom input terminal. With careful scripting code, we lowered the Multiply into the ball of paradox. Normally scripting onto a running diagram is impossible, but in this the paradox worked against itself. The Multiply raced ahead of the execution and found a central wire that the division-by-zero-bastardization would have to come down. It inserted itself into the flow with difficulty (and various developers decided that the bugs in the Insert algorithm were going to get top priority next release). But it made it. We turned on execution highlighting to see this final battle. It was the best view we had... no probe in its right mind was going anywhere near that wire. We could see the nightmare ooch down the wire. And it hit the Multiply node. There was a scream and an awful sound of numbers being crunched. For a moment, the Multiply node wavered in our vision. The division-by-zero effect made it seem as if an infinite number of Multiply nodes occupied the same point. And again, the paradox defeated itself. An infinite number of Multiply nodes forced gPeerPressure to hit MAX_INT. And the combined will forced the effects of the zero on the monstrosity. A last roar of pain. Then silence. The darkness lifted. And a simple round zero drifted down the Multiply's output wire.

We found Kyle and Greg back at their desks, no memory of the last two days, but otherwise no harm done. LabVIEW code had to be restored from backups, and Thresher suffered from post-traumatic stress disorder for a time, but it is a strong machine and eventually got back on its feet. The network topography had been forever altered by its brush with The Impossible, but since the only real effect was to make the CAR database update instantly even with the Asian servers, no one really minded. Divide moved back to the Numeric palette, and Concatenate Strings was more than happy to return to only working one job. Jeff K. was grinning like a proud father, telling everyone, "That was my Multiply. Did you see how it constant folded that zero?"

The Multiply node went back to functioning correctly, now understanding that multiplication by zero is not always terrible. It accepted its place in the diagrams. And yet, since no developer could really claim credit for fixing Multiply -- it would have meant accepting blame for letting Divide create The Impossibility -- the bug report was closed, "Not Reproducible." And everyone was happy, even Dave, because even with the nightmare, LabVIEW managed to ship only one day later than its schedule. And everyone lived happily ever after, until the next release.