Employ Smart Redundancy for Reliability

Nuclear submarines are high-reliability designs that have redundancy built into them. Many of these systems are split into two halves (port and starboard), and each system has two pumps (a total of four). Each side could run fully on one pump. Then the two halves are cross-connected, so the entire system could run at partial capacity on a single pump.

All of this redundancy comes at a cost — about $1 billion in this particular case. Naturally, one of the comments I hear when describing these systems is that we could never afford such redundancy in our profit-making companies. I challenge that there is a place for what I call “smart redundancy.”

Smart redundancy has to do with taking a critical look at your systems, looking at risks and searching for opportunities where a little redundancy may pay big dividends. Utility and packaging systems are ripe for this kind of review.

One utility case in point is a fresh-water supply I inherited. After drawing the water from a well and treating it, a single pump provided the pressure to supply the entire factory. Failure of the single pump or its motor, electrical controls or pressure switch would shut down the entire factory within about three minutes. Depending on the failure, we could be down from 10 minutes to a couple of hours.

While we performed preventive maintenance on the components, the harsh environment of this pump still resulted in occasional failures.

After some study, we decided to put in a second pump, with a pressure set point a little lower than the first. The pump cost us around $2,000; with installation and controls, it was a bit more. But the point is we never, ever shut down the entire plant again for lack of water, and we offset the entire cost by avoiding only a few minutes of downtime.

Packaging lines present their own problems. We often put 10 or more components in series (including the conveyors), and the failure of any one will shut down the entire line. Even if each component has a 97-percent uptime, the line will only perform at 74 percent (.97 x .97 x etc. for 10 times).

Have you experienced a time when an upstream component on one line was down at the same time as a downstream component on another line? Suddenly two lines are not available.

If you could cross-connect the two lines somewhere in the middle, you could avoid some of these cases. That is sometimes as easy to do as designing a movable rail where two conveyors run together.

Don’t let the newspaper headline in your town read, “$2,000 pump shuts down entire factory.”