The lamps have a very low MTBF. Every time the computer is turned on, several of the lamps are burned out. They go through 50 lamps per week...this in spite of a very low lamp activation...normally, only 1 or 2 lamps are on. Occasionally, one of the lamp drivers is also found to be bad.

The engineer examines the circuit and the signals going to the lamps. The circuit is a simple 2-stage switch. The final transistor is rated at 500 mA and 30 V. The lamp draws 300 mA at 28 V. The transistor gets quite warm if the associated lamp is driven continuously. The voltage from the power supply is measured at 27.4 V and the voltage at the lamp is 26.9V (ie, germanium transistor...WHAT, you don't remember germanium?).

The engineer knows that the inrush current to a cold lamp may be quite large. He measures it with a current probe and finds that indeed it is over 1 A! His fix is to add a resistor to each lamp driver to keep the filaments warm (but not visible), to minimize the load on the drivers. This fix has no effect on the MTBF of the lamps, but now there is a new problem: the plastic sheet over the status display area melts! The 100 warm lamps emit a lot of IR energy, which is absorbed by the tinted plastic...

The engineer solves this little hiccup by substituting glass...the glass cracks! He uses tempered glass. Finally success!

But the original problem persists. Whatever is causing the lamps to burn out is still present. The engineer notices that when the "Lamp Test" button is pressed (simultaneously turning on all lamps), several bright "flashes" signal where lamps have just burned out. He again checks the signals to the lamps and discovers that when the "Lamp Test" button is pressed, the voltage to the lamps rises to about 35 V before dropping to around 27 V. Monitoring the power supply, he sees that the ferroresonant supply, presented with a 50 A load change (there were other lamps that came on with "Lamp Test" in addition to the status lamps), over-controlled by raising it's output from 28 V to 37 V! This voltage "spike" and the recovery from the supply are slow, due to the nature of ferroresonant regulators...presenting excessive voltage to both the lamps and the drivers for almost a second (the driver transistor was rated at 30 V).

Changing to an electronically regulated, series pass power supply (this was before switching power supplies were commonly available) solved the problem, but at a cost:
 

• The efficiency of the new supply is about 50%, vs the 85% of the original supply...a fan had to be added to remove the 1.5 KW of heat given off the new supply
• The new power supply is much larger, requiring a major structural change to the enclosure
• The new supply costs three times as much as the original supply

The development team for the new Instrument was under severe time pressures. They wisely did not reinvent everything. After all, the project was a secret from upper management...they would kill it, if they knew. Where a component from the old instrument line was adequate, it was employed in the new design. After all, they had been proven reliable over years of hard use.

One of the components carried over was the vibrating pump that emptied the incubator on the old system. The application in the new instrument was similar; the water from the incubator did feed the pump, but in addition, all waste water and reagents were also run through this pump. It was called, appropriately, the "Waste Pump".
 
Since the application was slightly different, the engineer in charge tested the pump thoroughly. He ran it continually for weeks. It worked perfectly.

The new instrument, being much bigger than the old instrument and having higher throughput, was targeted in large hospitals and big reference labs. In these settings, the machine was used all day, every day. The pump was constantly being used. It proved to be up to this task.

When the installed instrument base in Europe got larger, the pump soon started failing. It took several years for the data to be analyzed and communicated back to the factory. The problem, the Failure Analysis Lab decided, was that the displacement of the pump at 50 Hz was much larger than at 60 Hz and this higher displacement caused the leaf springs that supported the pump body to fracture.

The Failure Analysis Engineer, worked with the manufacturer of the pump and determined that by reducing the coil from 21 watts to 14 watts, the displacement would be less and the springs would not fracture.

This configuration of the pump was tested by Engineering at both 50 Hz and 60 Hz. The Engineer ran it continuously for months (testing was more important now that we were ISO 9001 certified). It worked perfectly.

Unfortunately, when the retrofit pump was released to the field and installed in new instruments, the Waste Pump would at times fail to prime and the waste fluid would overflow, making a big bio-hazard mess.

Another Engineer investigated the problem and found that the 14 watt coil had indeed reduced the pump stroke...so much so that it almost did not pump at all. When the valves in the pump dried out, they did not seal well and this made it difficult to prime. Some customers had waste tubing that provided a lot of back-pressure and these were the sites where most of the problems happened.

Another factor was discovered by Field Quality: as we had saturated the large hospital and big reference lab market, the new instrument had increasingly been sold to small clinics and even doctors offices. In these settings, the machine was only used during the day and the valves in the Waste Pump dried out overnight. The next morning, the pump would not prime, leading to overflows.

Another Engineer investigated and proposed that the problem was that the tubing leading into the pump was of too small a diameter and too thin a wall. It would at times limit the flow to the pump. He redesigned the tubing. This "fix" never got incorporated.

Yet another Engineer investigated (this time with the support of an outside consultant), and concluded that the leaf spring was the biggest problem. If it had not broken at 50 Hz, the coil would not have been reduced to 14 watts and the priming problem would not have occurred. Their solution was to redesign the pump to use coil tension springs and substitute a 25 watt coil.

This version was tested extensively. The test protocol now included shutting off the pump and letting it dry out. The back-pressure was varied and documented in the test results. This pump really PUMPED...and it was reliable, even at 50 Hz.

There was one small problem: the new PUMP was much louder than the old 14 watt unit. When the Marketing manager heard the pump, he nixed it...TOO LOUD!
Not to be derailed by this setback, the team added a big resistor in series with the coil, dropping it to about 16 watts. Considering the heat load of the instrument was about 3000 watts, adding 9 more would not be noticed. But the noise persisted and the marketing Manager was un-moved...a loud pump would not be approved.

The consultant went to work on a cover for the pump to isolate the noise. When installed, it made little difference. The Engineer concluded, after several tests, that the noise was coming from the turbulence of the waste water in the tubing. When the pump ran dry, it was quiet. The noise was only heard when the pump was pumping water.

Thus the project stalled, because nobody would admit that a vibrating pump of this type was wrong for this application. [The manufacturer told us years ago, during the design of the instrument, that this pump was not designed to self prime or to run dry.]

Soon, it was discovered that the Teflon mixer tip that extended into the cell could occasionally pick up a small drop of the reaction mixture and, as Teflon is rather hydrophobic, drop it off at any time. If it fell off into another reaction cell, that assay would be contaminated and the results affected. The obvious solution was to wipe the tip of the Mix Arm against the side of the cell as it was being withdrawn, causing the drop (if present) to be pulled off into the cell where it belonged. However, the tip at rest had no defined position. The motor that shook the tip was a moving magnet design (because a moving iron motor tended to shake at 120 Hz and that splattered the assay badly. Actually, 60 Hz was too fast too, thus, requiring a mechanical adjustment to set the stroke very accurately to insure proper mixing without splattering. This adjustment was also required to lower the stroke when the instrument was powered by 50 Hz) so it could be attracted to either pole while at rest. This caused a problem: it was important to know the tip position to gently wipe the tip on the cell wall. The two error conditions due to undefined tip position were: (1) missing the wall and (2) hitting the wall forcefully, which caused noise, wear, potential damage to the cell and/or mixer tip, and the mixer tip to flick sideways (as the mixer tip cleared the top of the cell wall), sending droplets flying. The engineer thus designed a simple half-wave rectifier into the circuit that continuously biased the moving-magnet armature of the Mix Motor to a known position. This required that the polarity of the diode, coil, and magnet be controlled (where previously, the coil and magnet orientation did not matter).

This worked well, except that the half-wave rectified, 50/60 Hz bias voltage caused the tip to "buzz" continuously. The solution was to add a capacitor across the coil to provide some filtering. A small capacitor was chosen, because a big capacitor would limit the mixing action when the triac fired. Even the small capacitor caused a problem, which was not noticed initially. Soon it was found that the triac did not always fire. The capacitor limited the dV/dT and the coil limited the dI/dT. Together, they caused the triac to occasionally refuse to fire and this caused the Mix Arm to fail to shake. This caused the reaction in the cell to start slowly and affected the results. The solution was to add a SIDAC, a high-voltage bi-directional trigger in series with the coil and capacitor. This device conducted only when the voltage exceeded its threshold and when this happened, the voltage would rapidly increase, allowing the triac to trigger. That this caused the inrush current to both the triac and SIDAC to be an order of magnitude greater than allowed by the manufacturers specifications, escaped everyone.

Also un-noticed was the holding current specification of the SIDAC. The small Mix Coil drew 30% of the required holding current, thus the SIDAC would often fail to conduct fully...it lapsed into an undefined state, neither conducting nor blocking (a true "semiconductor"). In this state, the on resistance was high and the power dissipation was so high that the temperature of the SIDAC quickly reached 160 deg C, much higher than the specified max operating temperature of 125 deg C. Nobody noticed that the circuit was internally hemorrhaging...until the first instrument fire!

• A "Trap Door" seal. This type seal was common in competitors products. It was simple and effective, but had two problems: 1) typical materials were hard and abrasive and quickly dulled the trocar as it was inserted to open the "trap door" and 2) it was impossible to extract excised material out through cannulae with "trap door" seals, without having a handle or lever outside the body to hold the "trap door" open, otherwise the material catches on the edge of the "door" and jams.

• A "duck bill" seal. This type seal has the second problem listed for the "Trap door" seal, but there was no apparent way to externally open the seal to allow excised material to pass out. This mandated that the "duck bill" seal be located externally, so that it could be removed if material was to be removed.

• A "lip" seal. This type seal is similar to the "duck bill" seal, except that it is flatter, allowing it to be pulled inside-out, which allowed material to be pulled out through it. A negative is that the 2 lips can be distorted through use and fail to seal completely. This problem is transient...given time, the seals recover and seal.

The key marketing parameter was to allow simple tissue extractions, thus a "lip" seal was chosen. They decided that 2 lips were enough and fabricated the seal from silicone. The lip seal was oval shaped, with the slit parallel to the long axis of the oval. It was housed in a hard plastic housing, which had an internal rim to support the seal. The seal, in turn, had a lip that was positioned behind the rim, to keep it from pulling out.

Initially, the seal was designed to allow instruments from 5mm to 10.5mm to be used. However, after the design was almost complete, Marketing insisted that it accept instruments up to 12.5mm. The housing was revised to remove part of the rim, near the center, as this was the place where the large shafts would squeeze the seal. The revised housing was used only on the 12.5mm size cannulae.

Experience in the field with this seal design showed a few problems:

• Some instruments (with sharp corners and projections) would catch on the seal, sometimes cutting it. If the seal was cut badly, it would leak. In a few instances, the instrument caught the seal and pulled it completely out of the housing, pushing into the patients body!
 
• The lubricity of the silicone with slick, polished instruments was poor. There was a lot of drag upon insertion, which disturbed the surgeon.
 
• The design was intended to be self-sealing under insufflation pressure (the positive gas pressure which distends the abdomen, prior to insertion of a cannula and maintained during the complete procedure), as the shape of the seal was slightly concave and the pressure on it would tend to flatten it, causing the lips to be pushed together. However, when some instruments were removed, the lips ended up on top of each other. The pressure forced the upper lip to separate from the lower lip, causing a leak. When the leak occurred, the pressure would drop and the upper lip would fall back into contact with the lower lip. This happened several hundred to several thousand times a second. It made a noise (not a very nice noise)! Small leaks are normally not a serious problem, since the insufflation pressure is regulated. However, when the seal announced that it was leaking by "honking", it was a disaster.

The company, of course, "fixed" these problems. A silicone lubricant was applied to the seal. This improved the lubricity greatly. Different silicone materials were tried, until one was found that had some resistance to being cut and torn. A guide was added above the seal to restrict the insertion and removal angle. Little else could be done about the noise, but it seemed to happen very infrequently.

Every cannula manufactured was leak tested. This 100% test was judged to be necessary to ensure that all products were leak free. Yet there were continual field complaints about leakage. The company had a standing committee, composed of every middle manager (except for the Engineering manager), who reviewed all complaints on a weekly basis. This committee referred these complaints to the proper group for analysis and resolution. Often the Engineering department got copies of these complaints, but rarely ever saw the offending product. Thus, it was difficult to pinpoint what was leaking.

After several years it was discovered that the silicone seal absorbed the silicone lubricant and expanded. This change in size and shape caused the seal to be tight in the housing and to distort, causing leakage. It was futile to test immediately after assembly, because the lubricant had not yet been absorbed (waiting 24 hours before testing showed that the seals were not working).

Investigation showed that it was well known that silicone elastomers absorbed silicone lubricants, and subsequently had different physical characteristics (not just size). There were, of course, silicone lubricants that would not be absorbed into the silicone elastomer of the seal. However, they formed small drops on the seal surface, instead of a smooth coat. There were solutions to this, in the form of blends of two different silicone lubricants, one of which was incompatible with the elastomer, which would neither be absorbed or "ball" up.

Another approach to solve the same problem was to blend a lubricant into the seal, prior to molding.

Just as the seal vs lubricant problem was seemingly being solved, the problem of having the seal pull out of the housing was being addressed. One approach was to modify the mold for the housing.. In order to prevent seals from being pulled out of the housing (which only occurred with the big, 12.5mm version) the split rim design could be abandoned, returning to the original continuous rim, which gave better support for the seal. It would, however, internally hemorrhage each time an instrument larger than 10.5mm was used, because there was insufficient room between the seal and the inside of the housing.

Another way to fix this same problem was proposed: to glue the seal into the housing. This was tried and found to work well. It was rejected because of the difficulty of applying the adhesive and the cure time of the adhesive. In order for this to be a good solution, the company would need to purchase a robot to apply a fixed and consistent glue line to the housing and then devise a process for storing the assembled housings with seals for 24 hours before packaging. The glue has a strong smell of acetic acid prior to being cured and, depending on the packaging, the cure could be compromised by limiting air, plus, it was judged not pleasant to open a new, sterile cannula and smell vinegar!

Ultimately, the glue idea was rejected and the housing was modified.

As last seen, the company was still trying to fix the problems with the lip seal. It was still leaking and still squeaking. Oh, and the company was on the market, with few buyers...

re the Prologue: How do you normally proceed when there is no evidence that what you know is sensible, has no objective evidence to support it? Why does this example appear obviously wrong to you, but not to the characters in the story? Is it really that silly?

re Act 1: The lamps only burned out when the "Lamp Test" button was pushed...so how about changing what the "Lamp Test" button does? Like turning all the lamps on, one-at-a-time? Or turning one-lamp-on-at-a-time in a "ripple" sequence? Would these ideas just be another step down the same, wrong path? If so, what is the primary "wrong" idea?

re Act 2: Would adding a small water leak (or a solenoid valve that sprays water in the drain line periodically) to keep the valves in the waste pump wetted, so that they never have to operate when dry, be another in the same series of mistakes

re Act 3: When the problem was discovered 10 years after being designed, near the end of life of the instrument, how would you suggest that the problem be fixed? How far back would it be necessary to go in this saga to properly fix ALL the problems?

re Act 4: what is the real problem?
 

What is needed at junctures such as these is a paradigm shift...a different way of approaching the problem. As an exercise, try to devise one new paradigm that is related to your most serious problem. Note that it must be NEW, ie, never considered by anyone before...as such, it will probably be silly. But if you can think of one new (even a silly) paradigm, there are others, and some of them will be better than what you are doing now! Why do you think that what you are doing today is not silly? What will technically competent people say about your work 100 years from today? Or 10 years from today, in the fast pace of change in the 21st century? Would you be embarrassed to describe your designs (and their problems over their life) to an audience populated by truly competent peers? Why?