Space Shuttle Atlantis Tanking Test in Progress

Space Shuttle Atlantis at the Pad during the December, 18th tanking test in support of ECO sensor troubleshooting.Today’s tanking test gathers additional data on ECO sensor (What is an ECO sensor?) failures with the ECO sensor circuits being fully instrumented. In order to gather data, an ECO sensor failure is needed. Before the start of the tanking test, it is anticipated that ECO sensor #3 will fail wet and all others will work correctly.

2:08a: The tanking test just went into fast fill mode. During the last launch attempt, ECO sensor #3 failed only a few minutes into fast fill mode.

8:14a: If I got the commentator right, there already is a failure of a sensor. That would be good.

8:21a: NASA TV commentator: “It appears that #1 sensor has failed, but the hydrogen console has not officially made that call. So they are still looking at it. Number 2 sensor has been intermittent. The engineers at the main propulsion system console are very intently looking at their data right now. The intermittent data on sensor #2 is somewhat of a surprise because we haven’t seen that before on that sensor.”

So as it looks, the ECO sensor system behaves again different from the last tanking, which was at the December, 9th launch attempt.

8:23a: NASA TV: “The propulsion console has provided confirmation that sensor #1 on liquid hydrogen has hard failed, which is what we wanted. The #2 and #3 are intermittent, we are watching for what those are going to do. We have not seen any activity of number 2 sensor on prior tankings.”

8:26a: NASA TV: “The hard fail on #1 is definitely what we need in order to do the troubleshooting at the pad later this morning.”

So far, this looks very good. Based on analysis done in the past days, sensor #1 is not expected to return to a non-failed state.
So this one will hopefully a good candidate for the troubleshooting which will then be able to pinpoint the culprit. The intermittent failures of #2 and #3 are outside of expectations, at least as far as I understood the analysis done. However, if they now remain functional there still may be a good explanation for that. Let’s see how things evolve…

9:32a NASA commentary just announced that engineers have made up their troubleshooting plan. “The TDR sensor equipment will be focussed on sensors #2 and #3 as they have been intermittent and it is the greatest source of interest to see where that intermittent reading is coming from. Number 1 and 4 will be recorded continuously. … Right now they are configuring for the console activity that would be required once we go into stable replenish … very shortly. … So the final inspection team is preparing to enter into the pad, but we probably have another half an hour or so before we are into stable replenish.”

9:48a NASA TV: “The liquid hydrogen tank is full now and in stable replenish. The liquid oxygen (LOX) is at 80%. We’ve probably about another 25 to 30 minutes or so before liquid oxygen is in stable replenish.

10:00a: NASA TV: “We will begin to drain the tank between 1p and 2p and once the tank is drained the teams go back to do additional trouble shooting. A lot of data has already been collected on sensor #1 because it failed early.”

10:13a: “The problem with the liquid ox pump appears that a fuse has blown, so they are setting to the backup lox pump, … which will us back to where we were close to the stable replenish. This will take around 45 minutes to an hour. Why the fuse blow is still not clear”. So we have some delay in the troubleshooting plan.

11:45a: I needed to go out for a while, thus no updates. Atlantis is now in stable replenish state and troubleshooting is right now happening at the pad. The NASA TV commentary just announced that teams are now looking at ECO sensor #4 and the 5% sensor. Here are some pictures from the testing:

The picture above shows a room that is right inside the mobile launcher platform. This is where data today is analyzed.

11:52a: NASA TV “The … team is now on the mobile launcher platform and gathering data on sensor number 2 and 3.”

11:53a: a side-note: the ISS spacewalk just has completed, the spacewalkers are back into the airlock, which is now being repressurized. They did not find any obvious problem, but collected samples to be returned to earth in January on board STS-122.

12:06p: NASA TV: “Data will be collected remotely during de-tanking.” The equipment is currently being set up for this process. People are permitted close to the space shuttle only during stable replenishment phase (which essentially means no tank or detank operation is taking place). During detanking, they must leave for safety reasons. The set up process can be seen in this picture:

6:12p: and now those members at the controls can be seen leaving the control room. Everything is now set up for remote monitoring. The NASA TV commentary tells that good data has been gathered. The tank will be drained to 5% while instrumentation is on. Once it is fully drained, members of the red crew will return for further troubleshooting.


And on this picture, a wire can be seen. This wire is tapped into the ECO sensor circuits and connects to the time domain reflectometer (TDR) equipment that is used to do a detail analysis of the circuits. TDR is an off-the-shelf technology in wide spread use e.g. by cable companies to detect faulty lines.


around 12:25p: I was too slow for a quote, but NASA TV commentary stated that everyone seems to be quite happy with the data gathered. It sounded like a successful test. The commentator also announced that more – and official – information is available in the news briefing later today. For me (being six hours ahead), it probably is too late, but I’ll then do a wrap-up tomorrow morning. Let’s hope that they have actually managed to find the culprit and nail it down – and also let’s hope that the rest of the day (detanking test) also goes well.

12:40p: it looks like the red crew is now leaving the pad after finishing late inspections. I noticed they used binoculars and telescopes during that process. I (now) think they also visually checked for ice buildup and maybe some other things. Probably just part of the usual procedure. Here, the red crew can be seen leaving the mobile launcher platform (the last frame I was able to capture, sorry for the blur…).


The NASA TV commentator announced a few minutes before that engineers are ready for detanking once the red crew leaves. So I assume detanking will begin shortly.

12:48p: NASA TV: “The final inspection team has now left the pad … In the mean time we had a failure of ECO sensor #3, the second sensor that had failed. It was intermittent … and we are collecting data on sensor #3 … right now which is able to be done remotely much as we got the initial data from sensor #1 when it failed earlier today.”

I didn’t get the full quote, but as far as I understood, sensor #3 has now also a hard failure. That, IMHO, would be somewhat bad news because it means that the condition will change once the tank is loaded. However, this is just my uninformed guess. Also, on a actual launch, the shuttle would be long gone and fly in orbit. So that may not mean anything at all. I should stop speculating ;)

12:52pm: NASA TV: “We are close now to start the external tank draining operation. and we did indeed get the kind of data that we needed to get to the (bottom of?) that trouble.”

12:58p: NASA TV: “We did see sensor #3 also fail, so we try to get some data from it before we start draining.” So it actually has a hard fail and NASA uses the opportunity to get something out of that. To me this sounds like it failed just a few minutes ago, otherwise the red team would have looked at it (wouldn’t they?).

01:00p: NASA TV: “on the recently failed ECO sensor #3” – and indeed, it recently failed. Right in time, one could say…

3:00p: as I wrote, I have been away from my computer for a while. NASA TV has ended coverage of the tanking test in the mean time, so I can not provide any more status updates. I’ll try to do another post after the news briefing, but that will probably be tomorrow (I hope I’ll be able to get hold of an archived version of the news briefing). All in all, it looks like an excellent day at Kennedy Space Center, with good results that will bring up the root cause of the ECO sensor issue. Let’s hope that I have the right reception of today’s events.

Thanks to everyone who cared reading my post! I hope it has been useful and will be as a reference.

I’ve now also written down some results from the post-test news briefing.

Tuesday: ISS Spacewalk and Shuttle Fueling Test…

Tomorrow is a busy day for NASA – both on earth as well as in orbit. Seems to be “troubleshooting Tuesday”: The international space station crew performs a spacewalk to check out what is wrong with the orbiting laboratory while the ground crew at Kennedy Space Center checks out their supply vessel. Both activities are in support of the international space station program. Let’s hope everything turns out well.

But now let me quote a mail that I received from NASA, it is an excellent wrap-up of tomorrow’s activities, including ways to experience them first hand:

NASA Television will provide simultaneous live coverage of a spacewalk by the International Space Station crew and a shuttle fueling test at NASA’s Kennedy Space Center, Fla., on Tuesday, Dec. 18.

Expedition 16 Station Commander Peggy Whitson and Flight Engineer Dan Tani are set to venture outside the station at about 6 a.m. EST to perform a detailed inspection of a giant rotary joint where contamination was found last month. The joint is used to rotate the starboard solar arrays of the complex to face the sun. The astronauts also will devote part of the spacewalk to an inspection of a device that tilts the starboard arrays toward the sun. The device, known as a Beta Gimbal Assembly, experienced unrelated electrical problems last weekend.

NASA TV’s public channel will begin coverage of spacewalk activities at 4:30 a.m. A briefing will follow the spacewalk, originating from NASA’s Johnson Space Center, Houston, no earlier than 1:30 p.m.
Reporters will be able to ask questions from participating NASA sites. The briefing participants are:

— Mike Suffredini, International Space Station Program manager
— Ginger Kerrick, International Space Station spacewalk flight director
— Tomas Gonzalez-Torres, International Space Station spacewalk officer

At 7 a.m. EST, Tuesday, NASA TV’s media channel will begin coverage of a fueling test of space shuttle Atlantis at Kennedy’s Launch Pad 39A. The fueling test will assist engineering efforts to resolve a problem with an engine cutoff sensor system that prevented Atlantis’ launch attempts earlier this month. Reporters will be notified of any plans to hold a news briefing following the test’s conclusion.

For NASA TV streaming video, schedules, and downlink information,visit:

http://www.nasa.gov/ntv

For more information about the space station and the Expedition 16 crew, visit:

http://www.nasa.gov/station

For more information about space shuttle Atlantis’ upcoming STS-122 mission, visit:

http://www.nasa.gov/shuttle

STS-122: Atlantis being instrumented…

Space shuttle Atlantis stands on Launch Pad 39A at NASA's Kennedy Space Center in Florida. Photo credit: NASA/Cheryl MansfieldTroubleshooting efforts at Kennedy Space Center (KSC) continues: Space shuttle Atlantis has been instrumented for a tanking test scheduled to begin Monday. In parallel to the work at KSC, engineers at other NASA centers gather data about how the elements of the ECO sensor system should respond during the tanking test. This provides a baseline against which the actual tanking test results can be compared. Unfortunately, it is not sure if the failure will re-occur during the tanking test. But according to the latest findings it is quite likely.

Please also let me quote the relevant part of NASA’s shuttle home page:

Engineers and technicians at NASA’s Kennedy Space Center continue preparations to evaluate the hydrogen fuel sensor system on space shuttle Atlantis’ external fuel tank during a procedure next week. Working at Launch Pad 39A where Atlantis remains pointed to space, workers attached wiring to the cables that lead from the aft compartment of Atlantis to the external tank’s engine cutoff sensor system. Engineers will use the special instruments next Tuesday to send electrical pulses into the wiring and look for indications that will show the location of the issue that caused the sensors to return false readings last week. The failed readings showed up during launch countdowns on Dec. 6 and Dec. 9. Launch controllers postponed the liftoff on both occasions to find out the problem and develop solutions. Evaluations of the instruments themselves are also under way to show technicians what a normal reading on the external tank looks like. Those readings will be compared to the results from the test Tuesday during which the tank attached to Atlantis will be filled with super-cold liquid hydrogen. NASA is targeting Jan. 10 as the next possible launch opportunity for Atlantis on mission STS-122. Atlantis will carry the European-built Columbus laboratory to the International Space Station.

STS-122 now set to Launch January, 10th 2008

The target launch date for space shuttle Atlantis STS-122 mission to the international space station has now been moved to January, 10th. Originally, it was set for January, 2nd when an ECO sensor problem made it impossible to lift off during the December 2007 launch window. The January, 10th date has now been selected to allow NASA workers to get some rest. They have been extremely busy. The holiday period is now a perfect time to make sure everybody is in a great shape when it comes to the next launch attempt.

As far as I know, launching exactly on January, 10th will not affect the overall shuttle launch schedule for 2008. However, I suspect even a further one day delay means trouble for the flight plan.

And finally, this is what the NASA shuttle home page has to say:

NASA’s Space Shuttle Program managers have targeted Jan. 10 for the launch of shuttle Atlantis’ STS-122 mission to the International Space Station.

“The workforce has stepped up to and met every challenge this year,” said Wayne Hale, Space Shuttle Program manager at NASA’s Johnson Space Center. “Moving the next launch attempt of Atlantis to Jan. 10 will allow as many people as possible to have time with family and friends at the time of year when it means the most. A lot has been asked of them this year and a lot will be asked of them in 2008.”

The liftoff date from NASA’s Kennedy Space Center, Florida, depends on the resolution of a problem in a fuel sensor system. The shuttle’s planned launches on Dec. 6 and Dec. 9 were postponed because of false readings from the part of the system that monitors the liquid hydrogen section of the tank.

Shuttle Troubleshooting continues…

Just a short note, a longer one will follow (maybe tomorrow, it’s late over here…). I have just listened to the news teleconference with Mr. Wayne Hale. I am very glad to say that he is determined to fix the ECO sensor issue and sees good chances in doing so. To do that, a tanking test will be conducted next week.

Probably the following quote describes the whole situation “The primary goal is to troubleshoot the system as it is and restore its functionality. We would only consider other measures if we fail with this.

I hope they will succeed with that and we will have a great space shuttle launch in early January.

Shuttle Manager Hale’s Teleconference Statements

These are the notes I have taken during the December, 11th teleconference. I am posting them as I have taken them. They are largely unedited, but IMHO speak for themselves. Direct quotes are within quotation marks, the rest are my observations”. I penned this down during the teleconference, so I am pretty sure it is the exact wording.

“We set up 2 investigation teams. One is a near term team working on the current vehicle, second is a longer term team with experts from all around the agency.”

“do instrumented tanking test next Tuesday. Some instrumentation that we can put on some appropriate places at the circuits … We can capture the location in circuit in TDR, a commercially available technology … we have a high degree of confidence in pinpointing the location and once we know the location we can put together a fix.”

“STS-122 launch could definitely be a bit later than January, 2nd”

“I’ve been committed to fixing it. An intermittent electrical problem is really hard to fix. We thought we had fixed it, but we didn’t. So I might say we are RE-committed to fixing it”

Reporter: How concerned are you about the whole eco sensor system (touching on the email where Mr. Hale considered retiring this system as unreliable).

“This low level cutoff capability is a safety system that has never been used in flight… like a seat belt. If you don’t have anything bad, you probably don’t need it. If you needed it and it didn’t work it could be really bad. We would like to have this system functional and we would like to restore functionality of this system”

“There are other test, bench test of the equipment at manufactures in parallel to tanking test. Tanking test is a hazardous operation.”

Reporter: if STS-122 does not launch on January, 2nd, when then? “I let you know when we have our tanking test done and we have some data”

“The problem only occurs when we have cryo conditions present” It all works well at normal temperaturs. We are gonna to find out where the problem is.”

If the LCC criteria (number of working ECO sensors) would be changed: “Our point is to try to fix the problem and then go back to the previously LCC, rather than speculate let us wait and try to fix this problem.”

“TDR is not flight equipment, its not qualified. Its a ground system only.”

Once again on the email about retiring the eco sensors: “Our thinking has evolved from Friday when I wrote that little note.”

“We hope it repeats one more time on our test next week when we have the instrumentation”.

“Until we come to the bottom of this mystery, we are in no better shape launching any other orbiter” when asked about swapping assembly flights or orbiters to get off the ground.

“a single circuit is around a hundered feet of wire from the PSB to the sensing element.”

Test details:
“We have to physically cut the wires, we are talking about the ECO and the 5% sensor, which is in the same pass-trough and connector. We have to have people present to run that equipment. We send the red crew out during the stable replenishment phase. And if the problem is as in the past, it will stay with us during the stable replenish phase. We can not have people during tanking for safety concerns”

On a Christmas break: “We are thinking about taking a few days off to allow our folks to have a few days with their families. We’ll make that decision shortly after the tanking test.”

“The splicing of wires in the aft compartment is a standard procedure. We have identified a place where we can access the circuits that are readily accessible.”

“And then restore the circuits together when we are done with the tanking test.”

“Take the sensor’s wire harness and the LH2 pass trough connector and put them in a facility with cryo fluids and monitor how these tings respond in a lab setting. either liquid helium, warmer, or liquid hydrogen (lh2). we look at all these piece parts work. we did piece part testing before, we are doing integrated circuit from the lh2 pass through to the sensors themselves.”

“The liquid helium low level cutoff ability (LLCO) was present from the beginning , I am pretty sure it was sts-1 (99% positive). It is not a new system that has changed dramatically in design or manufacturing. The FPR is not significantly different the past 10/15 years than it is today. Just enough reserves to protect it from normal engine and operating procedures. The voltage indication system has been on sts 118 and 120 only, which were flawless.”

“The primary goal is to trouble shoot the system as it is and restore its functionality. We would only consider other measures if we fail with this.

Next tuesday full tanking test? “We need to fill the tank fully because as I said, it is our safety requirement to have people on the pad we need to be in stable replenish. It is reviewed which aux systems need to be operating. The launch team is finalizing the procedures.”

“I can not put my finger at anything that is especially difficult, its just normal operations in an unusual environment.”

ISS Stage EVA next week, Thursday afternoon, 2ET, possibly next Tuesday, SARJ

Space Shuttle ECO Sonsors: an in-depth View

Space Shuttle ECO Sensor during Testing.After the scrub of space shuttle Atlantis December 2007 launch window, everyone is interested in the ECO sensors. That shuttle component is responsible for the scrub. Unfortunately, detailed information about it is hard to find.

However, I was able to obtain some good information. Most helpful was NASA’s “STS-114 Engine Cut-off Sensor Anomaly Technical Consultation Report“. I also used other NASA sources for my writeup, including information conveyed at the post-scrub press conferences.

Let’s start with some interesting fact that space shuttle program manager Wayne Hale provided in a press conference. According to him, the ECO sensors are an Apollo heritage. Their design dates back to the 1960s. Consequently, they are analog “computer systems”, which look quite strange compared to today’s technology.

I could not find any indication of sensor malfunction prior to STS-114, the “return to flight” mission. However, I have been told that pre-STS-114 flights did not have the same rigor checks in the flight procedure as they exist today. So it may very well be that there always were problems with the sensors, but these were “just” never detected.

It should also be noted that there was never a space shuttle main engine cutoff due to an ECO sensor (I need to correct this a bit later – but let’s keep it this way for the time being). It is believed, however, that on some flights the cutoff happened just a second or so before the ECO sensors would have triggered one. The amount of fuel left in the tank can not be analyzed post-flight, as the external tank is the only non-reusable component of the shuttle stack and lost after being separated from the orbiter.

But now let’s dig down into some hard technical facts
: A good starting point are the graphics that NASA posted on the space shuttle home page. I’ll reproduce them here, but due to the blog theme, they are a bit small. Click on each image for a high-res version. It will open up in a new window, so that you can read along.

There is a drawing that puts together all the pieces. It is an excellent starting point:

Space Shuttle ECO Sensors: OverviewA brief word of caution, though: the picture titles “LH2 ECO Sensor Locations” for a good reason. It is about the liquid hydrogen (LH2) tank sensors. There are also others, as we will see below. Let’s for the time being stick with the LH2 one. As far as I know, the LH2 sensors were also the only trouble source in recent shuttle launch attempts.

This is also where I need to correct myself. There actually have been main engine cutoffs due to ECO sensors, but none of them happened due to the liquid hydrogen sensors. As far as I know, there were three missions where it happened and among them were STS-51F and STS-93.

The image shows that the ECO sensors are located right at the bottom of the tank – which makes an awful lot of sense, as they should indicate depletion. There are four of them mounted in a single row on the shock mount. Each of them has their housing containing the actual sensing element. Even though this is not show on the above overview, let me add that there is are a lot of additional components that make up the so-called “ECO sensor”. That can be nicely seen in this schematic:

Space Shuttle ECO Sensors: Overall Schematic
The actual sensing element of the space shuttle's ECO sensor system.First of all, you’ll probably notice that it is more appropriate to speak of a “sensor system” than just of a “sensor”. If we talk about sensors, most of us simply think about the actual sensing element, seen to the right here. Obviously, that takes us far too short. You must think about the whole system to understand the problem. So think sensor element, electronics and electrical connections. All of this makes up what we call the “ECO Sensor”. In my personal opinion, there is a lot of misleading information and discussions on the public Internet these days. Part of this misunderstanding IMHO seems to stem back to the “sensor” vs. “sensor system” issue. Many folks express that they don’t understand why “such a simple sensor issue” can not be fixed. I guess that was even the motivation to write this post, but, hey, I am becoming off.-topic. On with the technical facts.

Next, you’ll notice that the ECO sensors are just few of the many sensors that make up the tank level information (the “point sensors”). All of these sensors are the same. The ECOs are in no way special, except for their name. ECO stems from “Engine Cut Off” and is attributed to the fact that these sensors are a emergency line of defense to shut down the engines if other things have already gone wrong (if all goes right, the ECOs are never used, but it is the ECOs that ultimately determine the fact that something went wrong…).

If you count, you’ll find twelve sensors: the four ECO sensors, one 5%, two 98%, one 100% minus, two 100%, one 100% plus and one overfill point sensor. Note that there are sensors both in the liquid hydrogen (LH2) and liquid oxygen (LOX) tank. Each of them has twelve, so there is a total of 24.

A notable difference is the location of the ECO sensors: for LH2, they are at the bottom of the external thank, while for LOX they are in the feedline inside the orbiter. In plain words that means that the LOX ECO sensors report very late while the LH2 sensors report early in the process of tank draining. This can be attributed to the fact that a fuel(LH2)-rich engine shutdown is required. I also assume that the risk of fuel pump overspeed and explosion is by far higher for the LH2 part of the system (but that just my guess, found no hard fact backing it).

The number of sensors at each position tell you something about their importance: it for sure is no accident that most positions are covered by one sensor, the 98% and 100% locations have two and the depletion location has four! Obviously, depletion is a major concern.

Which brings us to the point: why four? Let’s spell it out if it is not clear yet: it’s “just” for redundancy and backup. If there would be just one sensor, a single-sensor failure could be fatal. If it failed dry, it would cause an unnecessary (and comparatively risky) launch abort, if it failed wet and something else goes wrong, it could lead to vehicle destruction. Either way is not really desired, though obviously one case is better than the other.

To mitigate that risk, there are four sensors. But how put these to use? A simplistic approach could be that a poll is taken and the majority wins. So if we have one sensor telling dry and three telling wet, we would go to wet. Obviously, there would be a problem with a 2 dry/2 wet state. So our simplistic model is too simplistic. But I hope it conveyed the idea. What the system really does is a bit different:

First of all, there is a construct called “arming mass”. Keep in mind that the ECO sensors themselves are “just” a backup system to handle the case when other things have gone wrong before. Space shuttle computers continuously monitor engine performance and calculate fuel used. So there is a rough idea of how much fuel is left in the tank at any given moment. However, these calculations may not be 100% perfect and may not detect some malfunction, thus it is risky to rely on them alone. To mitigate that risk, the ECO sensor system has been put in place.

Now let’s take an extreme example. Let’s say an ECO sensor switches to dry just one second after launch. Would you trust it and assume the tank is already drained? I hope not. There are some points in flight where both logic and physics tell us the the tank can not be depleted. In fact, during most of the ascent it can not. But when we come close to main engine cutoff, then fuel may actually be used up. Only at that stage it is useful to look at the ECO sensors. This is what “arming mass” is all about. The shuttle’s computers continuously compute estimated fuel left and only when the estimate comes within the last 8 to 12 seconds of fuel depletion, the ECO sensors are armed.

This has some bonus, too. If an ECO sensor indicates “dry” before we reach arming mass, we can assume the sensor has failed. So that sensor will no longer be able to cast its vote when it later comes to aborting the launch. Please note, however, that it is not possible to detect a “failed wet” sensor in the same way. Sensors are expected to be “wet” during ascent and doing so obviously does not disqualify a sensor.

The ECO sensor mountpoint inside the space shuttle's external tank. As can be seen, they are mounted close to each other.Once the arming mass has reached, shuttle computers look at those sensors with a healthy status. If a single sensor indicates “dry”, computers initially assume a sensor failure. Remember: all sensors are mounted at the same location (see picture to the right), so they theoretically should indicated “dry” all at the same instant. However, that sensor is not disqualified. When now any second of the healthy sensor joins the other one in reading “dry”, shuttle computers assume an actual tank depletion.

They do not wait for the remaining qualified sensors, in a case now assuming these have failed “wet”. So whenever two qualified ECO sensors indicate “dry” after the space shuttle has reached “arming mass”, an abort will most probably be initiated. That means the space shuttle main engines will be cut off in a controlled and non-destructive way (which means a fuel-rich shutdown). Depending on when and how exactly this happens, it may lead to either an abort to the transatlantic landing (TAL) sites or an abort to orbit (ATO). I guess it may even be possible to reach the desired orbit with the help of the orbital maneuvering system if the engine cutoff happens very soon before its originally scheduled time.

Please let me add that the actual procedure for tank depletion must be even more complicated than briefly outlined here. For example, what happens if three of the ECO sensors disqualify themselves by indicating “dry” early in the ascent? Will the remaining single sensor than decide about launch abort? Also, what happens if all four fail early? I don’t like to speculate here, if somebody has the answer, please provide it ;) In any case, you hopefully have gotten some understanding now that the ECO sensor system and putting it to use is not as simple as these days it is often written on the web…

Now let’s look a little bit about where the sensors are located. If you paid attention to the above drawing, you have noticed the black lines which separate parts in the tank from parts in the orbiter (and yet from those at mission control center on the ground).

The best picture of the actual ECO sensor housing I could find is this one:

Space Shuttle ECO Sensors during a test procedurePlease note that it shows the ECO sensors during a test, in a test configuration. The mount is different from the actual one in the external tank.

The computers driving the sensors are located in the orbiter’s avionics bay:

Space Shuttle ECO Sensors: Orbiter Avionics BaysThis, and the following, drawings mention the “point sensor box”, PSB for short. Remember that the sensors together are the “point sensors” and the ECO sensors are just point sensors with a special name and function. NASA also lets us know where exactly the point sensor box is located in the shuttle’s aft:

Space Shuttle ECO Sensors: Orbiter Aft Avionics BaysAnd finally, we have some more information on the point sensor box itself:

Space Shuttle ECO Sensors: Functional Block Diagram of Point Sensor BoxThe point sensor box interprets sensor readings. The sensor elements provide a voltage. Certain low voltage level means “dry” while certain high voltage levels are interpreted as “wet”. However, somewhat above the “wet” levels, they indicated “dry” again. This level is reached when there is an open circuit.

NASA also provided an the exploded view of the point sensor box:

Space Shuttle ECO Sensors: Exploded View of Point Sensor Box
To me, it just looks like a box for electronics and I do not get any further insight from looking at the drawing. But anyways – it’s nice to know…

I could not find pictures of the not-yet-mentioned sensor system parts: the connectors and cables. Somehow the in-tank sensors and the on-board point sensor box must be connected to each other. This is done via some cables and connectors. Those must also be looked at when thinking about the system as whole. Especially as the failure reading we see points to an open circuit. I have read that some of the cable are below external tank foam. So its not easy to get to them.

I have heard that cryogenic temperatures are probably part of the trouble. Because failure readings seem to happen only when the tank ins filled (and thus very cold). One could assume that shrinking of ultra-cold material is part of the problem, but again, I have not found any credible references for this – or any other thermal results.

So it is now probably time to going right to the source. Below, find reproduced the deep technical description from the STS-114 paper quoted right at the start of this posting (quoted text in italics):

The MPS ECO system consists of point-sensors installed in the ET liquid hydrogen (LH2) tank and the Orbiter’s liquid oxygen (LO2) feedline. Point sensor electronics are designed to condition signals and to provide appropriate stimulation of the sensors and associated wiring and connectors.

Space Shuttle ECO Sensors: Overall Schematic

The point sensor electronics interprets a low resistance at a sensor as the presence of cryogenic liquid, which provides a “wet” indication to the Multiplexer/De-Multiplexer (MDM) for use by on-board General Purpose Computers (GPCs) and the ground Launch Processing System (LPS). Conversely, a high resistance is interpreted as a “dry” indication. The point sensor electronics include circuitry suitable for pre-flight verification of circuit function and are designed to fail “wet”. For example, an open circuit in the sensor, or an open or short in the signal path, will provide a “wet” indication to the MDM. The system is then activated and checked out during launch countdown and remains active during ascent.

The actual sensing element of the space shuttle's ECO sensor system.An ECO sensor is depicted in the next Figure. The sensor consists of a platinum wire sensing element mounted on an alumina Printed Wiring Board (PWB) and is encased in an aluminum housing. The sensing element acts as a variable resistance which changes on exposure to cryogenic liquid. This resistance variation is detected by post-sensor (signal conditioning) electronics and is used to generate either a “wet” or “dry” indication as noted above.

Space Shuttle ECO Sensors: System Overview

The ECO system is designed to protect the Space Shuttle Main Engines (SSMEs) from catastrophic failure due to propellant depletion. Flight software is coded to check for the presence of “wet” indications from the sensors within 8 to 12 seconds of SSME shutdown. The software rejects the first “dry” indication observed from any of the ECO sensors, but the presence of at least two more “dry” indications will result in a command to shutdown the SSMEs (i.e., “dry” indications from two of four “good” sensors are required for SSME shutdown). Early SSME shutdown would probably lead to a contingency Trans-Atlantic (TAL) abort. A failed “wet” indication cannot be detected. The system is designed so that LO2 depletion should occur first. However, a failure “wet” indication of three of the four LH2 sensors, coupled with an SSME problem that results in early LH2 depletion, could result in catastrophic failure of a SSME. Failure probability is considered remote, but would almost certainly be catastrophic to the flight vehicle. The system architecture addresses redundancy with one point sensor box containing four groups of sensor conditioner circuit cards. Each card can accommodate one hydrogen and one oxygen sensor. Each card group has its own power converter and one sensor conditioner card from each group services a pair of ECO sensors (again, one hydrogen and one oxygen). Wiring for each of the eight ECO sensors is split into one of two groups of sensors which are routed through separate Orbiter / ET monoball connections.

Let’s wrap-up: I hope you got a more in-depth view of the ECO sensor system by reading this post. At least, I think I have so by doing the research and writing it. Remember that I am no expert in this area, so I may be wrong. If you spot something that needs to be corrected, just drop me a note, for example in the form of a comment.

In regard to recent (STS-122…) developments, the question now is: what should be done if the root cause of the ECO sensor system failure can not be found. I don’t know, I miss too many facts. and my understanding is limited. But my guess is that if there can be rationale found to fly without it, that’s probably the best option to carry out. But hopefully tanking tests will show where it is flawed and a solution can be applied. Either way, I trust those wizards at NASA (and its contractors, of course). They have the have the training, they have the insight and they have the excellence. What else could one ask for?

Astronauts have left Kennedy Space Center

The astronauts have left Kennedy Space Center, but not without a big thank you to the launch support guys:

“We want to thank everyone who worked so hard to get us into space this launch window,” the astronauts said in a statement. “We had support teams working around the clock at KSC, JSC, and numerous sites in Europe. We were ready to fly, but understand that these types of technical challenges are part of the space program. We hope everyone gets some well-deserved rest, and we will be back to try again when the vehicle is ready to fly.”

They are now back to Houston, where they will continue their practices in support for a space shuttle Atlantis launch in January 2008. The launch is scheduled to be no earlier than January, 2nd. The date is obviously affected by the result of the ECO sensor troubleshooting that is currently being conducted. First news on that troubleshooting effort is expected on Tuesday.

No Space Shuttle Launch in December 2007…

NASA has waived off any further space shuttle launch attempts for the December launch window. NASA’s shuttle home page has a quick note about that:

The launch of space shuttle Atlantis has been rescheduled for no earlier than Jan. 2, 2008. The postponement will give engineers time to evaluate false readings from the engine cutoff sensor system that measures liquid hydrogen in the external tank.

As far as I know, a January, 2nd launch will be around 5:45am ET.

It is actually no surprise to me, given the new sensor problems. There is not much more news available as of now, I will keep you posted as I get updates.

Shuttle launch day? Unfortunately not: scrub!

Space Shuttle Atlantis LAUNCH HAS officially BEEN SCRUBBED. This post contains a full log of the order of events from tanking begin at 5:55am up until conclusion of the first post-scrub press briefing at around 8:30am.

Today should have seen the second launch attempt for space shuttle Atlantis’ STS-122 mission to the international space station ISS. Atlantis should have deliver the European Columbus lab module to the orbiting complex. Read why this now doesn’t happen…

Tanking has begun at 5:55a ET and so far everything is proceeding nominally. At around 6:40a a first status of the ECO sensors, responsible for a three-day launch scrub, is expected. All for sensors must work perfectly today, otherwise the launch will be scrubbed. If all goes well, Atlantis will lift-off at 3:21pm ET, within a very short one-minute launch window. Weather looks favorable, with just a 20% chance of weather prohibiting the launch.

Liquid hydrogen sensor number 3 has failed!

tanking has begun for space shuttle Atlantis second launch attempt on December, 9thAt 6:25a, guys in the control center look relaxed. Let’s hope it remains that way…


All four ECO sensors now indicate “wet”. This is good, but not yet a relief. The problem that caused launch scrub on Thursday did only show up after a series of test commands were sent to the sensors. As of my information, we are still about half an hour to an hour away from these checks.

6:47a: tanking has changed to “fast fill” mode. Last time, the ECO sensor problem occured 16 minutes into fast fill. According to the NASA TV commentator, we should get results of the sensor test in about half an hour.

6:52am: Liquid hydrogen sensor 3 has failed! A minute before that, the NASA TV commentator announced that all four sensors had passed the check, but then, he sadly had to announce that ECO sensor number three failed after a few seconds. Based on the information provided in yesterday’s press briefing, a launch scrub is highly probable.

7:00am: NASA will tank for another half hour. The team is now doing troubleshooting. No launch scrub yet!

7:02am: NASA TV commentator: “the ground rules layed out that we have to have four sensors to proceed with launch. And we have had sensor number 3 fail. So, we are going to do some trouble shooting over the next half hour. At that point we would stop, asses whether we do any further testing at that point and then drain liquid oxygen. Liquid hydrogen will stay in filled configuration.” … “An official launch scrub has not yet been declared, but according to the plan, the rest of the morning is evolving into a tanking test.

7:09am: NASA TV: “The MMT has asked the propulsion console to come up with a time line on how long it would take to drain the liquid oxygen and then drain liquid hydrogen to 5%”. “The mission management team will … shortly … talk about what our official status will be. Although we have not officially declared the scrub, the commit launch criteria does not permit to continue…

7:13am: NASA TV: “We continue to fill the tank for another 15 minutes”. Me: Note that this is not in support for a launch attempt but for troubleshooting purposes. As outlined yesterday, NASA will use the tanking to gather additional data, which hopefully provides more insight into the root cause of that problem. Let’s hope that NASA manages to get that highly in demand data.

7:24am: NASA TV officially announces the launch scrub.

7:39am: The NASA homepage officially states that space shuttle Atlantis’ Sunday launch has been scrubbed.

7:55am: Commentator announces that a short news briefing will be held within the next ten minutes or so. Meanwhile, the launch attempt has been converted into a tanking test. NASA is hopeful to retrieve some data pointing to the root cause of the ECO sensor problems. It was also noted that the failure scenario this time was different from what has been seen at the last launch attempt on Thursday.

8:00am: mission management team meeting set for 9:00am. Liquid oxygen tank is being drained.

I just picked up this picture from NASA TV. It shows members of the mission management team discussing after space shuttle Atlantis second launch attempt had been scrubbed.

Members of the mission management team are discussing in launch control center after space shuttle Atlantis second launch attempt had been scrubbed.
8:14am: Press briefing begins. NASA launch director Doug Lyons is interviewed by public relations officer George Diller.

Mr. Lyons explained: “All the voltages had good readings as well. We were very excited. We thought we had a good system and ready to fly today. We continued monitoring and then we saw sensor number 3 go dry to wet, which was a failure.” He added that based on the launch commit criteria set yesterday, that meant the launch had to be scrubbed.

As already said, today is now devoted to troubleshooting. Mr. Lyons: “We do have a troubleshooting plan in place. We stopped the flow on the liquid hydrogen (LH2) system and put it into a stable posture configuration. And we drain the liquid oxygen (LOX) tank, than we focus on LH2, we drain down to 5% and stop there and then monitor the system for four hours and see how these systems behave. Then we drain and secure the pad.” I assume that this is done in order to see how thermal changes may affect sensors and their connections to the orbiter.

Mr. Lyons noted that the failure was not much different from Thursday’s failure: “The only difference is sensor 3 and 4 failed Thursday, and today just sensor number 3. It failed in the same time frame and the same manner.” It should be said, however, that every time before there was trouble with the ECO sensors, that trouble “magically disappeared” (to quote Wanye Hale) on second tanking. That was the rational for attempting a launch today. So something is different to previous experience.

Asked on how to proceed now, Mr. Lyons declined to comment: “We have a 9am mission management team meeting and discuss our options. It would be speculation at this time to try to make a guess on which direction we head. We have multiple options. We will put something together and then implement it after that meeting.

After the interview, NASA TV ended its coverage of today’s launch attempt at 8:21 am. ET.

Press conference is whenever the mission management team meeting concludes. My personal guess is this will be in the late afternoon/evening time frame.

I, too, will now conclude coverage of the launch attempt on this blog page. I’ll now stick to other things and wait for the press conference. Should exciting news happen, I hope to pick it up. If so, I’ll create a new posting on my blog. Thanks everyone for reading.

And a shameless self-promo plug: if you liked this article, share it and send a link to it to your friends ;)