Rainer Gerhards

2007-12-11

A design problem…

Folks, I am facing a design problem – and it looks so simple that I am pulling out all my hair ;)

I am currently preparing the next steps in modular rsyslog redesign. I am not sure yet, there are a couple of candidates what to do first. One is to add a real expression capability, another one is to add threaded inputs (which would be quite useful). In support of these enhancements, a number of things need to be changed in the current code. Remember, we are still running on large parts of the original sysklogd code, which was never meant to do all these advanced things (plus, it is quite old and shows its age). A cleanup of the core, however, requires some knowledge of what shall be done with it in the future.

My trouble is about a small detail. A detail I thought that should be easy to solve by a little bit of web search or doing a forum post or two. But… not only did I find the relevant information, I did not even find an appropriate place to post. May be I am too dumb (well possible).

OK, enough said. Now what is the problem? I don’t know how to terminate a long-running “socket call” in a proper way under *nix. Remember, I have done most of my multithreading programming in the past ten years or so under Windows.

What I want to do: Rsyslog will support loadable input modules in the future. In essence, an input module is something that gets data from a data source (e.g. syslog/udp, syslog/tcp, kernel log, text file, whatever …), parses it and constructs a message object out of it and injects that message object into the processing queue. Each input module will run on its own thread. Fairly easy and well-understood. The problem happens when it comes to termination (or config reload). At that instant, I need to stop all of these input module threads in a graceful way. The problem is that they are probably still in a long-lasting read call. So how to handle this right?

Under Windows, I have the WSACancelBlockingCall() API. Whenever I call that method, all threads magically wake up and their read and write calls return an error state. Sweet. I know that I can use signal() under Linux to do much of the same. However, from what I read on the web I have the impression that this is not the right thing to do. First of all it seems to interfere with the pthreads library in a somewhat unexpected way and secondly there is only a very limited set of signals available … and none left for me?

The next approach would be to have each blocking call timeout after a relatively short period, e.g. 10 seconds. But that feels even worse to me. Performance wise, it looks bad. Design-wise it looks just plain ugly, much like a work-around. It looks like I needed to do something not knowing what the right thing is (which, as it turns out, is the right description at the time being ;)).

To make matters worse, I have a similar problem not only with the read and write calls but with other constructs as well. For example, I’d like to have a couple (well, one to get started) of background threads that handle periodic activity (mark messages immediately come to my mind). Again, I would need a way to awake them when it comes to termination time – immediately.

And, of course, I would prefer to have one mechanism to awake any sleeping thread. Granted, can’t do that under Windows either, so I may need to use different constructs, here, too.

This is the current state of affairs. There is still enough work to do before the question MUST be answered in order to proceed. But that point in time approaches quickly. I would deeply appreciate any help on this issue. Be it either advise on how to actually design that part of the code – or be it advised where to ask for a solution! Really – a big problem is that I did not find an appropriate place to ask. Either the forum is not deeply technical enough, or there are some mailing lists where the topic is on something really different. If you know where to ask- please tell me!

[update] In the mean time, I have found a place to ask. Blieve it or not, I had forgotten to check for a dedicated newsgroup. And, of course, there is ;) The discussion there is quite fruitful.

2007-12-10

Space Shuttle ECO Sonsors: an in-depth View

After the scrub of space shuttle Atlantis December 2007 launch window, everyone is interested in the ECO sensors. That shuttle component is responsible for the scrub. Unfortunately, detailed information about it is hard to find.

However, I was able to obtain some good information. Most helpful was NASA’s “STS-114 Engine Cut-off Sensor Anomaly Technical Consultation Report“. I also used other NASA sources for my writeup, including information conveyed at the post-scrub press conferences.

Let’s start with some interesting fact that space shuttle program manager Wayne Hale provided in a press conference. According to him, the ECO sensors are an Apollo heritage. Their design dates back to the 1960s. Consequently, they are analog “computer systems”, which look quite strange compared to today’s technology.

I could not find any indication of sensor malfunction prior to STS-114, the “return to flight” mission. However, I have been told that pre-STS-114 flights did not have the same rigor checks in the flight procedure as they exist today. So it may very well be that there always were problems with the sensors, but these were “just” never detected.

It should also be noted that there was never a space shuttle main engine cutoff due to an ECO sensor (I need to correct this a bit later – but let’s keep it this way for the time being). It is believed, however, that on some flights the cutoff happened just a second or so before the ECO sensors would have triggered one. The amount of fuel left in the tank can not be analyzed post-flight, as the external tank is the only non-reusable component of the shuttle stack and lost after being separated from the orbiter.

But now let’s dig down into some hard technical facts: A good starting point are the graphics that NASA posted on the space shuttle home page. I’ll reproduce them here, but due to the blog theme, they are a bit small. Click on each image for a high-res version. It will open up in a new window, so that you can read along.

There is a drawing that puts together all the pieces. It is an excellent starting point:

A brief word of caution, though: the picture titles “LH2 ECO Sensor Locations” for a good reason. It is about the liquid hydrogen (LH2) tank sensors. There are also others, as we will see below. Let’s for the time being stick with the LH2 one. As far as I know, the LH2 sensors were also the only trouble source in recent shuttle launch attempts.

This is also where I need to correct myself. There actually have been main engine cutoffs due to ECO sensors, but none of them happened due to the liquid hydrogen sensors. As far as I know, there were three missions where it happened and among them were STS-51F and STS-93.

The image shows that the ECO sensors are located right at the bottom of the tank – which makes an awful lot of sense, as they should indicate depletion. There are four of them mounted in a single row on the shock mount. Each of them has their housing containing the actual sensing element. Even though this is not show on the above overview, let me add that there is are a lot of additional components that make up the so-called “ECO sensor”. That can be nicely seen in this schematic:

First of all, you’ll probably notice that it is more appropriate to speak of a “sensor system” than just of a “sensor”. If we talk about sensors, most of us simply think about the actual sensing element, seen to the right here. Obviously, that takes us far too short. You must think about the whole system to understand the problem. So think sensor element, electronics and electrical connections. All of this makes up what we call the “ECO Sensor”. In my personal opinion, there is a lot of misleading information and discussions on the public Internet these days. Part of this misunderstanding IMHO seems to stem back to the “sensor” vs. “sensor system” issue. Many folks express that they don’t understand why “such a simple sensor issue” can not be fixed. I guess that was even the motivation to write this post, but, hey, I am becoming off.-topic. On with the technical facts.

Next, you’ll notice that the ECO sensors are just few of the many sensors that make up the tank level information (the “point sensors”). All of these sensors are the same. The ECOs are in no way special, except for their name. ECO stems from “Engine Cut Off” and is attributed to the fact that these sensors are a emergency line of defense to shut down the engines if other things have already gone wrong (if all goes right, the ECOs are never used, but it is the ECOs that ultimately determine the fact that something went wrong…).

If you count, you’ll find twelve sensors: the four ECO sensors, one 5%, two 98%, one 100% minus, two 100%, one 100% plus and one overfill point sensor. Note that there are sensors both in the liquid hydrogen (LH2) and liquid oxygen (LOX) tank. Each of them has twelve, so there is a total of 24.

A notable difference is the location of the ECO sensors: for LH2, they are at the bottom of the external thank, while for LOX they are in the feedline inside the orbiter. In plain words that means that the LOX ECO sensors report very late while the LH2 sensors report early in the process of tank draining. This can be attributed to the fact that a fuel(LH2)-rich engine shutdown is required. I also assume that the risk of fuel pump overspeed and explosion is by far higher for the LH2 part of the system (but that just my guess, found no hard fact backing it).

The number of sensors at each position tell you something about their importance: it for sure is no accident that most positions are covered by one sensor, the 98% and 100% locations have two and the depletion location has four! Obviously, depletion is a major concern.

Which brings us to the point: why four? Let’s spell it out if it is not clear yet: it’s “just” for redundancy and backup. If there would be just one sensor, a single-sensor failure could be fatal. If it failed dry, it would cause an unnecessary (and comparatively risky) launch abort, if it failed wet and something else goes wrong, it could lead to vehicle destruction. Either way is not really desired, though obviously one case is better than the other.

To mitigate that risk, there are four sensors. But how put these to use? A simplistic approach could be that a poll is taken and the majority wins. So if we have one sensor telling dry and three telling wet, we would go to wet. Obviously, there would be a problem with a 2 dry/2 wet state. So our simplistic model is too simplistic. But I hope it conveyed the idea. What the system really does is a bit different:

First of all, there is a construct called “arming mass”. Keep in mind that the ECO sensors themselves are “just” a backup system to handle the case when other things have gone wrong before. Space shuttle computers continuously monitor engine performance and calculate fuel used. So there is a rough idea of how much fuel is left in the tank at any given moment. However, these calculations may not be 100% perfect and may not detect some malfunction, thus it is risky to rely on them alone. To mitigate that risk, the ECO sensor system has been put in place.

Now let’s take an extreme example. Let’s say an ECO sensor switches to dry just one second after launch. Would you trust it and assume the tank is already drained? I hope not. There are some points in flight where both logic and physics tell us the the tank can not be depleted. In fact, during most of the ascent it can not. But when we come close to main engine cutoff, then fuel may actually be used up. Only at that stage it is useful to look at the ECO sensors. This is what “arming mass” is all about. The shuttle’s computers continuously compute estimated fuel left and only when the estimate comes within the last 8 to 12 seconds of fuel depletion, the ECO sensors are armed.

This has some bonus, too. If an ECO sensor indicates “dry” before we reach arming mass, we can assume the sensor has failed. So that sensor will no longer be able to cast its vote when it later comes to aborting the launch. Please note, however, that it is not possible to detect a “failed wet” sensor in the same way. Sensors are expected to be “wet” during ascent and doing so obviously does not disqualify a sensor.

Once the arming mass has reached, shuttle computers look at those sensors with a healthy status. If a single sensor indicates “dry”, computers initially assume a sensor failure. Remember: all sensors are mounted at the same location (see picture to the right), so they theoretically should indicated “dry” all at the same instant. However, that sensor is not disqualified. When now any second of the healthy sensor joins the other one in reading “dry”, shuttle computers assume an actual tank depletion.

They do not wait for the remaining qualified sensors, in a case now assuming these have failed “wet”. So whenever two qualified ECO sensors indicate “dry” after the space shuttle has reached “arming mass”, an abort will most probably be initiated. That means the space shuttle main engines will be cut off in a controlled and non-destructive way (which means a fuel-rich shutdown). Depending on when and how exactly this happens, it may lead to either an abort to the transatlantic landing (TAL) sites or an abort to orbit (ATO). I guess it may even be possible to reach the desired orbit with the help of the orbital maneuvering system if the engine cutoff happens very soon before its originally scheduled time.

Please let me add that the actual procedure for tank depletion must be even more complicated than briefly outlined here. For example, what happens if three of the ECO sensors disqualify themselves by indicating “dry” early in the ascent? Will the remaining single sensor than decide about launch abort? Also, what happens if all four fail early? I don’t like to speculate here, if somebody has the answer, please provide it ;) In any case, you hopefully have gotten some understanding now that the ECO sensor system and putting it to use is not as simple as these days it is often written on the web…

Now let’s look a little bit about where the sensors are located. If you paid attention to the above drawing, you have noticed the black lines which separate parts in the tank from parts in the orbiter (and yet from those at mission control center on the ground).

The best picture of the actual ECO sensor housing I could find is this one:

Please note that it shows the ECO sensors during a test, in a test configuration. The mount is different from the actual one in the external tank.

The computers driving the sensors are located in the orbiter’s avionics bay:

This, and the following, drawings mention the “point sensor box”, PSB for short. Remember that the sensors together are the “point sensors” and the ECO sensors are just point sensors with a special name and function. NASA also lets us know where exactly the point sensor box is located in the shuttle’s aft:

And finally, we have some more information on the point sensor box itself:

The point sensor box interprets sensor readings. The sensor elements provide a voltage. Certain low voltage level means “dry” while certain high voltage levels are interpreted as “wet”. However, somewhat above the “wet” levels, they indicated “dry” again. This level is reached when there is an open circuit.

NASA also provided an the exploded view of the point sensor box:

To me, it just looks like a box for electronics and I do not get any further insight from looking at the drawing. But anyways – it’s nice to know…

I could not find pictures of the not-yet-mentioned sensor system parts: the connectors and cables. Somehow the in-tank sensors and the on-board point sensor box must be connected to each other. This is done via some cables and connectors. Those must also be looked at when thinking about the system as whole. Especially as the failure reading we see points to an open circuit. I have read that some of the cable are below external tank foam. So its not easy to get to them.

I have heard that cryogenic temperatures are probably part of the trouble. Because failure readings seem to happen only when the tank ins filled (and thus very cold). One could assume that shrinking of ultra-cold material is part of the problem, but again, I have not found any credible references for this – or any other thermal results.

So it is now probably time to going right to the source. Below, find reproduced the deep technical description from the STS-114 paper quoted right at the start of this posting (quoted text in italics):

The MPS ECO system consists of point-sensors installed in the ET liquid hydrogen (LH2) tank and the Orbiter’s liquid oxygen (LO2) feedline. Point sensor electronics are designed to condition signals and to provide appropriate stimulation of the sensors and associated wiring and connectors.

The point sensor electronics interprets a low resistance at a sensor as the presence of cryogenic liquid, which provides a “wet” indication to the Multiplexer/De-Multiplexer (MDM) for use by on-board General Purpose Computers (GPCs) and the ground Launch Processing System (LPS). Conversely, a high resistance is interpreted as a “dry” indication. The point sensor electronics include circuitry suitable for pre-flight verification of circuit function and are designed to fail “wet”. For example, an open circuit in the sensor, or an open or short in the signal path, will provide a “wet” indication to the MDM. The system is then activated and checked out during launch countdown and remains active during ascent.

An ECO sensor is depicted in the next Figure. The sensor consists of a platinum wire sensing element mounted on an alumina Printed Wiring Board (PWB) and is encased in an aluminum housing. The sensing element acts as a variable resistance which changes on exposure to cryogenic liquid. This resistance variation is detected by post-sensor (signal conditioning) electronics and is used to generate either a “wet” or “dry” indication as noted above.

The ECO system is designed to protect the Space Shuttle Main Engines (SSMEs) from catastrophic failure due to propellant depletion. Flight software is coded to check for the presence of “wet” indications from the sensors within 8 to 12 seconds of SSME shutdown. The software rejects the first “dry” indication observed from any of the ECO sensors, but the presence of at least two more “dry” indications will result in a command to shutdown the SSMEs (i.e., “dry” indications from two of four “good” sensors are required for SSME shutdown). Early SSME shutdown would probably lead to a contingency Trans-Atlantic (TAL) abort. A failed “wet” indication cannot be detected. The system is designed so that LO2 depletion should occur first. However, a failure “wet” indication of three of the four LH2 sensors, coupled with an SSME problem that results in early LH2 depletion, could result in catastrophic failure of a SSME. Failure probability is considered remote, but would almost certainly be catastrophic to the flight vehicle. The system architecture addresses redundancy with one point sensor box containing four groups of sensor conditioner circuit cards. Each card can accommodate one hydrogen and one oxygen sensor. Each card group has its own power converter and one sensor conditioner card from each group services a pair of ECO sensors (again, one hydrogen and one oxygen). Wiring for each of the eight ECO sensors is split into one of two groups of sensors which are routed through separate Orbiter / ET monoball connections.

Let’s wrap-up: I hope you got a more in-depth view of the ECO sensor system by reading this post. At least, I think I have so by doing the research and writing it. Remember that I am no expert in this area, so I may be wrong. If you spot something that needs to be corrected, just drop me a note, for example in the form of a comment.

In regard to recent (STS-122…) developments, the question now is: what should be done if the root cause of the ECO sensor system failure can not be found. I don’t know, I miss too many facts. and my understanding is limited. But my guess is that if there can be rationale found to fly without it, that’s probably the best option to carry out. But hopefully tanking tests will show where it is flawed and a solution can be applied. Either way, I trust those wizards at NASA (and its contractors, of course). They have the have the training, they have the insight and they have the excellence. What else could one ask for?

2007-12-10

Astronauts have left Kennedy Space Center

The astronauts have left Kennedy Space Center, but not without a big thank you to the launch support guys:

“We want to thank everyone who worked so hard to get us into space this launch window,” the astronauts said in a statement. “We had support teams working around the clock at KSC, JSC, and numerous sites in Europe. We were ready to fly, but understand that these types of technical challenges are part of the space program. We hope everyone gets some well-deserved rest, and we will be back to try again when the vehicle is ready to fly.”

They are now back to Houston, where they will continue their practices in support for a space shuttle Atlantis launch in January 2008. The launch is scheduled to be no earlier than January, 2nd. The date is obviously affected by the result of the ECO sensor troubleshooting that is currently being conducted. First news on that troubleshooting effort is expected on Tuesday.

2007-12-09

No Space Shuttle Launch in December 2007…

NASA has waived off any further space shuttle launch attempts for the December launch window. NASA’s shuttle home page has a quick note about that:

The launch of space shuttle Atlantis has been rescheduled for no earlier than Jan. 2, 2008. The postponement will give engineers time to evaluate false readings from the engine cutoff sensor system that measures liquid hydrogen in the external tank.

As far as I know, a January, 2nd launch will be around 5:45am ET.

It is actually no surprise to me, given the new sensor problems. There is not much more news available as of now, I will keep you posted as I get updates.

2007-12-09

Shuttle launch day? Unfortunately not: scrub!

Space Shuttle Atlantis LAUNCH HAS officially BEEN SCRUBBED. This post contains a full log of the order of events from tanking begin at 5:55am up until conclusion of the first post-scrub press briefing at around 8:30am.

Today should have seen the second launch attempt for space shuttle Atlantis’ STS-122 mission to the international space station ISS. Atlantis should have deliver the European Columbus lab module to the orbiting complex. Read why this now doesn’t happen…

Tanking has begun at 5:55a ET and so far everything is proceeding nominally. At around 6:40a a first status of the ECO sensors, responsible for a three-day launch scrub, is expected. All for sensors must work perfectly today, otherwise the launch will be scrubbed. If all goes well, Atlantis will lift-off at 3:21pm ET, within a very short one-minute launch window. Weather looks favorable, with just a 20% chance of weather prohibiting the launch.

Liquid hydrogen sensor number 3 has failed!

At 6:25a, guys in the control center look relaxed. Let’s hope it remains that way…

All four ECO sensors now indicate “wet”. This is good, but not yet a relief. The problem that caused launch scrub on Thursday did only show up after a series of test commands were sent to the sensors. As of my information, we are still about half an hour to an hour away from these checks.

6:47a: tanking has changed to “fast fill” mode. Last time, the ECO sensor problem occured 16 minutes into fast fill. According to the NASA TV commentator, we should get results of the sensor test in about half an hour.

6:52am: Liquid hydrogen sensor 3 has failed! A minute before that, the NASA TV commentator announced that all four sensors had passed the check, but then, he sadly had to announce that ECO sensor number three failed after a few seconds. Based on the information provided in yesterday’s press briefing, a launch scrub is highly probable.

7:00am: NASA will tank for another half hour. The team is now doing troubleshooting. No launch scrub yet!

7:02am: NASA TV commentator: “the ground rules layed out that we have to have four sensors to proceed with launch. And we have had sensor number 3 fail. So, we are going to do some trouble shooting over the next half hour. At that point we would stop, asses whether we do any further testing at that point and then drain liquid oxygen. Liquid hydrogen will stay in filled configuration.” … “An official launch scrub has not yet been declared, but according to the plan, the rest of the morning is evolving into a tanking test.“

7:09am: NASA TV: “The MMT has asked the propulsion console to come up with a time line on how long it would take to drain the liquid oxygen and then drain liquid hydrogen to 5%”. “The mission management team will … shortly … talk about what our official status will be. Although we have not officially declared the scrub, the commit launch criteria does not permit to continue…“

7:13am: NASA TV: “We continue to fill the tank for another 15 minutes”. Me: Note that this is not in support for a launch attempt but for troubleshooting purposes. As outlined yesterday, NASA will use the tanking to gather additional data, which hopefully provides more insight into the root cause of that problem. Let’s hope that NASA manages to get that highly in demand data.

7:24am: NASA TV officially announces the launch scrub.

7:39am: The NASA homepage officially states that space shuttle Atlantis’ Sunday launch has been scrubbed.

7:55am: Commentator announces that a short news briefing will be held within the next ten minutes or so. Meanwhile, the launch attempt has been converted into a tanking test. NASA is hopeful to retrieve some data pointing to the root cause of the ECO sensor problems. It was also noted that the failure scenario this time was different from what has been seen at the last launch attempt on Thursday.

8:00am: mission management team meeting set for 9:00am. Liquid oxygen tank is being drained.

I just picked up this picture from NASA TV. It shows members of the mission management team discussing after space shuttle Atlantis second launch attempt had been scrubbed.

8:14am: Press briefing begins. NASA launch director Doug Lyons is interviewed by public relations officer George Diller.

Mr. Lyons explained: “All the voltages had good readings as well. We were very excited. We thought we had a good system and ready to fly today. We continued monitoring and then we saw sensor number 3 go dry to wet, which was a failure.” He added that based on the launch commit criteria set yesterday, that meant the launch had to be scrubbed.

As already said, today is now devoted to troubleshooting. Mr. Lyons: “We do have a troubleshooting plan in place. We stopped the flow on the liquid hydrogen (LH2) system and put it into a stable posture configuration. And we drain the liquid oxygen (LOX) tank, than we focus on LH2, we drain down to 5% and stop there and then monitor the system for four hours and see how these systems behave. Then we drain and secure the pad.” I assume that this is done in order to see how thermal changes may affect sensors and their connections to the orbiter.

Mr. Lyons noted that the failure was not much different from Thursday’s failure: “The only difference is sensor 3 and 4 failed Thursday, and today just sensor number 3. It failed in the same time frame and the same manner.” It should be said, however, that every time before there was trouble with the ECO sensors, that trouble “magically disappeared” (to quote Wanye Hale) on second tanking. That was the rational for attempting a launch today. So something is different to previous experience.

Asked on how to proceed now, Mr. Lyons declined to comment: “We have a 9am mission management team meeting and discuss our options. It would be speculation at this time to try to make a guess on which direction we head. We have multiple options. We will put something together and then implement it after that meeting.“

After the interview, NASA TV ended its coverage of today’s launch attempt at 8:21 am. ET.

Press conference is whenever the mission management team meeting concludes. My personal guess is this will be in the late afternoon/evening time frame.

I, too, will now conclude coverage of the launch attempt on this blog page. I’ll now stick to other things and wait for the press conference. Should exciting news happen, I hope to pick it up. If so, I’ll create a new posting on my blog. Thanks everyone for reading.

And a shameless self-promo plug: if you liked this article, share it and send a link to it to your friends ;)

2007-12-09

Sunday Space Shuttle Launch Scrubbed!

Now it is official – today’s space shuttle Atlantis launch has been scrubbed due to a problem with ECO sensor number three.

Quote from the NASA home page:

Today’s launch of space shuttle Atlantis has been officially scrubbed. It was announced on NASA TV at 7:24 a.m. EST.

Of the four engine cutoff sensors, ECO sensor number three gave false readings. NASA’s current Launch Commit Criteria require that all four sensors function properly. The sensor system is one of several that protect the shuttle’s main engines by triggering their shut down if fuel runs unexpectedly low.

Atlantis’ scheduled launch on Thursday was delayed after two of the four engine cutoff, or ECO, sensors in the shuttle’s external fuel tank gave false readings. A third sensor failed after the tank was drained of fuel. The sensor system is one of several that protect the shuttle’s three main engines by triggering their shut down if fuel runs unexpectedly low.

2007-12-08

Atlantis “go” for Sunday launch

The meeting today went along the lines of the last meeting. Atlantis is now set to launch on Sunday at 3:21pm ET. NASA TV launch coverage begins at 6am ET. Tanking will start at 5:55am. Any sensor problems should manifest within one hour after tanking start, but of course it may also occur at any time later. The final test on the sensors is at T-9 minutes but they will be monitored down until T-31 seconds. Then, the automatic procedures are called in. At this point, a sensor failure will no longer cause a launch scrub.

The information in my last space shuttle Atlantis launch write-up is still valid. Except, of course, that NASA is now go for launch on Sunday. Weather conditions are favorable for launch, around 80% chance of launch. The emergency landing sites are also mostly “go”. A 24 hour delay bring slightly less favorite weather, with a “only” a 70% chance for launch on Monday. Interestingly, 70% were also mentioned for Tuesday (not previously considered a potential launch day).

A launch scrub on Sunday because of the ECO sensors does not outrule a launch attempt on Monday.

Mr. Hale strongly expressed the view that even if things go really wrong after liftoff, there are ample safe abort modes. This includes landing in Europe as well as aborting to a lower orbit. He re-iterated that a launch abort scenario is highly unlikely.

Launch attempts on Sunday and Monday will preserve the potential two-day mission extension to do an extra spacewalk. Later launch attempts may also preserve it, there is some planning underway.

The number of tankings and detankings, including tests, is limited because each tanking increases the risk of foam debris. This is because the ultra-cool propellants cause stress on the tank material, cause it to shrink and expand.

Future missions will see a number of changes, for examples in the engines. Mr. Hale states that he has tasked “considerable resources” to fix the ECO sensor system issue for future flights.

Finally, let me just quote the NASA home page:

Today’s Mission Management Team meeting has concluded. We are “go” for a Sunday launch attempt.

A post MMT news conference on NASA TV is set for approximately 4:30 p.m. EST. The participants will be Wayne Hale, Space Shuttle Program manager, LeRoy Cain, MMT chairman, Doug Lyons, STS-122 launch director, and U.S Air Force Captain Chris Lovett, 45th Weather Squadron.

So let’s hope we see this tomorrow afternoon:

2007-12-08

Update on Atlantis Launch Delay

Thanks to the “Space Multimedia” site, I have been able to watch yesterday’s press conference. Now the big picture clears up.

First of all, I was very wrong in my statement in my earlier STS-122 status article that NASA will launch without working ECO sensors – well, kind off…

In fact, it is quite the opposite. As NASA’s space shuttle program manager Wayne Hale stated the proposal is to require all four ECO sensors to work during launch preparations. Thus, the criteria has actually been tightened. Previously, only three good sensors were needed. Mr. Hale said that this plan came from the astronaut’s office. This goes well together with a posting I read in the Flickr STS-122 group created by Armando Perdomo. There, someone from an astronauts family posted (excerpt):

Anyway, what I heard that is interesting is that NASA offered the astronauts to go “as is” – with 2 sensors. They declined. My thought is that they didn’t decline out of fear of personal safety, but more of the consequences to NASA should something go wrong.

This not only proves what great people the astronauts are, it fits well within the press briefing picture. Mr. Hale also explained the reasoning for the new launch commit criteria: previous experience showed that failed ECO sensors always worked on the second launch attempt. Nobody knows why. Mr. Hale actually said we saw “that they magically work”.

Even if the cause is not know, one can assume that if the failure cause is the same as on previous missions, the sensors should behave the same. So seeing an additional failure on the next tanking would indicate that the failure cause is different. In this case, the launch will be scrubbed, as it would be an even less understood situation.

If the sensors work as expected during countdown, there is a good chance they will continue to function (based on previous experience). However, the problem was quite erratic and what was seen was different than before. This is the additional risk that NASA is accepting: the ECO sensor system may fail again during the launch. This is where new, still to be fully developed procedures kick in. They use new sensor instrumentation plus other systems to decide what to do in such a case. Here, some manual interaction may be required. NASA is actually reading itself to do the ascent without the ECO sensors.

So to sum it up: NASA requires 4 working sensors during countdown. If they don’t work, launch will be scrubbed. After liftoff, loss of the ECO sensors will be acceptable due to new procedures. However, it is hoped that they will continue to work if they did during countdown.

The launch has been moved to Sunday to support creation of the new procedures. The launch window has been shrunk to one minute to provide as much spare fuel as possible. Both of these measures are to address a failure scenario which will hopefully not occur due to the four-sensor launch scrub requirement.

It is important to note that this plan is not final. The mission management team meeting did not find concurrence from everyone. So folks were sent back to think about the plan, gather more data, run more simulations, get a better grip on everything – and meet again today at 1pm ET. Today, the final decision will be made. Depending on the findings, we may end up with a totally different set of options. The time of the post MMT press conference has not been announced yet. I guess they now refrain from posting any time because we’ve seen that any prediction is at least inaccurate ;)

Let’s assume NASA sticks with the plan. So what may happen? If there is a launch scrub on Sunday due to the ECO sensors, I guess it is mostly “game over” for the December launch window. If there is a scrub for weather reasons (more probable now due to the short one minute launch window), an additional launch attempt can be done on Monday. Chances of weather prohibiting launch are just around 30% for both days, so there is a fairly good chance that either of these attempts will succeed. If they don’t, the oxygen and hydrogen tanks for the fuel cell system must be topped off. That requires a three-day stand-down. Interestingly it was said that this means the next possible launch attempt would be on Thursday (is my math wrong?). It was mentioned that they could also launch on Friday. The launch time will be approximately 25 minutes earlier for each attempt.

With a Sunday or Monday launch, the two-day mission extension to do the extra ISS solar array rotary joint spacewalk is still an option. For Thursday and Friday this seems not to look well. They intentionally didn’t go into any specifics, but information I previously obtained tells me an extension would not be possible if launched late next week.

Finally, the commented that the overall shuttle launch schedule would not necessarily be affected if Atlantis’ launch would need to be deferred to early January 2008. If, however, the sensor issue would really need to be troubleshooted, the whole launch schedule would probably be affected. And it was strongly expressed that all of this post-December launch scenarios are highly hypothetical and should not be considered real for the time being. I concur to this – let’s solve today’s problems today and look at further issues when they come up.

Even though I am in no position to judge, the plan presented seems to be very sound. I hope they will not find any hidden problems and can attempt for launch on Sunday. Let’s keep our fingers crossed…

2007-12-08

Next shuttle launch attempt Sunday

The next shuttle launch attempt for Atlantis STS-122 mission is now set for Sunday at 3:21p. As it looks, NASA now actually attempts to fly without the ECO sensors. As I have written in my essay yesterday, that is not necessarily a bad thing to do.

To do this, new guidelines (procedures) for both the astronauts as well as mission control need to be developed. This is quite complex. It could not be done to support a Saturday launch. It is hoped that the additional one day of delay allows to finish those procedures.

On close look, the launch window has also been shrunk. It is now just a one-minute launch window. It obviously is done to preserve fuel. The primary risk of launching without the ECO sensors is fuel depletion. The best mitigation is to make sure to have enough propellants on board. Depending on when the shuttle is launched, there is more or less fuel needed to reach the desired orbital position. This is a matter of seconds. So NASA is now going for the optimal launch Window with the least fuel consumption. While this increases the risk of launch scrub, it reduces the risk of running out of fuel. It obviously is an excellent decision to go for the short launch window.

There is another mission management team meeting today. They will look at new data as well as the status of the new launch procedures. Depending on how things are, they will give a “go” for Sunday launch – or not. This is how to react to a situation like the one that is faced now: do the right thing to support the mission, but keep an eye on all options until the last minute.

And a word to those of you who need to leave Kennedy Space Center before launch (like fellow launch viewer Bill Rose). I know your disappointment. I know it too well. I went through all of this myself in summer of 2006, when I tried to view Atlantis STS-115 mission. Even though I had quite a lot of time to stay there, it was delayed so often (and even because of a Hurricane!) that I finally needed to leave. That was a sad experience. However, I am glad that they only launch when it is as safe as possible and I am sure you will agree with me.

What I recommend is to go to th Kennedy Space Center today and try to get on the up close tour. I am not sure if it runs due to the current pad activity. But if it does, it will probably provide a great experience, getting you as close to a real space shuttle as you never again will be in your life. It doesn’t matter if the RSS is still blocking some of the view, it will be spectacular in any case.

Now let me quote the NASA homepage to also get the official word over to you:

NASA is targeting the launch of space shuttle Atlantis no earlier than Sunday, Dec. 9, at 3:21 p.m. EST from the Kennedy Space Center, Fla. Shuttle program managers made the decision after a meeting Friday to review data on a problem with a fuel cutoff sensor system inside the shuttle and its external fuel tank.

Because of the length of the meeting, the managers agreed that targeting Sunday would allow the launch and management teams appropriate time to rest and prepare. The Mission Management Team will meet Saturday at 1 p.m. to decide whether to make a Sunday attempt. A news conference will be held after the meeting’s conclusion.

2007-12-07

To Launch or not to Launch?

The missing management team at NASA will have a very tough day today. As far as I know, a series of meetings have already begun. They are all about the ECO sensor issue – and how to proceed. It is hard to predict the outcome.

Let’s try to put some pieces together. Keep in mind that all of this is my personal guesswork. So tomorrow you may judge me based on what actually happens (hey, will I really do that…?).

From a technical point of view, it looks like the sensor issue can not be fixed quickly enough. The fact that there are intermittent problems in at least three (all four?) sensors makes it look like a problem with the electronics or cabling – not the sensors themselves. Fixing that would require at least three days at the pad – if it can be done there at all. Some think that a rollback is necessary. Troubleshooting the sensor system would best be done by a tanking test. However, that test has been called off by NASA in favor of consumables replenishment. To me, this is an indication that NASA has given up on fixing the ECO sensor system. In the press conference, they also mentioned that they are looking into rationale to fly as is. That also supports my argument.

But stop. Don’t say “better save than sorry”. Of course crew safety is first. But then remember that the ECO sensors are part of a backup system that kicks in when some other things already went wrong. They prevent the space shuttle main engines from running dry. So what? Do you think NASA puts not enough fuel into the tank? Obviously that isn’t the case. There is more than enough fuel in the tank for launch. So for the tank to run dry, something must be working quite wrong in the first place. Something like a leak or a similar serious issue. That’s an important point: the ECO sensors are an additional line of defense, but one that is never used in a normal flight. But, then, of course there is a reason for them to being there.

No let’s look at program constraints. If STS-22 can not make this launch window, it is not catastrophic, but will cause a wiggle in the schedule. Quite one, I think. That would cause already-much delayed international space station construction to be delayed even further. It will possibly also affect Constellation and Ares if the STS-125 hubble service mission must be moved. Pad 39B can only be handed over to Ares after the mission, because the STS-125 rescue launch-on-need mission needs to be at a pad to be able to launch it soon enough in case it is needed. So both pads are needed whenever STS-125 launches.

I guess there are also ISS constraints. We already know that construction is much delayed. Guess what? Space hardware has an expiry date. For example, the Columbus module that shall now be delivered by Atlantis is made to endure ten years in orbit. Of course, everyone hopes it will last longer. But its a simple fact: the longer hardware is in space, the less lifespan remains. Now think about all those years that other ISS components are already up there in space – waiting for the construction to finally complete. And every space ship is only as strong as its weakest part. May it be that any delay shrinks the time the ISS as whole can be productive in space (right now, its not really productive – much maintenance and construction going on and few science). Of course, a one month launch slip won’t hurt. But any larger delay will.

There is probably also one other risk with delaying the flight: if the ISS crew has to carry out vital tasks during STS-122 docked operations, their skills will fade. Astronauts are extremely well trained. But it is for a good reason that the practice until the last moment. Practicing at the ISS is rather limited. So the longer it takes, the less well-prepared the crew is. Of course, I do not know that really is an issue at this time. With STS-120, it was a very vital concern, because of all the complex staging spacewalks required by the ISS crew.

Think about it: it is not “just” the shuttle crew that must be kept save. There are also others (I have to admit that I too often overlooked that part of the picture).

As I said, its a tough decision…

Even if they launch as is, I do not think that the space shuttle’s crew life is more at risk than at any other launch (remember: spaceflight *is* a risky business). However, if something goes wrong and NASA needs to rely on the sensors, they will probably use ultra-conservative procedures. At least this is what I would expect. Thus, a launch abort would be much more likely.

And now think what happens if there is TAL (Transatlantic Abort Landing). First of all, it would be expensive. But even worse, how would the public react? Wouldn’t that be the last nail in NASA’s space shuttle program (and probably Constellation as well)? So there is a high risk in that, too. At least from an overall program perspective.

There are also political implications. Especially in this mission. Atlantis carries the European Columbus module into space. That cost roughly a billion (!) dollars. I guess the ESA would not be very amused if Columbus would be damaged due to some launch failure. Please note that I do not talk of a catastrophic failure, but of an abort, which can cause harm anyway. Given the fragile relationship between NASA and ESA, there is obviously some political thought that must be involved in the decision making.

I hope you by now have gotten an idea how many things and details need to be considered. I am sure I am still just scratching on the surface. The bottom line is that none of us outsiders will be in the position to judge it correctly. We should remind ourselves about that when we talk about the final mission management team decision.

Also, we do not have any solid data. The most important thing missing is probabilities. How probable is it that the ECO sensors are actually needed – I mean that the tank really runs dry? Is it 1 in 5,000? Or 1 in 100? The former one is probably a risk that can be accepted (the debris hit probability is much higher) while the later one obviously draws a different picture. Without solid data, you can not decide.

The whole situation looks much like the RCC panel issue we had with STS-120. However, at that time there was some more solid data at the time of decision making.

This is what the meetings are currently about: gathering data, looking at options and then deciding on what is the best thing to do after everything is put together. Such a decision, whatever it may be, easily upsets some folks (us, maybe?) who do not have the full reasoning at hand. I personally trust NASA guys to do the right thing. These are very bright people, doing a tremendous job in a very unforgiving business.

But would is my personal bet? I promised to do one… Tough, really tough. If I had to decide with just the information that is in this post… Well, I would probably fly as is on Saturday. Am I having launch fewer? Judge yourself. Here is my line of defense:

The ECO sensors are “just” a second line of defense. Something else must go seriously wrong in order to need them.
During launch, crew and mission control can manually monitor ECO sensor performance. If the show dry early into flight, this must be a failure. I admit that it gets harder the later we go into flight.
If an abort is needed, there are established procedures and no catastrophic outcome is to be expected. However, the shuttle program and NASA itself would probably pay a big price. That risk is accepted.
The launch and construction schedule can be maintained.
The risk for the ISS crew is minimized

If I look at my arguments, number one and two are the strongest one. If looking from a political point of view, you may also come to the decision to postpone to avoid the program risk. And, of course, depending on probabilities, you would like to avoid a lunch because of crew safety risks. But now I am spinning in circles ;)

I’ve done my bet, now let’s see what the real outcome is.