eco Archives - Rainer Gerhards

2008-01-05

Atlantis to launch on January, 24th?

The NASA space shuttle home page currently states that Atlantis could possibly launch on January, 24th. However, there are serious doubts about that date. From what I have found on the net, early February sounds much more realistic – with a launch on February, 2nd if there will be no further tanking test conducted. The most likely scenario, however, seems to be a launch no early then February, 8th.

Unfortunately, I am currently very busy with one of my projects and thus can not report more in-depth. That will follow hopefully soon. In the mean time, let me quote the NASA shuttle home page:

NASA flight control teams and ground operations teams have been requested to protect for a Jan. 24th launch date for Space Shuttle Atlantis. As work progresses, that date will be modified as required, says John Shannon, deputy manager for the Space Shuttle Program. The schedule depends on test results and modifications to a fuel sensor system connector on the external fuel tank Atlantis will use for launch on its STS-122 mission to the International Space Station. Other launch opportunities could come between Jan. 24th and the first week of February.

The connector suspected of prompting false readings during two previous launch attempts is undergoing intensive testing at NASA’s Marshall Space Flight Center in Huntsville, Ala. Engineers also will test potential modifications to the connector to certify it for flight. Marshall has a test facility that allows the connector to be subjected to the same conditions it saw during the earlier launch attempts.

The modification and testing plans were discussed along with the launch preparation schedule during a meeting of Space Shuttle Program managers Thursday.

Technicians at NASA’s Kennedy Space Center, Fla., will modify a replacement connector for the one that was removed. Metal pins inside the connector will be soldered to the socket, Shannon explained. The new connector is scheduled to be in place by Jan. 10.

“We’re fairly confident that if the problem is where we think it is, that this will solve that,” Shannon said.

Atlantis remains at the launch pad as the agency studies ways to modify the connector. The shuttle will carry the European Space Agency’s Columbus laboratory to the space station during the STS-122 mission.

2008-01-01

Shuttle Feedthrough Connector Removal Pictures

As a new year’s gift, NASA has place twelve interesting pictures from the December, 29th removal of the feedthrough connector in the media gallery. The original format is quite a bit hard to read (at least in my opinion), so I thought I recompile them in this post.

The feedthrough connector was removed to be shipped to NASA’s Marshall Space Flight Center for further cryogenic testing. This is part of the ongoing space shuttle ECO sensor troubleshooting. If you doubt why further troubleshooting is needed, you may want to have a look at my “xmas decoration and space shuttle similarities” post ;)

Very interesting to see the technicians at work.

First, the external connector cable is cut:

Then, a pair of support brackets is removed:

Before disconnecting the connector assembly, it receives a cleaning, removing any residual foam insulation:

Then, the connector assembly, with its associated electrical harness, is pulled away from the tank:

Technicians set up equipment that will be used to take X-rays of the connector cable:

Then, the connector is disconnected before it is demated from the external tank:

And finally the demate occurs:

The technician then inspects the connector just removed from the external tank:

Technicians wrap the connector for transport to NASA’s Marshall Space Flight Center in Huntsville, Ala., for further cryogenic testing:

… and place the wrapped connector in a shipping container:

… which is then finally carried away for transport to the Marshall Space Flight Center:

Nice work, guys! And now I am eager to hear about the testing results in MSFC! Stay tuned…

Image Credit for all pictures: NASA

2007-12-30

Atlantis troubleshooting continues…

The analysis of space shuttle Atlantis ECO-Sensor trouble continues. Everybody focuses now on the Feedthrough Connector.
I have been involved in some (heated ;)) forum discussion on why NASA takes the time to analyse the issue and not provide a quick fix now that the culprit is know (remember, the Feedthrough Connector has been identified as the trouble spot via the TDR data tanken on the last tanking test).

A quick wrap-up on the connector: it is used to feed several signals through from the tank’s interior to the Orbiter systems. Among them are the ECO sensors as well as the 5% sensor signal. The connection essentially consists of three parts. The schematic can be found in the small picture above (click to enlarge). I found a more detailed sketch in the STS-114 Flight Readiness Review (FRR) document:
As you can see, the system consists of an external connector, the actual feedthrough that goes through the tank and an internal connector. As far as I know, the sole purpose of that system is to feed the internal signals through to the external stack parts while ensuring that there is no leak in the tank.

The external part of the connector has yesterday been unmounted and sent over the Marshall Space Flight Center for analysis. As far as I know, analysis results will be available on January, 3rd and will be the basis for the discussion on how to continue.

But back to my forum discussions. Why at all is an analysis been made?

There are a lot of technical words that could describe it. I will take a different route. I’d like to use a real-world analogy that most of us would probably be able to follow ;)

Let’s assume you own a house and it is xmas time. Chances are good you like to decorate your front yard with some nice lights. These lights need power and you need to draw that power from an outlet somewhere in, let’s say, your garage. Of course, the light’s power cord is to short, so you use an extension cord to connect the in-garage power outlet to the decoration’s plug in your yard. Everthing works perfectly and you are really proud of your fine lights.

But then, out of the sudden, a fuse blows and your lights go off. You begin to analyze the problem. One thing you notice is that it started to rain. But everything worked well a couple of times when there was rain, too. You blame the fuse, after all it was a pretty old one. So initially, you just go away with the issue, use a new fuse and be happy again.

After a while, on another rainy day, the fuse blows again. This time, you know it is a real problem. You do another analysis. In that course, you know that it must somehow be related to the extension power cord (let me just assume you somehow magically know it is ;)).

This is where we are with the ECO sensor system currently. The extension power cord is my analogy of the feedthrough connector. Testing done during the tanking test pointed at that connector, just as you now know it is the extension cord. However, test data did not say exactly what is wrong with the connector. In my sample, we also do not know what’s wrong with the extension cord.

Back to the sample: so what to do? If do not want to do a more in-depth analysis, we could simply replace the extension cord (just like we replaced the fuse) and hope that all is well. This might work, especially if we had little trouble in the past. It is also a quick fix, which is useful a few days before xmas (aka “time is running out).

If we take a bit more time, however, we might want to analyze what the root cause is. If we do, we may find out that the extension cord indeed was faulty. Maybe its water protection was damaged. Then, we’ll end up with swapping the cord, but this time with a very good feeling and confidence that our lights will stay on.

But analysis may also show less favorable results: maybe we find out that the cord is perfectly OK. But we made a “design error”. Maybe we find out that we used a non-outdoor rated cord in our “yard light system”. Replacing that cord with a like part would bring no improvement at all. In this case, we would need to do some more change – using an outdoor rated cord would be appropriate. Again, we could modify our power supply and have a very good feeling about its future reliability.

Unfortunately, space hardware is a bit more complex than xmas decoration. So analysis takes a bit longer. But it still offers the same benefits: if you look at the root cause AND be able to find it, you can reliably fix the system. Of course, there are limits and constraints. Too large delays bear other risk. It is NASA management’s task to weigh benefit and risk and do the right thing.

Oh, and one more note: I’ve heard so often “NASA just needs to do fix and everything will be easy. Just throw out that feedthrough connector…”.

Let me use my xmas decoration analogy once again. What that means is that you get a big digger and get rid of the extension cord at all – by creating a permanent electrical circuit with its own fully-outdoor-proven outlet right at the decoration. Of course, this is doable. Of course, it’ll fix the problem. It is just a “simple redesign” of your system. If think, however, that it is not the real smart answer to the problem you faced.

And I think many of all those quick fixes now being proposed “just let’s redesign the shuttle…” are along the same lines. If, of course, they were technically sound… ;)

I hope my sample helps clarify why there is analysis on the ECO sensor problem and why this is a good thing to have. Even though it may push Atlantis launch date a bit further down the calendar.

2007-12-27

Atlantis’ launch moves to February 2008

Space Shuttle Atlantis is likely to launch not earlier than February. The target launch date of January, 10th 2007 for Atlantis STS-122 mission has been pushed back to early February. This is the result of today’s mission management meeting.

The official NASA shuttle home page is even a bit more cautious:

NASA space shuttle managers met Thursday and decided to modify a fuel sensor system to correct false readings that postponed shuttle Atlantis’ planned launches on Dec. 6 and Dec. 9.

Testing and analysis indicate that false readings from the engine cutoff sensor system occur in a three part feed-through connector. The connector passes electric signals from sensors inside the external fuel tank to shuttle electronics outside the tank. Technicians will remove portions of the connector and redesign the interface by soldering the pins to sockets at the external-to-feed-through side of the connector prior to installing the replacement into the external tank.

It is unknown how long it will take to complete the modifications and reapply foam to the shuttle’s external tank. Managers will assess the progress of the work before determining a new target launch date for Atlantis.

The emphasis is mine. The repair is quite time consuming. I have currently no details, but it looks like there will be no full root cause analysis. That would probably have required a rollback to the VAB and a destack, and that process would make an early February launch date impossible. I hope to get more details soon and will post them when I have.

2007-12-27

STS-122 Launch Date Decision Today?

Today is the mission management team meeting for space shuttle Atlantis STS-122 mission. The meeting reviews work progress, data gathered and plans drawn and will finally conclude with a recommendation for the next steps. It is assumed that the launch date is targeted.

While January, 10th is still given as an option, news have condensed that this date will most probably not see a launch. It looks like a launch late in January or mid-February is more realistic. In any case, we’ll know much more when the meeting has concluded later today. A press briefing is expected shortly after the meeting. As far as I know, it was set for 10:30am ET, which means the meeting must have started right now.

Let’s see how the day evolves…

2007-12-18

Shuttle Tanking Test – good results, launch date affected?

Unfortunately, I was not able to view the full news briefing (hadn’t expected it to happen that soon). But I know the most important facts and would like to pass them on here. On the briefing appeared space shuttle program manager Wayne Hale.

Mr. Hale said that the problem is with the “feedthrough” connector. This is a connector on the external tank that connects the in-tank ECO sensors with the lines that go to the point sensor box inside the space shuttle. That connector fails when put under thermal stress, causing some circuits to become open. This is a very rough description, but I hope it helps you get the idea. Mr. Hale said: “We know it is in the connector, but not if it is in the inside or outside part“.

How today’s test results will affect the launch schedule is not yet clear – the results were somewhat unexpected (I am NOT quoting Hale here) and NASA is now checking which forward actions to be taken (this, however, is a Hale quote ;)). One of the exact quotes I was able to get hold of: “The program is being asked to assemble the ISS. We’d like to do that as quickly as we can in regard to safety. We don’t need to go fly if it is not safe. We will follow that trail and see where it leads us. And when we have fixed it we’ll go fly – no matter if it is Jan 10 or Feb 10th or whenever“.

To me, the bottom line is that the January 10th launch is under consideration. We will know for sure only in a few days when the data gathered has been analyzed and a proper plan has been crafted. After all, this is why such launch schedules are always “no earlier than”. And I think it is a good thing to put safety first!

With that, I conclude my report for today. I’ll try to do a sum-up tomorrow but guess I’ve already covered the most important things.

2007-12-18

Space Shuttle Atlantis Tanking Test in Progress

Today’s tanking test gathers additional data on ECO sensor (What is an ECO sensor?) failures with the ECO sensor circuits being fully instrumented. In order to gather data, an ECO sensor failure is needed. Before the start of the tanking test, it is anticipated that ECO sensor #3 will fail wet and all others will work correctly.

2:08a: The tanking test just went into fast fill mode. During the last launch attempt, ECO sensor #3 failed only a few minutes into fast fill mode.

8:14a: If I got the commentator right, there already is a failure of a sensor. That would be good.

8:21a: NASA TV commentator: “It appears that #1 sensor has failed, but the hydrogen console has not officially made that call. So they are still looking at it. Number 2 sensor has been intermittent. The engineers at the main propulsion system console are very intently looking at their data right now. The intermittent data on sensor #2 is somewhat of a surprise because we haven’t seen that before on that sensor.”

So as it looks, the ECO sensor system behaves again different from the last tanking, which was at the December, 9th launch attempt.

8:23a: NASA TV: “The propulsion console has provided confirmation that sensor #1 on liquid hydrogen has hard failed, which is what we wanted. The #2 and #3 are intermittent, we are watching for what those are going to do. We have not seen any activity of number 2 sensor on prior tankings.”

8:26a: NASA TV: “The hard fail on #1 is definitely what we need in order to do the troubleshooting at the pad later this morning.”

So far, this looks very good. Based on analysis done in the past days, sensor #1 is not expected to return to a non-failed state. So this one will hopefully a good candidate for the troubleshooting which will then be able to pinpoint the culprit. The intermittent failures of #2 and #3 are outside of expectations, at least as far as I understood the analysis done. However, if they now remain functional there still may be a good explanation for that. Let’s see how things evolve…

9:32a NASA commentary just announced that engineers have made up their troubleshooting plan. “The TDR sensor equipment will be focussed on sensors #2 and #3 as they have been intermittent and it is the greatest source of interest to see where that intermittent reading is coming from. Number 1 and 4 will be recorded continuously. … Right now they are configuring for the console activity that would be required once we go into stable replenish … very shortly. … So the final inspection team is preparing to enter into the pad, but we probably have another half an hour or so before we are into stable replenish.”

9:48a NASA TV: “The liquid hydrogen tank is full now and in stable replenish. The liquid oxygen (LOX) is at 80%. We’ve probably about another 25 to 30 minutes or so before liquid oxygen is in stable replenish.

10:00a: NASA TV: “We will begin to drain the tank between 1p and 2p and once the tank is drained the teams go back to do additional trouble shooting. A lot of data has already been collected on sensor #1 because it failed early.”

10:13a: “The problem with the liquid ox pump appears that a fuse has blown, so they are setting to the backup lox pump, … which will us back to where we were close to the stable replenish. This will take around 45 minutes to an hour. Why the fuse blow is still not clear”. So we have some delay in the troubleshooting plan.

11:45a: I needed to go out for a while, thus no updates. Atlantis is now in stable replenish state and troubleshooting is right now happening at the pad. The NASA TV commentary just announced that teams are now looking at ECO sensor #4 and the 5% sensor. Here are some pictures from the testing:

The picture above shows a room that is right inside the mobile launcher platform. This is where data today is analyzed.

11:52a: NASA TV “The … team is now on the mobile launcher platform and gathering data on sensor number 2 and 3.”

11:53a: a side-note: the ISS spacewalk just has completed, the spacewalkers are back into the airlock, which is now being repressurized. They did not find any obvious problem, but collected samples to be returned to earth in January on board STS-122.

12:06p: NASA TV: “Data will be collected remotely during de-tanking.” The equipment is currently being set up for this process. People are permitted close to the space shuttle only during stable replenishment phase (which essentially means no tank or detank operation is taking place). During detanking, they must leave for safety reasons. The set up process can be seen in this picture:

6:12p: and now those members at the controls can be seen leaving the control room. Everything is now set up for remote monitoring. The NASA TV commentary tells that good data has been gathered. The tank will be drained to 5% while instrumentation is on. Once it is fully drained, members of the red crew will return for further troubleshooting.

And on this picture, a wire can be seen. This wire is tapped into the ECO sensor circuits and connects to the time domain reflectometer (TDR) equipment that is used to do a detail analysis of the circuits. TDR is an off-the-shelf technology in wide spread use e.g. by cable companies to detect faulty lines.

around 12:25p: I was too slow for a quote, but NASA TV commentary stated that everyone seems to be quite happy with the data gathered. It sounded like a successful test. The commentator also announced that more – and official – information is available in the news briefing later today. For me (being six hours ahead), it probably is too late, but I’ll then do a wrap-up tomorrow morning. Let’s hope that they have actually managed to find the culprit and nail it down – and also let’s hope that the rest of the day (detanking test) also goes well.

12:40p: it looks like the red crew is now leaving the pad after finishing late inspections. I noticed they used binoculars and telescopes during that process. I (now) think they also visually checked for ice buildup and maybe some other things. Probably just part of the usual procedure. Here, the red crew can be seen leaving the mobile launcher platform (the last frame I was able to capture, sorry for the blur…).

The NASA TV commentator announced a few minutes before that engineers are ready for detanking once the red crew leaves. So I assume detanking will begin shortly.

12:48p: NASA TV: “The final inspection team has now left the pad … In the mean time we had a failure of ECO sensor #3, the second sensor that had failed. It was intermittent … and we are collecting data on sensor #3 … right now which is able to be done remotely much as we got the initial data from sensor #1 when it failed earlier today.”

I didn’t get the full quote, but as far as I understood, sensor #3 has now also a hard failure. That, IMHO, would be somewhat bad news because it means that the condition will change once the tank is loaded. However, this is just my uninformed guess. Also, on a actual launch, the shuttle would be long gone and fly in orbit. So that may not mean anything at all. I should stop speculating ;)

12:52pm: NASA TV: “We are close now to start the external tank draining operation. and we did indeed get the kind of data that we needed to get to the (bottom of?) that trouble.”

12:58p: NASA TV: “We did see sensor #3 also fail, so we try to get some data from it before we start draining.” So it actually has a hard fail and NASA uses the opportunity to get something out of that. To me this sounds like it failed just a few minutes ago, otherwise the red team would have looked at it (wouldn’t they?).

01:00p: NASA TV: “on the recently failed ECO sensor #3” – and indeed, it recently failed. Right in time, one could say…

3:00p: as I wrote, I have been away from my computer for a while. NASA TV has ended coverage of the tanking test in the mean time, so I can not provide any more status updates. I’ll try to do another post after the news briefing, but that will probably be tomorrow (I hope I’ll be able to get hold of an archived version of the news briefing). All in all, it looks like an excellent day at Kennedy Space Center, with good results that will bring up the root cause of the ECO sensor issue. Let’s hope that I have the right reception of today’s events.

Thanks to everyone who cared reading my post! I hope it has been useful and will be as a reference.

I’ve now also written down some results from the post-test news briefing.

2007-12-15

STS-122: Atlantis being instrumented…

Troubleshooting efforts at Kennedy Space Center (KSC) continues: Space shuttle Atlantis has been instrumented for a tanking test scheduled to begin Monday. In parallel to the work at KSC, engineers at other NASA centers gather data about how the elements of the ECO sensor system should respond during the tanking test. This provides a baseline against which the actual tanking test results can be compared. Unfortunately, it is not sure if the failure will re-occur during the tanking test. But according to the latest findings it is quite likely.

Please also let me quote the relevant part of NASA’s shuttle home page:

Engineers and technicians at NASA’s Kennedy Space Center continue preparations to evaluate the hydrogen fuel sensor system on space shuttle Atlantis’ external fuel tank during a procedure next week. Working at Launch Pad 39A where Atlantis remains pointed to space, workers attached wiring to the cables that lead from the aft compartment of Atlantis to the external tank’s engine cutoff sensor system. Engineers will use the special instruments next Tuesday to send electrical pulses into the wiring and look for indications that will show the location of the issue that caused the sensors to return false readings last week. The failed readings showed up during launch countdowns on Dec. 6 and Dec. 9. Launch controllers postponed the liftoff on both occasions to find out the problem and develop solutions. Evaluations of the instruments themselves are also under way to show technicians what a normal reading on the external tank looks like. Those readings will be compared to the results from the test Tuesday during which the tank attached to Atlantis will be filled with super-cold liquid hydrogen. NASA is targeting Jan. 10 as the next possible launch opportunity for Atlantis on mission STS-122. Atlantis will carry the European-built Columbus laboratory to the International Space Station.

2007-12-11

Shuttle Troubleshooting continues…

Just a short note, a longer one will follow (maybe tomorrow, it’s late over here…). I have just listened to the news teleconference with Mr. Wayne Hale. I am very glad to say that he is determined to fix the ECO sensor issue and sees good chances in doing so. To do that, a tanking test will be conducted next week.

Probably the following quote describes the whole situation “The primary goal is to troubleshoot the system as it is and restore its functionality. We would only consider other measures if we fail with this.“

I hope they will succeed with that and we will have a great space shuttle launch in early January.

2007-12-11

Shuttle Manager Hale’s Teleconference Statements

These are the notes I have taken during the December, 11th teleconference. I am posting them as I have taken them. They are largely unedited, but IMHO speak for themselves. Direct quotes are within quotation marks, the rest are my observations”. I penned this down during the teleconference, so I am pretty sure it is the exact wording.

“We set up 2 investigation teams. One is a near term team working on the current vehicle, second is a longer term team with experts from all around the agency.”

“do instrumented tanking test next Tuesday. Some instrumentation that we can put on some appropriate places at the circuits … We can capture the location in circuit in TDR, a commercially available technology … we have a high degree of confidence in pinpointing the location and once we know the location we can put together a fix.”

“STS-122 launch could definitely be a bit later than January, 2nd”

“I’ve been committed to fixing it. An intermittent electrical problem is really hard to fix. We thought we had fixed it, but we didn’t. So I might say we are RE-committed to fixing it”

Reporter: How concerned are you about the whole eco sensor system (touching on the email where Mr. Hale considered retiring this system as unreliable).

“This low level cutoff capability is a safety system that has never been used in flight… like a seat belt. If you don’t have anything bad, you probably don’t need it. If you needed it and it didn’t work it could be really bad. We would like to have this system functional and we would like to restore functionality of this system”

“There are other test, bench test of the equipment at manufactures in parallel to tanking test. Tanking test is a hazardous operation.”

Reporter: if STS-122 does not launch on January, 2nd, when then? “I let you know when we have our tanking test done and we have some data”

“The problem only occurs when we have cryo conditions present” It all works well at normal temperaturs. We are gonna to find out where the problem is.”

If the LCC criteria (number of working ECO sensors) would be changed: “Our point is to try to fix the problem and then go back to the previously LCC, rather than speculate let us wait and try to fix this problem.”

“TDR is not flight equipment, its not qualified. Its a ground system only.”

Once again on the email about retiring the eco sensors: “Our thinking has evolved from Friday when I wrote that little note.”

“We hope it repeats one more time on our test next week when we have the instrumentation”.

“Until we come to the bottom of this mystery, we are in no better shape launching any other orbiter” when asked about swapping assembly flights or orbiters to get off the ground.

“a single circuit is around a hundered feet of wire from the PSB to the sensing element.”

Test details:
“We have to physically cut the wires, we are talking about the ECO and the 5% sensor, which is in the same pass-trough and connector. We have to have people present to run that equipment. We send the red crew out during the stable replenishment phase. And if the problem is as in the past, it will stay with us during the stable replenish phase. We can not have people during tanking for safety concerns”

On a Christmas break: “We are thinking about taking a few days off to allow our folks to have a few days with their families. We’ll make that decision shortly after the tanking test.”

“The splicing of wires in the aft compartment is a standard procedure. We have identified a place where we can access the circuits that are readily accessible.”

“And then restore the circuits together when we are done with the tanking test.”

“Take the sensor’s wire harness and the LH2 pass trough connector and put them in a facility with cryo fluids and monitor how these tings respond in a lab setting. either liquid helium, warmer, or liquid hydrogen (lh2). we look at all these piece parts work. we did piece part testing before, we are doing integrated circuit from the lh2 pass through to the sensors themselves.”

“The liquid helium low level cutoff ability (LLCO) was present from the beginning , I am pretty sure it was sts-1 (99% positive). It is not a new system that has changed dramatically in design or manufacturing. The FPR is not significantly different the past 10/15 years than it is today. Just enough reserves to protect it from normal engine and operating procedures. The voltage indication system has been on sts 118 and 120 only, which were flawless.”

“The primary goal is to trouble shoot the system as it is and restore its functionality. We would only consider other measures if we fail with this.“

Next tuesday full tanking test? “We need to fill the tank fully because as I said, it is our safety requirement to have people on the pad we need to be in stable replenish. It is reviewed which aux systems need to be operating. The launch team is finalizing the procedures.”

“I can not put my finger at anything that is especially difficult, its just normal operations in an unusual environment.”

ISS Stage EVA next week, Thursday afternoon, 2ET, possibly next Tuesday, SARJ