National Aeronautics and Space Administration

Academy of Program / Project & Engineering Leadership Space-to-Space Communications System (SSCS)

www.nasa.gov

casestudy

SPACE TO SPACE COMMUNICATIONS: IN-HOUSE HARDWARE DEVELOPMENT

When Johnson Space Center’s Matt Lemke showed up for work as the project manager of the Space-to-Space Communications System at the tail end of 1994, he discovered that the project he had inherited was not the one he expected. On his second day on the job, Lemke, an experienced avionic engineer but newly assigned project manager, attended a kickoff meeting with Litton Industries, who had recently been selected as the project’s prime contractor. Lemke had read the contract beforehand and understood the terms of the agreement. He planned to hand the Litton team a set of engineering drawings that had been developed before his arrival so they could get started. After handshakes and introductions, he turned to his NASA colleague who had supported the contract agreement with Litton just weeks before, and asked her for the drawings. Her response stunned Lemke: “What drawings?” A First-of-Its-Kind Radio System The Space-to-Space Communications System (SSCS) is a sophisticated two-way data communication system designed to provide voice and telemetry among three on-orbit systems: the Space Shuttle orbiter, the International Space Station (ISS), and the Extra Vehicular Activity Mobility Unit (EMU). (An EMU is a space suit worn by an astronaut during a space walk, or Extravehicular Activity.) SSCS is designed to allow simultaneous communication among up to five users. The system comprises three product lines: space suit radios (SSER), the Shuttle orbiter radio (SSOR), and the Space Station radio (SSSR). All three share some commonalities, but have unique features and different designs. (See Appendices 1.1 and 1.2.) This networked communications system was conceived as a “breadboard” (engineering prototype) concept in the avionic systems laboratories at Johnson Space Center (JSC) in the early 1990s with funding from the Space Station program. A few years later, the Shuttle program authorized SSCS into formal development to satisfy a critical need for interoperable space communications among the three vehicles in order to execute the construction of the ISS. Lockheed Martin, which provided technical support service to the JSC Engineering Directorate, did much of the early design work on the prototype system. (See Appendix 1.3.) An In-House Development After soliciting proposals from commercial contractors including Rockwell and Motorola, NASA decided to treat SSCS as an “in-house” development, meaning that its own personnel would design and deliver the system. (See Appendix 1.4.) The rationale behind this decision was twofold. JSC Engineering liked to provide its technical workforce with opportunities to gain hands-on experience working with hardware. This ensures that the Agency retains a highly skilled workforce that knows the challenges of developing new technologies across a project life cycle and can engage with contractors on a peer-to-peer level. The primary reason for this decision, however, was cost. 1

Motorola and Rockwell had independently proposed to build the system for over $90 million. The NASA team thought it could handle the job to design the system and to outsource the manufacturing for $20 million, a substantial savings for the government. NASA held a competitive bidding process to hire a manufacturer for the radios, and it selected Litton Industries from the three proposals it received to serve as prime contractor. Litton was expected to refine the manufacturability of the design during the Preliminary Design Review (PDR) and Critical Design Review (CDR) and subsequently manufacture the flight articles, with anticipated efficiencies of scale during the production phase. The contract called for Litton to work on a cost-plus basis during the design refinement phase and then on a firm-fixed price basis for the actual manufacturing of the radios. NASA retained primary design responsibility for the SSCS, while Litton served as the manufacturing vendor. (See Appendix 1.3.) A Difficult Reorganization The formal start of the SSCS project coincided closely with a reorganization within the Engineering Directorate at JSC. Two divisions, the Tracking and Communications Division and the Flight Data Systems Division, merged into a new Avionic Systems Division. At the same time, a new Project Management Office was created to manage the engineering project teams that in the past had interacted directly with their “customers” (e.g., the Space Shuttle or Space Station programs). Both administrative changes led to mixed morale, and several key engineers with radio expertise opted not to work for the Project Management Office, which now had oversight over the SSCS project. At about the same time, the Engineering Directorate awarded a new engineering support contract to Lockheed. While Lockheed was also the previous support contractor, the new contract had a number of provisions that drove off many engineers. All of the contractor designers on SSCS left the project in the six months prior to Litton being awarded the manufacturing contract. Starting Over In short, Lemke found himself at the beginning of his first significant NASA project management assignment under a new internal organization, with no engineering drawings, none of the designers who had worked on the earlier phase of the development, and a project team with no expertise in the SSCS complex radio system architecture. At this point he could apprise management of the gravity of the situation or he could find a path forward. He chose to persevere. He had a team that was ready to work hard, despite its inexperience with the inherited technical design. The project itself was a motivator; it was the biggest project in the division, the work was important and challenging, and it offered a rare opportunity to do hands-on hardware development. In Lemke’s estimation, hard work could overcome the setback he had been handed. Lemke and his in-house team of designers began the painstaking process of deriving drawings from the prototypes, using calipers, Ohm meters, and every other reverseengineering tool of the trade to determine the exact specifications of the boards. Every

2

measurement was an opportunity for a mistake; a single missed connection might mean that an entire circuit wouldn’t work. The NASA team’s progress proved excruciatingly slow, and Lemke realized that at this pace the project would never be completed. He approached Litton, the prime contractor, and explained the situation. Litton said that its engineers could re-create the drawings. Lemke initiated a contract change and handed the boards over to Litton. Litton was eager to prove itself on this project; the SSCS contract was its first at JSC. It hoped to expand its business there, and was willing to go to great lengths to ensure that the SSCS project was successful. At the end of eight months, the drawings were complete. Lemke’s project was now where it should have been when he arrived for his first day on the job. A Schedule Driven by Space Station The primary consideration driving the SSCS schedule was the construction of the Space Station. Since the Space Shuttle was an existing operational program with working radios, there was not schedule pressure from the Shuttle program or the EVA Project Office to deliver the SSCS by a certain flight. The Shuttle program maintained a busy launch schedule in the late 1990s, and another flight was always just around the corner. The Space Station, however, had hard and fast deadlines. From Lemke’s vantage point, the schedule pressure associated with Space Station stemmed from political considerations, not technical concerns. Most of the Space Station was being built by a large government contractor that was running behind schedule and over budget. As a result, any piece of government-furnished equipment (GFE) that was late, regardless of its function, provided an opportunity for the contractor to claim that the government was holding up the works and impeding its ability to deliver on time. “There was enormous pressure to deliver on schedule, even though everybody knew Station was way behind, even though everybody knew you didn’t need those radios to test out the U.S. lab module,” Lemke said. “It was an add-on. It was a game of chicken, and the government couldn't be late delivering.” An Immature Design The project sailed through its multi-staged Preliminary Design Reviews (PDR) and Critical Design Reviews (CDR), but the time it had lost re-creating the drawings also inhibited the maturation of the design. (See Appendix 1.5.) Litton was supposed to have spent that time turning the engineering prototypes into radios that could be manufactured and building some test units. Instead, Litton re-created the laboratory units, which didn’t meet the project’s requirements. This was clear from the performance of the SSCS Design Verification Test Units (DVTU) based on the drawings Litton had faithfully recreated. The DVTUs could communicate over the radio waves, but didn’t work well as a five-radio network.

3

With the schedule to deliver the radio for the Space Station closing in, Lemke elected to make the necessary fixes to the DVTUs in a piecemeal fashion rather than inserting another DVTU cycle to improve the design and address the problems on a systems level. As long as his team practiced good configuration control, the working DVTUs would lead straight to the qualification test unit and then the first flight production unit. Division management agreed with this plan in the hopes of meeting the Station delivery schedule. By this stage, Litton informed Lemke that it was having problems with its units. None of the units would consistently pass the specification tests. There was now a real concern that many key parts had a combination of quality and tolerance problems. However, the greatest concern was the growing realization that the overall NASA design had problems. “They were saying, hey, we don’t think your design really has enough margin. It doesn’t really seem to us that these are very manufacturable, that we’re going to be able to get repeatable results,” Lemke said. The chief engineer on Lemke's team was confident in its design, however, and thought that Litton might be looking for an expensive extension of the cost-plus part of its contract. The chief engineer asserted that the problem was with Litton’s manufacturing processes, not the design. Lemke pressed Litton to stick to the design and build the qualification test units as though they were flight units. Litton did as its customer requested. At this point the commonalities among the different radios became extremely critical. The modem and receiver boards, for example, were identical for all of the radios, so flaws in one would be reproduced in the others. Once Litton built a qualification unit for each of the three radios, the problems it had predicted began to surface. The units did not yield repeatable results, and the RF (radio frequency) performance was below par. A seemingly endless series of quick fixes ensued at the same time that Litton kept producing more radios, which led to constant reworking of all the existing radios, with changes that had to be tracked by serial numbers and recorded in data packages. The project operated in “fire drill mode,” scrambling from one problem to the next, which led to schedule changes on a weekly basis and no time for rigorous systems testing. During this time Lemke temporarily left his position as the SSCS project manager for another job. His replacement, Dave Lee, had joined the team earlier as Lemke's deputy and was fully aware of the difficulties Lemke had faced along the way. The managerial transition was a smooth one, but no one could have predicted the need for Lemke's return to once again help lead the project out of trouble. Within a year Lemke was back, and the team continued to persevere.

Flight Time The radios made it through acceptance testing, performance testing, and qualification testing. Some individual radios did not perform as well as expected, but they passed.

4

The time came to modify the Space Shuttle Orbiter and the space suits to accommodate the new radios. In the fall of 1998 the SSCS underwent a test flight on Space Shuttle Mission STS-95, which did not include an EVA (space walk). The test flight uncovered some minor glitches, including an instance in which one radio would not talk to another. This problem was attributed to operator error and solved by re-cycling the system’s power (turning it off and back on). The SSCS team thought the radios were ready for a real in-flight trial. The project delivered its radio to the Space Station in November 1998. At this point, the problems seemed to be getting smaller; there were still lots of fixes, but the work seemed manageable. Around the same time, Lemke and Lee accompanied the head of the Avionic System Division and other senior NASA managers to Litton to re-negotiate the requirements for the radios. The purpose was twofold: to establish a realistic baseline for technical performance on measurements such as RF output and sensitivity, and to clarify expectations for all the stakeholders—the project team, Litton, and the three customers for the radios. With the new baseline, the project had a clean slate. Preparations began at Kennedy Space Center (KSC) for Mission STS-96, which would launch in the spring of 1999. In the course of testing on the launch pad, the space suit radio (SSER) began making inexplicable noises. Lee and Lemke, who at this point was on board as a consultant to the project, flew to KSC to troubleshoot the situation. They began working around the clock trying to reproduce the intermittent noises that the radios were now making, but the causes eluded them. As they heard new sounds they characterized them with descriptive nicknames: motor-boating, rain on the roof, laryngitis. Their testing led to an examination of the radio environment at KSC, which was filled with RF interference. KSC conducted studies of the frequencies that filled the airwaves and tried to provide a quiet RF environment for testing the SSCS. It didn’t help; the team could never correlate the noises with any specific RF interference. But after hundreds of hours of testing, it managed to collect enough signatures—specific noises that behaved the same way each time—that it was able to develop recovery procedures, including power-cycling, that would allow the radio operators to work around the noises. With the launch fast approaching, the Shuttle crew had to be trained in recovery procedures in case the radios malfunctioned in flight. A Mission Operations Directorate official taught the crew how to respond, and made a cue card that actually flew on the Shuttle. (The crew was already trained in the use of hand signals in the event of a complete radio failure.) The last-minute training proved necessary. Astronauts experienced “motor-boating” during an EVA, and the procedures allowed them to recover gracefully from the malfunction. The Shuttle crew successfully carried out its mission despite the problems with the radios. * * * *

5

Aftermath and Recovery The problems with the SSCS before and during STS-96 were, in Dave Lee’s words, “a huge black eye for the project and the JSC Engineering Directorate” within NASA. The debriefings of the STS-96 crew after the flight drew the attention of senior management. The SSCS had failed dramatically in a high-risk and high-visibility situation. The Shuttle and the EMUs had to be retrofitted with the original radios at once for the next flight. At the same time, the failure also marked the beginning of the project’s turnaround. The SSCS team got unmistakable orders from management: go fix the system. Cost and schedule were secondary to finding the root causes of the problems. Every element of the design was reviewed. The break in the action allowed the team to conduct the extensive systems testing that it had foregone in the run-up to its first flight. The project also received additional resources to bring in experts who could help solve the problems. One expert likened the process to peeling an onion; each layer revealed its own problem, and peeling more deeply led to others. On close examination, it turned out there were multiple root causes for the failures. The receiver design was hyper-sensitive to RF interference. Taxicabs in Argentina were transmitting on a frequency that was close enough to cause noises. The signal processor software was also flawed. It required precise data from the modem—and the modem was fed by the receiver—or it had software problems, and would get confused and malfunction. The algorithms for networking the radios required extensive reworking. In addition, the testing revealed several problems unrelated to the root causes, such as poorly written software routines and floating leads in the hardware design. A year after STS-96, the SCSS was re-deployed for STS-101 in May 2000. There was a problem with the reference level on this flight, but the phantom noises that had plagued the system previously were gone. By the time of STS-106 four months later, the SSCS achieved error-free operations for the first time. It has continued to do so ever since. In the end, Lemke saw the failure as turning point that led to the resolution of the project’s difficulties. “That’s where we got to spend the time with our design to really get to the root cause. We got to do the testing, got to find out where the flaws were, and fix it,” he said. “It was just getting the team, the time, and the management support to solve it. There were no more Band-Aids. ‘Go solve it, and whatever it takes, you do it.’”

6

Appendix 1: Supporting Materials 1.1. General Project Description The Space to Space Communications System (SSCS) allows vehicles in close proximity to communicate voice, commands and telemetry via a radio frequency (RF) link. Each space vehicle utilizing the SSCS provides control, data, voice and power cabling as applicable. These cables are not considered part of SSCS.

Figure 1.1: Three radios enable communication among the Space Shuttle Orbiter (SSOR), the International Space Station (SSSR), and astronauts engaged in extravehicular activities (SSER). (Source: NASA SSCS Critical Design Review presentation, April 29, 1996.)

7

1.2. Space to Space Communications System Radios a. Space to Space EMU Radio (SSER)

Figure 1.2a: The Space to Space EMU Radio (SSER) allows astronauts to communicate with the Shuttle Orbiter and/or International Space Station during EVAs. (Source: David Lee, NASA)

8

b.

Space to Space Orbiter Radio (SSOR)

Figure 1.2b: The Space to Space Orbiter Radio (SSOR) allows the crew aboard the Shuttle Orbiter to communicate with the International Space Station and/or astronauts conducting EVAs. (Source: David Lee, NASA)

9

c.
• • • • •

Space to Space Station Radio (SSSR)

• • • •

Space to Space Communications System (SSCS) Time Division Multiple Access (TDMA) radio Mounted in Avionics Rack #1 in the U.S. Laboratory Module of the ISS Provides voice communications between the ISS and EVA astronauts. Receives suit status and biomedical signals from the EMU Provides a voice and telemetry/command link with the Orbiter Completely controlled by the Command and Control Multiplexer/Demiltiplexer (C&CMDM) on the ISS • Provides decryption of command link from the Orbiter to the ISS at one 64 word command/second • Provides a telemetry link from the ISS to the Orbiter • Command and control of the SSSR ISS provided mechanical interface - ‘6B’ box design Designed at JSC by a team of engineers from the Avionic Systems Division and Lockheed-Martin Manufactured by Litton Amecom Space Systems Operations First flight scheduled for May 1999 on STS 98 (ISS assembly flight 5A)

Boeing Picture

MODEM A

TRANSMITTER A DUPLEXER A RECEIVER A

CAVITY FILTER A

DIRECTIONAL COUPLER CP1

DIRECTIONAL COUPLER CP3

POWER DIV PW3

TO/FROM EXT ANT

AUDIO/CLK IN AUDIO OUT

SIGNAL PROCESSOR A

MODEM/SIGNAL PROCESSOR PWR CONVERTER A

• • • •

CMDS/DATA CHA CMDS/DATA CHB 120 VDC

1553 INTERFACE BOARD A

120/28 DC/DC MASTER A EMI FILTER A 120/28 DC/DC MASTER A

POWER DISTRIBUTION BOARD A IAA SW

TO/FROM IAA POWER DIV PW1 POWER DIV PW2

120 VDC RTN 120 VDC 120 VDC RTN CMDS/DATA CHA CMDS/DATA CHB 1553 INTERFACE BOARD B EMI FILTER B 120/28 DC/DC MASTER B 120/28 DC/DC MASTER B POWER DISTRIBUTION BOARD B

TO/FROM AIRLOCK TERM

AUDIO/CLK IN AUDIO OUT

SIGNAL PROCESSOR B TRANSMITTER B DUPLEXER B MODEM B RECEIVER B

MODEM/SIGNAL PROCESSOR PWR CONVERTER B

• • • • • • • •

CAVITY FILTER B

DIRECTIONAL COUPLER CP2

DIRECTIONAL COUPLER CP4

POWER DIV PW4

TO/FROM EXT ANT

Two radios within one enclosure Fully redundant except for a single RF switch on the Internal Antenna Assembly (IAA) Each radio can operate at either 414.2 MHz or 417.1 MHz Two power modes • Low power: 0.25 Watts (+24dBm) resulting in an operational range of 80 meters • High power: 5.0 Watts (+37 dBm) resulting in an operational range of up to 7 km along the Orbiter/ISS rendevous trajectory 6 antenna ports: 4 External Antennas, IAA, and Airlock Antenna Operational from 20 °F to 120 °F while sitting on a cold plate Frequency shift keying modulation with a burst rate of 695 kbps Intermediate frequency of 21.4 MHz Bit Error Rate performance of 1x10-5 when the RF input is -86.5 dBm Maximum operational power draw from the ISS of 72 Watts in high power mode Actual dimensions of 13.25”x18.6”x9.15” Weight is under 40 pounds

Figure 1.2c: The Space to Space Station Radio (SSSR) allows the crew aboard the International Space Station to communicate with the Shuttle Orbiter and/or astronauts conducting EVAs. (Source: David Lee, NASA)

10

1.3. SSCS Roles and Responsibilities (Notional)

Figure 1.3: Roles and responsibilities for the various elements of the SCSS program. (Source: David Lee, NASA)

11

1.4. SSCS Government Furnished Equipment (GFE) Deliverables

Figure 1.4: Many Space to Space Communications System components were delivered as Government Furnished Equipment (GFE) by the Avionic Systems Division of the Johnson Space Center’s Engineering Directorate. (Source: NASA SSCS Critical Design Review presentation, April 29, 1996)

12

1.5. Project Timeline

Figure 1.5: Project timeline and milestones from initial concept through 2005. Note: error-free operations continued up to the time of publication of this case study in early 2007. (Source: David Lee, NASA)

13