He who does not remember history is doomed to repeat it
-- Santayana
Keep an engineering notebook...this is the ultimate source of where and why things were done. Emphasize results of trade studies you performed, selection rationale...why did you do it?, results of prototype testing, simulations, and other demonstrations.
This is also the ultimate source of proof of intellectual property for PATENTS that can lead to licensing and royalties!! Every employee should have a LaRC Laboratory Notebook to give to the Office of Patent Counsel.
(1) Lay out boards so power comes in on same pins. Standards like VME and STD bus force designers to do this reasonable thing.
Communicate and document any custom interface/backplane/protocol conventions used in a subsystem to all designers involved.
(2) Key all cards to prevent damage
(3) Buy extra extender cards so they can be keyed also. This is very cheap compared to rebuilding an accidentally fried board
(4) Come up with "reasonable" spares policy and list early in Project Lifecycle
Constraints include cost, criticality (LASE 10% rule)
Identify "limited life" articles, parts that may go out of production
(5) Take advantage of internal peer reviews of design. Talk with the resident experts after you have walked through it yourself. Corporate knowledge and practical experience become available. This also gets requirements out on the table
Check each others drawings, at least within your work group
(6) A conservative approach that uses derated parts provides ample margin. Derate parts appropriately to meet reliability and application needs.
See Randy Regan for more information in Parts Engineering Group
Careful trade-offs must be made if flight constraints (volume, power) exist. For example, the commercial version of the LASE LDU had numerous parts that were marginal because they were not rated properly (e.g. 1000V diodes operating near 1000V became stressed and failed). Derated parts that fit into the volume were not available....the risk seemed acceptable.
Know your application and evaluate the part accordingly.
(6a) Carefully check sinking and sourcing capabilities of chips - you may need extra buffers
(6b) Read manufacturers' application notes carefully...may need a capacitor to act as a decoupler and minimize oscillation
(6c) Data sheets can mislead....read carefully and consider your application and the system characteristics.....For the GFCR, a "totally asynchronous" up/down counter had intermittent errors caused by the fact that it was in a system with a 50MHz clock (20 nanosecond period) that would occasionally hit the counter within 6 nanoseconds of its load or clear. The data sheet indicated that within this settle time 6 nanosecond window "all bets were off" ...solution...add a few gates to delay propagation...so much for total asynchronicity. In a slower system, the chip would probably behave "as advertised"
(7) Power Supplies & Distribution
(8) Treat GSE as a separate subsystem. It can be a project in itself. Identify GSE and tools you will need to perform and check out your design. This information will be necessary to identify possible scheduling bottlenecks during test & integration caused by "common equipment".
Use commercial systems whenever practical - usually easier to maintain, calibrate than custom stuff (LITE's 3 detector channels bear this out -- custom IEEE-488 s/w, test equipment require elaborate procedures or one person to be constant contact) This goes for software too - TLS of LASE used C Toolboxes, C Users library routines, Shareware to generate procedure dependency lists, etc.
Identify any special needs for the GSE such as 400 Hz power generators
(8a) Spend time & resources to build instrument simulator or data sources of reasonable fidelity. Incorporate features of data, such as "burstiness" -a 5Hz data rate does not necessarily mean one byte every 200 msec. These will be invaluable in checking out GSE. Have ability to generate repeatable known data pattern.
(9) Design for testability and maintenance. This requires close cooperation of electronic designers, layout, and mechanical folks. Design electronics boxes (enclosures) with access panels, board placement, cabling and "plumbing" for maintainability. The original commercial version of the LASE LDU is an example of an enclosure that was hard to maintain. Consider designs that let sides flip up or fold back for easy repair or wiring modifications. Consider the following in the total package:
(10) Fuses should be based on wire size, not expected current. (LITE LTM)
(11) Keep ICD up to date. Change it to reflect operation, not the other way around!
(12) Minimize use of pots. If it is only set once, and pSpice will help you calculate it, use a fixed resistor. But if you must use pots, stake them after calibration, before vibration. (LASE LDU)
(13) Provide over-voltage protection for ADCs, especially in monolithic microcontrollers. It may be possible to intervene between the MUX and the internal S/H. (LDU)
(14) Use common hardware in design wherever practical - e.g. same micro-controller, serial chips, interfaces, etc. This allows cost-sharing in development tools, re-use of any embedded software developed for components.
(15) In light of UARS solar array hassles, perhaps we should also consider a "minimum-power science mode", so we are not in an all (operational with science) or nothing (survival heaters) situation if power gets really low (HALOE)
Make sure that critical models and analyses are done before the concept is presented at the design review. Pay special attention when the model is being developed by members of another organization. Work closely with modeler in defining test cases! For example, a new Thermal Control System concept needed a basis in a good model. The LASE TCU had to replan when the thermal model showed deficiencies...unfortunately the results were delivered after the review.
• Bring in the software specialists early so a sensible allocation/partition of requirements can be made. The delivered system should let the software and hardware both do what they do best. Software can impact the hardware architecture when sizing (code and data storage) and performance requirements (interrupts, response time, real-time calculations) are well understood. Don't count the cards in your box until the software guys have been in on the design.
• Document requirements up front, but still design for flexibility
• Minimize the number of languages used in a project to maximize effective use of programmers, development tools and verbatim reuse of common code (drivers for peripherals, serial communications, etc.)
• Use compiler features to enhance portability of code across platforms - select ANSI C flagger in a C compiler for example - since proprietary extensions such as C++ for Borland can get you in trouble. Use of the ANSI standard constructs enabled C code for a SCSI card to be transported successfully from a PC to the TI TMS30 with only one source line changed
• Use PCLint, xref capabilities of C to aid in documentation. Have project standards in documentation - e.g. "meaningful" variable names (consider case sensitive languages, O/S), indentation, rev dates and notes as comments. Use language style-guide to enhance readability and maintainability
• Use compiler features and/or self-discipline to type all variables. VAX Fortran "TYPE NONE" is a great way to force definition of all variables. Also, C on different platforms and operating systems (DOS versus OS-9) assume different types - e.g. int may be a long or short integer!!!
• Use some form of version control! Polytron worked fine for LITE GSE. AMSD folks used it for TCS too. Remember, not just the source...there are include files, "make files", compiler features invoked, and others to keep in rev so the binary and sources all line up.
• Clearly identify versions in a non-automated system. Clarify differences between similar systems that are caused by non-software impacts, such as the calibration curves of sensors. The inadvertent use of different derived constants in equations can have harrowing results. The LASE LDU software looked identical for both units, except for pressure calibration equations and constants. Software for both was named LDU rev X. The units behaved exactly the same at ambient conditions, even if the wrong software was installed; but once in the thermal-vac chamber, the computed results were significantly different from the actual pressure in the chamber! After an incident, the software was clearly identified as LDU 1 rev X and LDU 2 rev x with a build sequence that prompts the user for the target LDU to ensure correct parameters.
• Back up files and archive the masters in a safe place. Remember to include flight and GSE software and critical data files and parameter files. Keep originals for commercial off the shelf software and license information in a controlled area. Input files generated to run beam code or ray-trace programs can be critical to mission success - hard drives can fail at any time.
• Use compiler features to aid in testing. VAX Fortran lets you check for array bounds violations, etc. Then "turn them off" for actual mission execution.
The #ifdef construct can help in C code to ensure code is executed in the correct environment.
• Use compiler or assembler features to check out interrupts in a controlled manner. (In ASM-86 there is the software interrupt instruction, that allows an interrupt to be invoked by software, rather than a hardware signal. This removes doubt about the performance of the actual code and correct vector and mask handling.
• Take the time to learn how to use emulator features. The use of procedures in the ICE made it easy to debug LASE WMS, portions of LITE IC.
• Select well-known "robust" commercial software (operating systems, languages, toolboxes). Problems with the MAPS GSE may have been caused by OS-9 deficiencies, not the custom applications programs.
• Clearly partition or mark non-flight code if codes are run in a breadboard and flight target environment. Use a convention like upper case for flight and lower case for non-flight. This will prevent "artifacts" that can cause trouble when code is moved to the target. (e.g. bit set/resets, print statements that should be removed, etc.)
• Make sure target environment is the same as test environment - that all libraries and functions are available. Have same clock rate and OS, if possible. CLASS-2 had problems because the ATAN2 function was not available in the flight configuration, but was in GSE! This was not uncovered until it was installed in the aircraft and the scanner went wild!! The LASE Wavemeter testing was done on an AT running at 10 MHz; results were then extrapolated to match expected performance on the flight 8 MHz CPU.
• Use dynamic memory allocation schemes carefully. Clearly document use of compiler features that support it (e.g. malloc and dalloc in C)
• Investigate use of test tools. AdaMAT and AdaTEST from IPL provide static testing, coverage reports, and dynamic testing on host and target. There is a set from IPL available for C and C++.
• Beware the use of reserved words!!! MOTIF, X11 and other GUI applications have reserved words that can get an unsuspecting developer in trouble. For example, environ is reserved.
• Beware of "endian wars" - make sure both hardware and software people know which bit is the MSB and which bit is the LSB! This also pertains to byte flips that can occur when data are shipped from a flight processor to a different machine for scientific analysis.
• Collect software metrics. A the end of the project have at least the following:
SIZE and MARGIN (How many lines of code? How much EPROM-code? How much RAM - data? And how much was available or did we have to add on and when?)
COMPLEXITY (How many modules? How tightly coupled to hardware and realtime events? How much complex calculations? How interactive? e.g. command modes used in LITE, number of tasks and rendezvous, or asynchronous interrupts?)
FAULTS and ERRORS (How many anomalies were uncovered at unit test? subsystem integration? system integration? environmental test? during mission operation?) Categorize the errors: Were these errors design errors, simple implementation errors?
EFFORT and SKILL MIX (How many people, skill level, FTEs were used to do the work...include flight, GSE, special testing, documentation, simulation and analysis codes...too often we remember only a few of the more visible folks, not the whole group!!! This information is necessary for reasonable cost estimates.
TOOLS and RESOURCES (How many workstations, printers, compilers, simulators, emulators, CASE tools were used by the software staff? Was there contention for these things?)
The Branch is developing a standard drawing format that will be documented in "Electronic Systems Branch Drawing Standard Practices", a draft document under construction. The document will elaborate on these notes. Any computerized schematic capture package, including the CADENCE Concept software, can support this format.
(1) KEEP SCHEMATICS UP TO DATE - deliver with as-built schematics and engineering logbook to give rationale for changes if not shown in ECNs
Each board should have a folder of drawings associated with it.
Make sure that components do not inadvertently get placed on wrong side of board
(2a) Make sure project schematics meet LMI requirements so the Fab Shop will go forward with layout - maintain contact with the Center's Drawing Committee rep - Regan (use approved borders for CASE and CADENCE schematics)
Attempt to get a standard style for all schematics for a Project (LITE)
(2) Create a special "harness" schematic that shows your cabling, termination of shields, lengths, etc. Include any notes describing special features, connector requirements.
(4) Use acceptable implementation approach - jumpers do not fly in high frequency world, but a MUX will; test points must be chosen with care - they can become an EMI threat
(5) If performing design activities on stand-alone machine, backup drawings, docs, etc on a weekly basis so valuable work is not lost if a disk crashes. CAEDE users are automatically backed up nightly.
(6) All HPGL plot files are not created equal! Non-CADENCE plot files may not work well on the E-size plotter currently at Version 2. Known problems:
Assess EMI threats early. Be on the lookout for high voltage power supplies, switchers, etc. Microprocessors can be a source of high frequency digital signals that interfere with sensitive detectors. Contact EMI/EMC Facility Manager for help in generating test procedures, evaluating design.
• Switching power supplies usually require EMI filters.
• Standard wiring practices for spaceflight are often based on mechanical reasons (i.e. use of "pigtails" to provide strain relief). However, they can cause EMI problems. ("pigtails" are great little loop antennas)
• Partition electronics into analog and digital sections, locate in shielded enclosures; shield and/or filter interconnections between analog and digital sections; provide separate/isolated regulators.
• Flex capsules for CERES.....TBD
(1) Carefully consider and document power distribution & grounding schemes. Use computer-aided design schematic capture software, such as Concept on CAEDE!
Use what satisfies your performance requirements and clearly document it! HALOE created E-size plots to show running of grounds and power through connectors and major components. (Bitnoise Tiger Team) These schematics were also created for the LASE TCS. In fact, as soon as grounding problems appear, the designer is asked for the grounding schematic - and usually has to create it on the spot!!
Consider most likely "sneak paths":
Consider use of central DC-DC converters and localized power regulators.
Try to keep "quiet" 5V for digital localized. Do not run it all over the place -- out to cards with pots, etc.
Multi-layer vs. two-sided boards: It is preferable to have separate 28V and 5V ground planes, but you may be reduced to creative use of thick traces and pads on two-sided boards if severely constrained to a specific format. (LASE LDU I redesign)
Cabling can add to problems - look closely at shielding, bundling and connectors. Cables are notorious antennas.
- need lessons from Hodson & Hines & Wilson
Shop around for a foundry. Even though ASICs are expensive, cost is not the bottom line. Time is. Get recommendations from experienced users, such as JPL. Skim the ASICs handbook from JPL in the EEE Parts Office.
(1) Get gate counts right so fab and layout estimates by foundry are not way off. After so much tweaking, the vendor will run out of real estate and then will propose a costly redesign.
(2) Ensure that all files needed for simulation are correct. There were questionable files in the SODR test case - capacitances were missing or incorrect.
(3) Currentness of vendor is key. One vendor required that we use a VeriLog that was two versions behind current supported software from CADENCE. This was a licensing nightmare. The vendor also did not support E-mail to electronically transfer files. Each design file had to be mailed as a cassette and turn around time was increased. JPL said they dealt with this group once, for a few months, and got out quick.
(4) Simulations took a very long time because they were run interactively. This ate into the schedule.
Before buying anything, check the bonded stores from previous projects. There may be free spares or replaceable parts (this saves schedule). Parts are also available from other Centers through the EEE Parts activities.
If using a credit card, be sure that the procurement rules about source selection are followed.
(1) Beware of the "minimum buy syndrome" when doing initial costing of parts. Sometimes you have to buy 50 to get 12. This also pertains to connectors that are often bought in piece parts like 1000 pins.
(2) Make lot buys on materials like fiber optics that can be traced throughout prototyping, engineering model and flight.
(3) Ask vendors the right questions. Is it on the shelf? Do not create a crisis for the buyer by discovering that the item has a long lead time or is only available in Canada!
(4) Work with the buyers. Understand their constraints and factor them into the project purchasing schedule. They need multiple sources (at least different distributors) whenever possible. Remember, above certain thresholds mandatory delays kick in to handle advertising in the CBD for up to 30 days!
(5) Determine whether a contract, credit card purchase or a P.R. is the proper vehicle. The SODR ASIC procurement through multiple large dollar value P.R.s should have been done as a contract.
(6) Do not use hi-rel parts for prototyping...use stock or commercially available functional equivalents. Unfortunately, there have been examples of 883-B parts being sent from stock when commercial grade components were requested. The 883-B parts are not useable because the documentation was not available!! In this case a $400 charge was run up when $10 should have covered it.
Bring in a fabrication specialist as early as possible! The layout must be compatible with both logical (electronic) performance requirements and physical constraints. Board sizing and placement cannot be determined without fab inputs.
(1) Do not place feedthrough "vias" in line with normal holes. If a package is inadvertently slipped over one hole, the error is hard to catch. A SIP package was "left justified" when it should have been "right justified" on the LASE DRS
(2a) Until we have a tightly integrated electronic design and layout environment and true back annotation, spend time to go over artwork ("pen plots") before committing to the shop. (LITE)
(2b) Help the shop by creating a "paper doll" layout. Unless a co-located layout specialist is assigned, this is the best way to communicate chip placement. (LITE)
(3) Keep schematics up to date (manual back annotation to reflect pin swaps, package changes, etc.) so there are "as built" drawings and parts lists. (LITE LTM)
(4) Communicate special considerations via notes on drawings, "turn-over" meetings with layout person (short run lengths in high frequency domains)
(5) Also consider obvious ways to identify component side of board such as etching a word on component side.
(6) Danger points in automated artwork -- be wary if your work is going through a translator! (LASE Ti:Sapph hole size change on flight board re-do that came for free because of "bad" translation program")
(7) Have spare boards made, but not necessarily populated. It costs as much to run 5 of the same card through the shop as it does one! The real cost driver is in the initial layout (MAPS & LITE)
(8) Ensure compatibility between layout and Gerber plotter. Use customized files that folks in IRD have created to access the "non-standard" features of the Gerber wheel currently used in Fab. (LASE TEC boards for RCS) This sort of incompatibility can happen in the TANGO system also...config files must be right
(9) When populating a complex board, put in components incrementally and test, rather than filling up whole board and going for broke. Design must support this approach that will readily reveal problems with boards and save diagnostic time in the long run. (LASE DSE & LDU redesign)
(10) Employ a reasonable fidelity mockup for cabling. The mockup should include realistic "obstacles" to enable reliable routing, connectors, hole patterns for tiedown placement, and heights for proper clearance around optical and mechanical components. Remember that cables cannot bend too tightly. This enables a parallel harness design and a chance at a right fit the first time. (LITE coldplate, PASDE mock canister assembly, MAPS)
(11) Cables, like luggage at airports, tend to look alike. Label everything! Clearly label the connector on the board or box to match the end of its mating cable.
(11a) Cable lengths can be critical...especially with high rate signals...may have to specify length equality tolerance for cable pairs based on the HDRR incident on LITE....say within plus or minus tbd inches. The CLK and CLK* pair of cables, and the DATA and DATA* cable pair had a 40 foot difference in lengths. Post-mission testing demonstrated that the data system could function with pairs that had up to a 35 foot discrepancy, then things fell apart!!!! Initially it looked ok, but this may have been an artifact of the GSE being more tolerant than the orbiter "real thing". Forty seconds of good data found on one of the flight tapes duped in February 1995 supports this "hairy-edge" clocking theory.
(12) Make sure that connectors and insulation for cables meet environmental specifications. Power cables for LASE had to be redone because the selected insulation was not approved for use on the ARC ER-2 aircraft.
(12a) When building a "GSE" cable, remember that the end that mates with flight hardware must be "good". The GSE end can come from Radio Shack. Also consider where the cables must be. Outgassing can be a problem in the Thermal-vac chamber. Shells may need to be sandblasted if they are not the flight style.
(13) Mount components on the board so they can easily be identified. Make it easy to read resistor values so board audits are simple.
(14) Surface Mount Technology (SMT) is still fairly new to the Center for flight programs. Use extra care during inspection of soldering of SMT components. Beads of solder may cling to the pin and the board, but not bridge the connection. This occurred on the LASE DRS with a "gull-wing" SMT and also on SUNLITE.
There are times when a very small board is needed in a hurry to prove out a concept. This is the time to use lower-end tools and mill your own board.
(1) Be aware that prototype boards may not have plated-through holes, so soldering on both sides of the board will be required.
(2) Rapid prototype tools such as Tango use lots of translation and configuration files. Check all parameters so artwork turns out ok.
(3) These set ups require some expendables such as board blanks and drill bits. Replace as you use.
Hardware designed for the flight environment must be ruggedized and protected. Requirements must be communicated clearly to the manufacturing people - a checklist or handover sheet may be necessary to prevent slips. Be sure to have a fabrication specialist evaluate the flight worthiness of commercial cards. An inspection may reveal additional costs (fab hours) and time needed to get a commercially built card through environmental tests.
(1a) Conformal Coating
Conformal coatings selected for flight project should be reviewed by a fab person (materials specialist) and the engineer who understands the application and environment (e.g. vented vs. non-vented, outgassing threats related to placement in shuttle, or nearby optical components) Identify portions of the board that should be masked during coating process. Document rationale behind coating selection (MAPS)
Consider other physical aspects of coatings such as prep time, shelf-life, cure time, thermal COE, strengthening characteristics and transparency. For example, one coating, CONAP, is clear and adds some strength to components, but is susceptible to flowing when exposed to high temperatures that would be encountered during solder repair. Red-ex is brownish red (obscures parts so visible inspection of damage is impossible), is easy to prepare, dries in minutes, and only provides insulation. A trick to play with an opaque coating is to only coat the leads, not the body of the part if possible. (LASE)
Also get all the necessary information on conformal coatings applied to commercial boards and possible effects of overcoating (LITE & SESCO boards)
Consider the effect the coating has around different electronic components. Solithane, the coating of choice for spaceflight, almost encapsulates the low profile surface mount parts. Current coating application procedures are based on the use of larger parts like DIPs. This can have a bad impact on stress relief during thermal cycling. (Orbital Meteor Debris Counter)
Remember - conformal coating and staking take time - and these operations usually cannot be performed on holidays or weekends so build it into your schedule!
Coating on top of coating has an insulating effect that may be detrimental to your design. Parts may overheat.
Clearly identify areas not to be coated. Don't coat chips in sockets. Do not coat areas where there are test points or mounting holes. Coating can break the connection.
(1b) Potting Materials
TBD
(1c) Paints
Paints can be conductive (iridited) or nonconductive (anodized). Clearly specify what you need or shop people will pick whatever is available and easy to use. Connectors do not meet the chassis for good ground contact on anodized coating and the surfaces must then be sanded and continuity rechecked. Rework can eat into the schedule. (PASDE)
Anodized paint can be used successfully if areas where contact must be made are masked off. This includes where bolts connect the box to the platform and any connectors that need a chassis ground. (LITE Aft-Optics)
The color of paint can also be an issue. HALOE thermal models assumed the instrument would see parts of the spacecraft painted black. In the end, UARS was white. HALOE had to change some of its mission operations because of the brightness of the background that changed thermal characteristics.
Paints can also outgas - check with optics specialists about requirements.
(1d) Staking
TBD
(1) Inform Model shop EXPLICITLY about use of enclosures and expected temperature ranges. The model shop built up and painted the enclosure used for LASE CDS Thermal Cycle testing performed in April 1992. The cognizant engineer told them that the enclosure would be used for thermal testing, but nothing was written down!! When the enclosure was opened after fours days of cycling, the cream colored paint or sealer applied to the box had outgassed significantly, leaving a residue similar to spring pollens and sap. This would have been the death knell for a system containing optics. The jury was out as to what this deposit may have done to connectors in the vacuum situation.
The paint or epoxy used on enclosures should be identified. The shop can use stuff that breaks down at 120` F or 200F. Unless specifically directed, they will use the compound with the lower breakdown temp because it is easier to handle. In case of disaster: Consult with fab about cleaning procedures for electronics and contaminated chamber, ohm out connectors, cycle a specimen and observe reaction.
(1) Run a witness plate through a thermal cycle when using a home-brew chamber. You never know what will outgas.
(2) Cap all unused connectors during thermal cycles.
(3) Run at least one operational cycle using specified operational range. This gives insight into temperature-induced behaviors such as "slow" ADCs when things got too cold in HALOE. (LASE too)
(4) Use the AMSD "YellowBook" and LMI for testing in Langley Program Assurance Manual as guides in drafting test procedure levels; true levels are determined by the actual carrier or designated document - AMSD Test Engineering group or Tom Jones may aid in test definitions
(5) Make sure all test equipment is calibrated when running formal procedures!! Be especially wary if you must borrow specialized equipment...lay your eyes on it early enough to get it calibrated if necessary (a MAPS lesson learned). Carefully document any "bootstrap calibration" done to use "out of cal" stuff. The test procedure should require checkoff for each piece of gear. Equipment is typically calibrated in FELICS "blue box" rolled into storage areas of B1202.
(6) Cabling into the EMI test chamber is critical. These act as antennas. Typically, cables have been wrapped in foil (HALOE, LASE TCS) which is a time-consuming and imperfect process. The preferred approach is to have flight or flight-like cables with the unit under test. This makes the testing reasonable and applicable to the flight environment. Another approach is to have test cables made up that allow shields to be tied down as needed. Make sure length is correct or "worst case".
(7) Thermal-vac testing will bring out the arcing and corona properties of the unit under test and its fixtures. Beware of high voltage bulkhead connectors - pot as necessary (LASE TLS Q-switch Driver Test)
(8) As long as the hardware is running - RECORD THE DATA!! The data and the system configuration when it was taken (h/w and s/w) can prove invaluable for trend analysis or characterization. Then ARCHIVE it so valuable storage space is not taken up. Nobody wants a test to abort because the disk filled up. Consider data compression packages (PKARC, etc.) if files must be transferred over serial lines. Use Ethernet whenever possible. File preening can take a long time when megabytes over YMODEM are involved!! (Note: MAPS GSE was incompatible with 8-bit EtherNET card)
(9) Understand policy of cleanroom vis-à-vis cleanliness of units under test. Cabling usually requires a 24-hour bake-out. Electronics may also require several days of bake-out...this time must be included in the schedule!
(10) Reconfirm your "appointments" with environmental test facility personnel. Let them know of schedule changes. Otherwise two things can show up at the same time!
(11) Determine requirements in test configurations. Is the proper power service available (400 Hz? 3-phase, etc.) are there enough receptacles? are cables ready? any special sensors?
(12) Make sure that good thermal contact is made for radiating components. Power supplies for LITE had a slight arc to them so only the four corners made contact with the base plate. Frank Hill had to hand sand the supplies after a thermal vac test showed that the supplies were getting too hot.
(1) Electro Static Discharge (ESD) can destroy hardware!!! Follow the rules...look for the tell-tale triangles on all ESD sensitive parts. Handle all flight and mission critical GSE hardware with care, especially the ESD sensitive parts with only one triangle on them (LITE and LASE)
(1a) Even laser diodes are ESD sensitive. Build in safety in handling (bag during transport and use static straps) and operational procedures (LASE Ti:Sapph)
(2) Don't bring up a system without its monitors functioning especially laser systems (LITE LTM)
(3) Be wary when equipment and hardware are moved to test facilities on the Center. Trade off the use of movers vs. riggers. Watch how they load things on and off the truck. Have a moving procedure or plan (MAPS & LASE)
(4) Test and label all cables and connectors - even for simple GSE-to-GSE hookups. Do not waste time because a bad cable has been introduced into the setup. (MAPS prep for EMI test)
(5) Be very careful around high voltage! Too often we only assume a TTL environment. Do not be near open boxes like the LDU. Have plenty of space around benches for workers. Use partitions to separate the work area from casual visitors...Woe unto he who drops a connector onto the SCRs! (LASE LDU)
(6) Clearly post Laser Safety Permits, lists of eye-safety certified personnel. Keep up with paperwork so facility will not be shut down. Do not move lasers from lab to lab without getting okay from laser safety office. (LASE TLS)
(7) When testing, setting up, or operating, DO NOT STRAY FROM THE PROCEDURE without talking to a "buddy" about possible impacts to people or equipment and redlining the procedure if the decision to change is made. For example, removing an attenuator or slightly changing an optical path can result in eye-damaging specular reflections or burned-out detectors. (LASE)
Projects as a whole run into the same challenges, no matter what disciplines are involved in the end item deliverable.
(1) Maintain consistent core of people if possible and practical learning curve battles are hard to fight - jerking people on and off projects as priorities dictate kills morale, makes Project Managers angry, and takes time away from productive work.
(2) Prevent "rampant engineeringitis" - requirements flowdown must be there - stay focused on goal, distinguish niceties from critical requirements (Better is the Enemy of Good Enough); this kills morale too - Zeno's paradox - you never get done. Work it out with the customer, by explaining the costs and benefits of various options.
(3) Communicate! And then communicate some more!
Small or large changes in one system may have significant impacts on others. For example, a radical departure from the original temperature and current controlling scheme for LASE Titanium:sapphire diodes to be used in the breadboard laser, flight laser, and diode characterization lab was made. These changes, which actually were significant improvements, were only "small matters of programming" made to BASIC code in the characterization lab. However, they were not passed on to the other software developer writing the corresponding flight code in C. Much time was wasted trying to understand why characterized lasers acted differently in the lab and flight environment! Once the software mods were made, progress continued.
Learn from the unexpected and communicate with fellow designers Failures are often more instructive than successes. (internal Engineering alert or lessons learned) Also, note the constraints so the reason for doing what was done is known, e.g. "there was only $500" or "it had to be pin for pin compatible". Hold these thoughts for the Lessons Learned Reviews
(4) Document "requirements" early. Even better, document the concept, proposed operational scenarios that ultimately define why we are here and drive out the requirements. TBDs are acceptable, even expected at CoDR level. A TBD means open for discussion and analysis!! LITE used the Software Requirements Document as its bible after months of negotiations with Science Team members and other Centers.
(5) Attempt the new NASA paradigm
Can we really build a Class D experiment? What sort of Product Assurance do we perform? Will COTS parts make the grade? What risks will we accept? Clearly document the true costs of "faster, better, cheaper".
How do we approach partnering? Clearly define responsibilities, deliverables, and accountability with the partners. Ask the direct questions: Who does the procurement? Who keeps the parts? Who does the testing? Who keeps the documentation? Who defines the schedule and cost? Who does the integration and mission ops? How are changes communicated?
Plan the Work; then Work the Plan
Define all tasks and resources needed to produce deliverable. Resources include people, facilities, equipment, hardware, software, and documentation (test procedures). Use an automated tool (e.g. TimeLine, MacProject, Artemis) to perform leveling and report any over-used resources (e.g. 150% of one person, lab conflicts). Make the tool the standard for the project and get all copies at same revision. Assess reality with project staff. Clearly communicate schedule to personnel and line managers. Listen to the staff. Everyone must buy in to the schedule in order to make the delivery date.
Take into account holidays, vacations, and possible sickdays. Do not start out with weekends and overtime. Include set-up and break-down time of equipment for environmental tests. Allow for meeting time - do not assume a 40-hour week! Set aside time for meetings, briefings and training. Some operations (e.g. conformal coating) cannot be performed on weekends, even if the schedule must be compressed.
When dealing with physically remote partners, consider logistics of shipping equipment and personnel to perform fabrication and testing. A simple modification may require three days of schedule time if partners are on different coats.
Add an extra task that will provide the time to deal with "unknown unknowns". Call it subsystem diagnostics or something generic.
Make sure adequate personnel and equipment (setup and training time) are available to perform tasks in parallel if schedule dictates (LITE detector channels)
Clearly define scheduling terms such as resource, duration, effort, and linkages so good data can be collected!! This may require a quick tutorial or a mini-glossary. Have decent forms for staff to fill out so a schedule can be generated.
Remember the "mythical manmonth".
• Some tasks are uncompressible. The classic example is one woman takes nine months for a successful pregnancy- nine women cannot each contribute one month! REVICC, a well-known software cost tool, will only allow a task to be compressed by 25% no matter what.
• Throwing more people at a late program usually makes it later because of learning curve and communication impacts. If the work is well defined and partitioned, schedule can be saved by carefully applying skilled personnel to specific tasks.
LaRC mandates certain formal reviews for spaceflight and Construction of Facility projects. The content and timing of these reviews are described in LMIs.
(7a) Hold peer reviews, not fear reviews. Be honest and point out possible problems. This is a good informal opportunity for having extra eyes evaluate the design. Remember, the design is being evaluated, not the person who is presenting it.
(7b) Schedule design reviews when sensible information is available that fulfills the specific agendas.
When is it time for CM? This is still up to the project, and typically formal CM occurs after the CDR. Responsible designers should save or checkpoint their hardware and software designs as they progress. Computer-aided design tools make checkpointing a simple process.
What should be covered under CM? CM should cover mission critical items. These include the flight deliverables and the GSE needed to test out the end item. GSE is complex and easily modified. Uncontrolled changes in GSE can appear as problems in the deliverable.
How much CM is enough? Do not overkill - the requirements for spaceflight and aircraft systems are different. Before setting up a CM program, consult with experienced engineers. CM and QA should not be burdens. CM provides an up to date description of the configuration of an item. Knowing this can actually save time. CM does not prevent change - CM provides an understood process for change, and the history that documents rationale for the changes. It is a consolidated collective memory: there is no need to hunt through personal notes or call out vague recollections.
Required:
• Up to date log books in a clearly visible spot near the deliverable/flight items and other designated mission critical elements (GSE).
In complex systems, a white board with the current status may be helpful so testers do not need to look at lots of log books. Entries in the LASE breadboard laser log book pointed a tiger team in the right direction toward the solution of the "doughnut shaped beam" and spectral purity problems.
• Communication of changes by circulated ECNs, CCBs, or an advertised log entry:
An uncommunicated change in the thermal pad for the LASE TLS HDU cost a week of troubleshooting. This change was the right thing to do, but it induced a thermal lag that was finally uncovered during line-lock testing and was initially ascribed to other sources.
• Shop Travellers should be with board or relevant assembly during all phases, especially any rework, ruggedization or repair. Travellers capture any changes made to a unit since it is baselined into the system.
• Bonded Stores and Controlled Access
Every project needs controlled storage of flight/deliverable parts, spares, and dedicated test equipment. Reasonable temporary storage area is needed for mission critical items during testing in B1250.
Testing is the discipline that currently provides the "proof of the pudding" to the Project. Testing requires a deliberate approach to verify that all performance specifications are met. The design should incorporate whatever hardware and software features - test points, diagnostic ports, test modes - will be required to verify performance. Consider use of error injection through simulators to put instrument hardware and software safely through anomalous operations to test error detection and recovery.
Define levels of formality in testing and control for both hardware and embedded software:
Axioms
If you spec it, test it
The LASE LCU duty cycle was inadvertently inverted because of change in polarity in fiber optic cables. The functional test only verified toggling of levels (that there was a duty cycle), not the required duty cycle. Two flight components were stressed because they were held on 95% of the time, instead of off. The duty cycle should have been monitored also.
If you test it, document it
Create a table of items specified for test, method of testing, and a checkoff for when the test was run. This makes traceability very clear and shows what items remain to be tested. The LASE DRS was required to recover from power cycling, but the system never encountered the situation in any functional tests. When it was put through the thermal-vac test, anomalies caused a power drop out, and a software error was discovered - and subsequently corrected to support recovery.
Corollary
If you can't test it, don't spec it
A lot of time and effort was spent in attempts to specify requirements on SAFIRE when there was really no way to verify the performance. A good question to ask is "How do we know..."
Bring back the engineering model or make a schedule so that the breadboard is truly flight-like! Time is lost when results from the breadboard are not applicable to the flight model or breadboard induces failures that will not be possible in flight version (troubleshooting at windmills). The engineering model is still the best risk retirement tool we have. For example, EMI/EMC is very configuration sensitive, so the more flight-like the model, the better off the project is at uncovering problems early. Spend the time testing at component, board and subsystem level. The payback will be a much smoother subsystem and system integration. Incidentally, both LITE and HALOE had engineering models that were referred to as wirewrap prototypes or "spread systems".
Also, any changes made to flight hardware require QA approval and paperwork. Even a simple wiring change or touch up on coating can take a whole day on a flight board, when it could be minutes on an engineering model. Significant time must be spent documenting how a change was made, rather than the more important why. Time could be taken to document the why on the engineering board.
LASE-
Signal Processing - finally replaced the HP system by a more flight like Camac system with a PC-based data acquisition system - needed to deal with grounding, power supply issues in realistic setting
Ti:Sapphire - serial vs. parallel lamps caused hassles in transferring results from lab to flight unit
realistic PFN needed --did a special build; actual characterization of diode in one system not applicable to operation in flight set-up!
Optical throughput from breadboard and flight optical benches differ enough that calibration of energy monitors on breadboard would not transfer to the flight.
TCS - EMI testing surfaced problems with power and grounding. Troublesome cables were not "flight-like". Time in the chamber was over a month because the problems had to be understood and fixed.
LITE -
HALOE-like "spread" system of spare flight boards provided good test facility to check out final software changes for the IC and Boresite. Most engineers provided "stealth engineering breadboards" of their subsystems. Sockets make it easy to put in new code, yet provides high fidelity environment that supports parallel testing.
HALOE -
The grand daddy of the breadboards...the "spread system", is still being used to check out software patches before they are uplinked to UARS.
| EMI/EMC Facility Manager | Courtney Rollins |
| Mechanical | Flight Systems Test Engineering group, or Tom Jones |
| C (PCs) | Chuck Antill |
| Mike Vanek | |
| C (TMS) | Mike Grant |
| Shelley Stover | |
| ASM-86 | Rosenbaum |
| ASM-51 | Mike Blythe |
| C (x96) | Frank Peri |
| Ruben Remus | |
| Chuck Antill | |
| Ada | Danette Allen |
Throughout its phases, a project requires people with varied skills. The OMDC project flown on the Clementine Mission of January 1994 that is an example of a small fast turnaround instrument (<6 months, <1M$) ended up with about 30 people on the team. They were not all full-time, but 30 different people from all over the Center were needed to complete even a small project!
This document generated by Pamela Rinsland, ESB, AESD.
If you are reading a hard copy or local copy, the Web document is maintained at http://meweb.larc.nasa.gov/amsd/refs/esb_less.html.