Teaching Chapter 15: Robust Design
Timing
While it is beneficial to consider product robustness as early as the concept stage, experiments for robust design are used most frequently during the detailed design phase as a way to ensure the desired product performance under a variety of conditions. Thus the topic is usually introduced later in the semester.
Objectives and Strategy
The process of Robust Design should enable organizations to design products that operate within the performance specifications under the effects of noise. Learning the principles of robust design is accomplished using an inclass exercise that demonstrates the design of experiment (DOE) process.
Session Outline
The session can follow this flow:
Introduction to Robust Design
Robust design is about designing for the variability effects of noise. Noise can come from manufacturing variations, user operating differences, wear, or environmental conditions. The approach to robust design presented in this session is based on a method called design of experiments (DOE). To develop a robust product through DOE, we teach the following process:
We have found it very effective to illustrate the DOE process for PD through an inclass exercise or a homework assignment. Several DOE teaching exercises have been developed over the years, including paper helicopters, paper airplanes, catapults, etc. Instructions and equipment for the XPULT table tennis ball catapult DOE exercise developed at University of Pennsylvania are available online (xpult.com). Below are instructions for the paper airplane DOE exercise.
InClass Exercise
The goal of the exercise is to design a paper airplane that reliably flies as far as possible and can be folded and thrown by anyone. Using the L9 orthogonal array, an airplane is folded in 9 different ways, and a paper clip is attached along the bottom edge as a weight. (The varying location of the weight changes the location of the center of mass, and thus the distance between the center of mass and the lift force on the wing.)
The steps below can be followed in class:
Step #1: Identify the control factors, noise factors, and performance metrics
A product’s functional characteristics can be affected by two categories of factors: controllable factors (inputs) and uncontrollable factors (noise). The control factors are the design variables to be varied, and the following are used in the airplane DOE: (A) position of weight (B) stabilizer folds (C) nose distance and (D) wing angle.
The robustness requirement that it must be “folded and thrown by anyone” suggests that there are uncontrollable noise factors that must be considered. Having multiple students build and then throw identically specified airplanes simulates a number of noise factors (accuracy of printing, folding, paper variations, throwing method, etc).
The performance metric is the total distance traveled during a single flight.
Step #2: Formulate an objective function
The performance metric (distance of flight) must be transformed into an objective function that relates to the desired robust performance. The following objective functions will be used:
The objective functions will be discussed in more detail in Step #5.
Step #3: Develop the experimental plan
The experimental plan consists of a design of experiment (DOE) that has four factors, each with three levels. Thus if all combinations of factors and levels were tested 34 = 81 would be required (full factorial). Furthermore, repeating these experiments five times to understand variations (noise) would require 81*5 = 405 trials. This exercise will instead use a carefully defined subset of experiments that is as small as possible while still identifying the main effects of each factor (L9 orthogonal array). This design consists of 9 experiments, which are listed in the table below as separate rows.
Table 1: L9 Orthogonal array of experiments for the paper airplane DOE.
While it is beneficial to consider product robustness as early as the concept stage, experiments for robust design are used most frequently during the detailed design phase as a way to ensure the desired product performance under a variety of conditions. Thus the topic is usually introduced later in the semester.
Objectives and Strategy
The process of Robust Design should enable organizations to design products that operate within the performance specifications under the effects of noise. Learning the principles of robust design is accomplished using an inclass exercise that demonstrates the design of experiment (DOE) process.
Session Outline
The session can follow this flow:
 Introduction to Robust Design
 Inclass exercise
Introduction to Robust Design
Robust design is about designing for the variability effects of noise. Noise can come from manufacturing variations, user operating differences, wear, or environmental conditions. The approach to robust design presented in this session is based on a method called design of experiments (DOE). To develop a robust product through DOE, we teach the following process:
 Identify control factors, noise factors, and performance metrics
 Formulate an objective function
 Develop the experimental plan
 Run the experiment
 Conduct the analysis
 Select and confirm factor setpoints
 Reflect and repeat.
We have found it very effective to illustrate the DOE process for PD through an inclass exercise or a homework assignment. Several DOE teaching exercises have been developed over the years, including paper helicopters, paper airplanes, catapults, etc. Instructions and equipment for the XPULT table tennis ball catapult DOE exercise developed at University of Pennsylvania are available online (xpult.com). Below are instructions for the paper airplane DOE exercise.
InClass Exercise
The goal of the exercise is to design a paper airplane that reliably flies as far as possible and can be folded and thrown by anyone. Using the L9 orthogonal array, an airplane is folded in 9 different ways, and a paper clip is attached along the bottom edge as a weight. (The varying location of the weight changes the location of the center of mass, and thus the distance between the center of mass and the lift force on the wing.)
The steps below can be followed in class:
Step #1: Identify the control factors, noise factors, and performance metrics
A product’s functional characteristics can be affected by two categories of factors: controllable factors (inputs) and uncontrollable factors (noise). The control factors are the design variables to be varied, and the following are used in the airplane DOE: (A) position of weight (B) stabilizer folds (C) nose distance and (D) wing angle.
The robustness requirement that it must be “folded and thrown by anyone” suggests that there are uncontrollable noise factors that must be considered. Having multiple students build and then throw identically specified airplanes simulates a number of noise factors (accuracy of printing, folding, paper variations, throwing method, etc).
The performance metric is the total distance traveled during a single flight.
Step #2: Formulate an objective function
The performance metric (distance of flight) must be transformed into an objective function that relates to the desired robust performance. The following objective functions will be used:
 Mean distance of flight: a measurement of performance
 Variation in the data: a measurement of the robustness
 Signal to noise ratio: a measurement of the robustness
The objective functions will be discussed in more detail in Step #5.
Step #3: Develop the experimental plan
The experimental plan consists of a design of experiment (DOE) that has four factors, each with three levels. Thus if all combinations of factors and levels were tested 34 = 81 would be required (full factorial). Furthermore, repeating these experiments five times to understand variations (noise) would require 81*5 = 405 trials. This exercise will instead use a carefully defined subset of experiments that is as small as possible while still identifying the main effects of each factor (L9 orthogonal array). This design consists of 9 experiments, which are listed in the table below as separate rows.
Table 1: L9 Orthogonal array of experiments for the paper airplane DOE.
Experiment #

A: weight dist.

B: stabilizer

C. nose length

D: wing angle

1

A1

B1

C1

D1

2

A1

B2

C2

D2

3

A1

B3

C3

D3

4

A2

B1

C2

D3

5

A2

B2

C3

D1

6

A2

B3

C1

D2

7

A3

B1

C3

D2

8

A3

B2

C1

D3

9

A3

B3

C2

D1

Step #4: Run the experiment
Each student is given a printed template for an airplane (LINK) and a paperclip. Assign each student an experiment number (1 through 9, see Table 1 above), and instruct them to fold the airplane using their assigned set of control factors.
At the end of a hallway or long room (at least 30 feet), place a strip of masking tape on the floor to designate the launching line. Perpendicular to this, lay a 30foot strip of tape on the floor, numbered in one foot increments starting at the launch line.
Have each student individually throw his or her folded airplane from the launch line. Record the flight distance in this spreadsheet (LINK) in the rows where it states “Enter Experimental Data Here”.
Step #5: Conduct the analysis
The same spreadsheet is used to analyze the data, and can be done in class in realtime since all of the calculations are built into the spreadsheet. The factor affects are analyzed using Analysis of Means. This method involves simply averaging all the computed objective functions for each factor level. First, the mean, variance, and signal to noise ratio are calculated using the data for all throws of a single experiment. Next, the experiments are grouped by factor levels, as shown in Table 2 in matrix form (example, A1 is used in experiment number 1, 2, and 3. D2 is used in experiment 2, 6, 7). The average of the mean, variance, and signal to noise ratio is automatically calculated for each factor level grouping, and plotted in separate sheets (these plots are called “factor effects charts”).
Table 2: The groupings of factor level experiments in matrix form.
Each student is given a printed template for an airplane (LINK) and a paperclip. Assign each student an experiment number (1 through 9, see Table 1 above), and instruct them to fold the airplane using their assigned set of control factors.
At the end of a hallway or long room (at least 30 feet), place a strip of masking tape on the floor to designate the launching line. Perpendicular to this, lay a 30foot strip of tape on the floor, numbered in one foot increments starting at the launch line.
Have each student individually throw his or her folded airplane from the launch line. Record the flight distance in this spreadsheet (LINK) in the rows where it states “Enter Experimental Data Here”.
Step #5: Conduct the analysis
The same spreadsheet is used to analyze the data, and can be done in class in realtime since all of the calculations are built into the spreadsheet. The factor affects are analyzed using Analysis of Means. This method involves simply averaging all the computed objective functions for each factor level. First, the mean, variance, and signal to noise ratio are calculated using the data for all throws of a single experiment. Next, the experiments are grouped by factor levels, as shown in Table 2 in matrix form (example, A1 is used in experiment number 1, 2, and 3. D2 is used in experiment 2, 6, 7). The average of the mean, variance, and signal to noise ratio is automatically calculated for each factor level grouping, and plotted in separate sheets (these plots are called “factor effects charts”).
Table 2: The groupings of factor level experiments in matrix form.
A

B

C

D

1

1, 2, 3

1, 4, 7

1, 6, 8

1, 5, 9

2

4, 5, 6

2, 5, 8

2, 4, 9

2, 6, 7

3

7, 8, 9

3, 6, 9

3, 5, 7

3, 4, 8

Step #6: Select and confirm factor setpoints
The factor effects charts help to identify which factors are best able to reduce the airplane’s variance (robustness factors) and which factors can be used to improve the performance (scaling factors). By choosing setpoints based on these insights, the team should be able to improve the overall robustness of the product. See exhibit 157 in the textbook.
Step #7: Reflect and repeat
View the results as a class and discuss the following:
(1) What can you say about the effects of the different factors on flight distance?
(2) What levels of each factor would you used to maximize flight distance?
(3) Why are there tradeoffs between variance (robustness) and performance?
(4) What can you say about the effects of the different factors on variation in flight distance? What levels of each factor would you used to minimize variation in the flight distance?
(5) What kinds of noise (variations) are considered in this experiment?
(6) How can interactions between factors be considered?
The factor effects charts help to identify which factors are best able to reduce the airplane’s variance (robustness factors) and which factors can be used to improve the performance (scaling factors). By choosing setpoints based on these insights, the team should be able to improve the overall robustness of the product. See exhibit 157 in the textbook.
Step #7: Reflect and repeat
View the results as a class and discuss the following:
(1) What can you say about the effects of the different factors on flight distance?
(2) What levels of each factor would you used to maximize flight distance?
(3) Why are there tradeoffs between variance (robustness) and performance?
(4) What can you say about the effects of the different factors on variation in flight distance? What levels of each factor would you used to minimize variation in the flight distance?
(5) What kinds of noise (variations) are considered in this experiment?
(6) How can interactions between factors be considered?