Sample Answers to Exercises and Thought Questions: Chapter 15
EX 15.1
Design an experiment to determine a robust process for making coffee.
The process of making coffee is composed of four basic steps; raw coffee beans must be roasted, the roasted coffee beans must then be ground, the ground coffee must then be mixed with hot water for a certain time (brewed), and finally the liquid coffee must be separated from the now used and unwanted grounds. I will assume that "making coffee" for this exercise only involves the grinding and brewing steps. Thus the bean has already been selected, roasted, etc. I will use the design of experiments (DOE) method described in the textbook, which follows the following process:
Each step is described in detail below.
Step #1: Identify the control factors, noise factors, and performance metrics
A product’s functional characteristics can be affected by two categories of factors: controllable factors (inputs) and uncontrollable factors (noise). The control factors are the design variables to be varied, and the following are used in this coffee DOE: (A) Coarseness of grinding (B) Amount of grounds for a set amount of water (C) temperature of water during brewing (D) duration of time for brewing.
The uncontrollable factors (noise) would include variations in measuring quantities of water and grounds, variations in operating procedures such as placing the grounds into the filter and how the filter is installed, different types of drip coffee machines and filters, variations in water quality, and variations in serving the coffee.
The performance metric is very subjective, since it largely depends on the individual. Thus, consumers from the target market (e.g. certain age or demographic) would be collected to "taste test" the various trials. Thus a subjective metric of 1 to 5 would be used, where 5 is the best tasting.
Step #2: Formulate an objective function
The performance metric must be transformed into an objective function that relates to the desired robust performance. The following objective functions will be used:
Step #3: Develop the experimental plan
I would start with an existing successful recipe for coffee, and deviate slightly from this (the normal recipe will be used as the level 2 values), with three levels total for each factor.
Design an experiment to determine a robust process for making coffee.
The process of making coffee is composed of four basic steps; raw coffee beans must be roasted, the roasted coffee beans must then be ground, the ground coffee must then be mixed with hot water for a certain time (brewed), and finally the liquid coffee must be separated from the now used and unwanted grounds. I will assume that "making coffee" for this exercise only involves the grinding and brewing steps. Thus the bean has already been selected, roasted, etc. I will use the design of experiments (DOE) method described in the textbook, which follows the following process:
- Identify control factors, noise factors, and performance metrics
- Formulate an objective function
- Develop the experimental plan
- Run the experiment
- Conduct the analysis
- Select and confirm factor setpoints
- Reflect and repeat
Each step is described in detail below.
Step #1: Identify the control factors, noise factors, and performance metrics
A product’s functional characteristics can be affected by two categories of factors: controllable factors (inputs) and uncontrollable factors (noise). The control factors are the design variables to be varied, and the following are used in this coffee DOE: (A) Coarseness of grinding (B) Amount of grounds for a set amount of water (C) temperature of water during brewing (D) duration of time for brewing.
The uncontrollable factors (noise) would include variations in measuring quantities of water and grounds, variations in operating procedures such as placing the grounds into the filter and how the filter is installed, different types of drip coffee machines and filters, variations in water quality, and variations in serving the coffee.
The performance metric is very subjective, since it largely depends on the individual. Thus, consumers from the target market (e.g. certain age or demographic) would be collected to "taste test" the various trials. Thus a subjective metric of 1 to 5 would be used, where 5 is the best tasting.
Step #2: Formulate an objective function
The performance metric must be transformed into an objective function that relates to the desired robust performance. The following objective functions will be used:
- Mean rating: a measurement of performance
- Variation in the data: a measurement of the robustness
- Signal to noise ratio: a measurement of the robustness
Step #3: Develop the experimental plan
I would start with an existing successful recipe for coffee, and deviate slightly from this (the normal recipe will be used as the level 2 values), with three levels total for each factor.
|
Factor
|
Levels
|
|
A. Coarseness of grinding
|
Fine, regular, coarse (grind)
|
|
B. Amount of grounds for a set amount of water
|
Low, medium, high (quantity)
|
|
C. Temperature of water during brewing
|
Hot, hotter, hottest (temperature)
|
|
D. Duration of time for brewing
|
Short, normal, long (time)
|
The coffee machine operator will measure out the prescribed quantity of beans, and then set the coffee machine to grind and brew the coffee. The paper filter is replaced after each experiment. The temperature and times are set digitally by the operator and controlled by the machine.
The experimental plan consists of a design of experiment (DOE) that has four factors, each with three levels. Thus if all combinations of factors and levels were tested 34 = 81 would be required (full factorial). Furthermore, repeating these experiments five times to understand variations (noise) would require 81*5 = 405 trials. This exercise will instead use a carefully defined subset of experiments that is as small as possible while still identifying the main effects of each factor (L9 orthogonal array). This design consists of 9 experiments, which are listed in the table below as separate rows.
The experimental plan consists of a design of experiment (DOE) that has four factors, each with three levels. Thus if all combinations of factors and levels were tested 34 = 81 would be required (full factorial). Furthermore, repeating these experiments five times to understand variations (noise) would require 81*5 = 405 trials. This exercise will instead use a carefully defined subset of experiments that is as small as possible while still identifying the main effects of each factor (L9 orthogonal array). This design consists of 9 experiments, which are listed in the table below as separate rows.
Table 1: L9 Orthogonal array of experiments for the paper airplane DOE.
|
Experiment #
|
A: grind
|
B: quantity
|
C: temperature
|
D: time
|
|
1
|
A1, fine
|
B1, low
|
C1, hot
|
D1, short
|
|
2
|
A1, fine
|
B2, medium
|
C2, hotter
|
D2, normal
|
|
3
|
A1, fine
|
B3, high
|
C3, hottest
|
D3, long
|
|
4
|
A2, regular
|
B1, low
|
C2, hotter
|
D3, long
|
|
5
|
A2, regular
|
B2, medium
|
C3, hottest
|
D1, short
|
|
6
|
A2, regular
|
B3, high
|
C1, hot
|
D2, normal
|
|
7
|
A3, coarse
|
B1, low
|
C3, hottest
|
D2, normal
|
|
8
|
A3, coarse
|
B2, medium
|
C1, hot
|
D3, long
|
|
9
|
A3, coarse
|
B3, high
|
C2, hotter
|
D1, short
|
Step #4: Run the experiment
To induce the uncontrollable factors (noise), five different coffee machines and operators will be used to run each experiment randomly according to Table 1.
A random selection of consumers will be chosen from the target market. The consumers will try each experiment trial, but from a random operator (thus they won't have the same operator each time).
Step #5: Conduct the analysis
The factor affects will be analyzed using Analysis of Means. This method involves simply averaging all the computed objective functions for each factor level. First, the mean, variance, and signal to noise ratio are calculated using the data for all trials of a single experiment. Next, the experiments are grouped by factor levels, as shown in Table 2 in matrix form (example, A1 is used in experiment number 1, 2, and 3. D2 is used in experiment 2, 6, 7).
To induce the uncontrollable factors (noise), five different coffee machines and operators will be used to run each experiment randomly according to Table 1.
A random selection of consumers will be chosen from the target market. The consumers will try each experiment trial, but from a random operator (thus they won't have the same operator each time).
Step #5: Conduct the analysis
The factor affects will be analyzed using Analysis of Means. This method involves simply averaging all the computed objective functions for each factor level. First, the mean, variance, and signal to noise ratio are calculated using the data for all trials of a single experiment. Next, the experiments are grouped by factor levels, as shown in Table 2 in matrix form (example, A1 is used in experiment number 1, 2, and 3. D2 is used in experiment 2, 6, 7).
Table 2: The groupings of factor level experiments in matrix form.
|
A
|
B
|
C
|
D
|
|
1
|
1, 2, 3
|
1, 4, 7
|
1, 6, 8
|
1, 5, 9
|
|
2
|
4, 5, 6
|
2, 5, 8
|
2, 4, 9
|
2, 6, 7
|
|
3
|
7, 8, 9
|
3, 6, 9
|
3, 5, 7
|
3, 4, 8
|
Step #6: Select and confirm factor setpoints
The factor effects charts help to identify which factors are best able to reduce the variance (robustness factors) and which factors can be used to improve the performance (scaling factors). By choosing setpoints based on these insights, the team should be able to improve the overall robustness of the product. My decision would depend on the data, as discussed in exhibit 15-7 in the textbook.
Step #7: Reflect and repeat
Below are a few reflection questions that would be asked once the data is collected and analyzed:
EX 15.2
Explain why the 1/4-fractional-factorial and orthogonal array plans shown in Exhibit 15-4 are balanced.
They are balanced because each factor is tested at every level the same number of times, during which each other factor is tested the same number of times at each level. For example, A1 is used half the time, while factors B1, C1, and D1 are all used in half of the A1 trials. This balance is true for each factor level.
EX 15.3
Formulate an appropriate signal-to-noise ratio for the seat belt experiment. Analyze the experimental data using this metric. Is signal-to-noise ratio a useful objective function in this case? Why or why not?
The mean and variance is often combined and expressed as a single objective in the form of a signal-to-noise ratio. Typically the signal-to-noise ratio is maximized, and the following function is used: s/n = 10*log(mean2/variance). However, in the case of the seatbelt experiment, the measurement of back angle is a smaller-is-better metric and only the range is known (not the variance). To maintain a maximization, I inverted the mean and divided by the range: s/n = 10*log((1/mean)2/range2)). However, the Ford team chose to utilize the compounded noise approach, which is when selected noise factors are combined to create several representative or extreme noise conditions. Thus the team tested each row using the two combinations of the three noise factors representing the best- (N+) and worst-case (N-) conditions. To combine both noise values, I wrote the metric as: s/n = 10*LOG((1/N-+1/N+)2/(N--N+)2).
I calculated the s/n for each experiment (Table 3), and the average of the s/n was calculated using the data for the factor level experiments (Table 4). The metric should be useful in quickly seeing that a high s/n relates to factor levels with the desirable combination of both low back angle and variance (range). This is true in all but case B, where the s/n is shown high for the clearly undesirable case when the back angle and variance are both larger. This result shows weakness in the s/n metric I am suggesting. If the setpoint was chosen purely based on which s/n was greater, then the following would be selected: [A2, B1, C2, D1, E1, F2, G1]. The setpoints shown in bold conflict with the actual selection made by Ford. Ford made various tradeoffs between performance and variance when making the final decision. Factors B, C, D, E, and G were chosen to achieve the desired robustness and factors A and F to increase performance. I'm concluding that it is not appropriate to only use the s/n as the deciding factor. Instead, the final selection of setpoints should be made by analyzing the Analysis of Means plots (average, range and s/n), and then making case-by-case tradeoffs using additional engineering knowledge and judgment.
The factor effects charts help to identify which factors are best able to reduce the variance (robustness factors) and which factors can be used to improve the performance (scaling factors). By choosing setpoints based on these insights, the team should be able to improve the overall robustness of the product. My decision would depend on the data, as discussed in exhibit 15-7 in the textbook.
Step #7: Reflect and repeat
Below are a few reflection questions that would be asked once the data is collected and analyzed:
- What can you say about the effects of the different factors on the coffee taste?
- What levels of each factor would you used to produce the best coffee?
- Why are there tradeoffs between variance (robustness) and performance?
EX 15.2
Explain why the 1/4-fractional-factorial and orthogonal array plans shown in Exhibit 15-4 are balanced.
They are balanced because each factor is tested at every level the same number of times, during which each other factor is tested the same number of times at each level. For example, A1 is used half the time, while factors B1, C1, and D1 are all used in half of the A1 trials. This balance is true for each factor level.
EX 15.3
Formulate an appropriate signal-to-noise ratio for the seat belt experiment. Analyze the experimental data using this metric. Is signal-to-noise ratio a useful objective function in this case? Why or why not?
The mean and variance is often combined and expressed as a single objective in the form of a signal-to-noise ratio. Typically the signal-to-noise ratio is maximized, and the following function is used: s/n = 10*log(mean2/variance). However, in the case of the seatbelt experiment, the measurement of back angle is a smaller-is-better metric and only the range is known (not the variance). To maintain a maximization, I inverted the mean and divided by the range: s/n = 10*log((1/mean)2/range2)). However, the Ford team chose to utilize the compounded noise approach, which is when selected noise factors are combined to create several representative or extreme noise conditions. Thus the team tested each row using the two combinations of the three noise factors representing the best- (N+) and worst-case (N-) conditions. To combine both noise values, I wrote the metric as: s/n = 10*LOG((1/N-+1/N+)2/(N--N+)2).
I calculated the s/n for each experiment (Table 3), and the average of the s/n was calculated using the data for the factor level experiments (Table 4). The metric should be useful in quickly seeing that a high s/n relates to factor levels with the desirable combination of both low back angle and variance (range). This is true in all but case B, where the s/n is shown high for the clearly undesirable case when the back angle and variance are both larger. This result shows weakness in the s/n metric I am suggesting. If the setpoint was chosen purely based on which s/n was greater, then the following would be selected: [A2, B1, C2, D1, E1, F2, G1]. The setpoints shown in bold conflict with the actual selection made by Ford. Ford made various tradeoffs between performance and variance when making the final decision. Factors B, C, D, E, and G were chosen to achieve the desired robustness and factors A and F to increase performance. I'm concluding that it is not appropriate to only use the s/n as the deciding factor. Instead, the final selection of setpoints should be made by analyzing the Analysis of Means plots (average, range and s/n), and then making case-by-case tradeoffs using additional engineering knowledge and judgment.
Table 3: DOE results.
|
A
|
B
|
C
|
D
|
E
|
F
|
G
|
N-
|
N+
|
Avg
|
Range
|
|
1
|
1
|
1
|
1
|
1
|
1
|
1
|
1
|
0.3403
|
0.2915
|
0.3159
|
0.0488
|
|
2
|
1
|
1
|
1
|
2
|
2
|
2
|
2
|
0.4608
|
0.3984
|
0.4296
|
0.0624
|
|
3
|
1
|
2
|
2
|
1
|
1
|
2
|
2
|
0.3682
|
0.3627
|
0.3655
|
0.0055
|
|
4
|
1
|
2
|
2
|
2
|
2
|
1
|
1
|
0.2961
|
0.2647
|
0.2804
|
0.0314
|
|
5
|
2
|
1
|
2
|
1
|
2
|
1
|
2
|
0.4450
|
0.4398
|
0.4424
|
0.0052
|
|
6
|
2
|
1
|
2
|
2
|
1
|
2
|
1
|
0.3517
|
0.3538
|
0.3528
|
0.0021
|
|
7
|
2
|
2
|
1
|
1
|
2
|
2
|
1
|
0.3758
|
0.3580
|
0.3669
|
0.0178
|
|
8
|
2
|
2
|
1
|
2
|
1
|
1
|
2
|
0.4504
|
0.4076
|
0.4290
|
0.0428
|
Table 4. The groupings of factor level experiments in matrix form.
|
A
|
B
|
C
|
D
|
E
|
F
|
G
|
|
1
|
1, 2, 3, 4
|
1, 2, 5, 6
|
1, 2, 7, 8
|
1, 3, 5, 7
|
1, 3, 6, 8
|
1, 4, 5, 8
|
1, 4, 6, 7
|
|
2
|
5, 6, 7, 8
|
3, 4, 7, 8
|
3, 4, 5, 6,
|
2, 4, 6, 8
|
2, 4, 5, 7
|
2, 3, 6, 7
|
2, 3, 5, 8
|
TQ 15.1
If you are able to afford a larger experiment (with more runs), how might you best utilize the additional runs?
Additional runs could be used to gain additional insight in a number of ways:
TQ 15.2
When would you choose not to randomize the order of the experiments? How would you guard against bias?
Randomizing the sequence of experimental runs ensures that any systematic trend over the duration of the experiment is not correlated with the systematic changes to the levels of any factors. For example, the temperature of the room might slowly increase throughout the day, and the effects of this might be incorrectly attributed to the changes in control factor A since this factor (typically) moves through its levels once over the non-randomized sequence of the DOE.
However, for some experiments, changing certain factors may be so difficult that all trials at each level of that factor are run together and only partially randomization may be achieved.
One way to guard against this type of bias is by running an additional experiment at the end using the selected setpoints, to ensure that the performance is still superior. This is referred to as a "confirmation run".
TQ 15.3
Explain the importance of balance in an experimental plan?
In a balanced experiment, each factor level is used the same number of times. This not only facilitates analysis of means (using simple averaging, rather than weighted averages), but it also assures that the experimental space is explored in a balanced manner. An unbalanced experiment would explore one portion of the space more completely and other areas less so. For example, a center-weighted experiment (moving one factor at a time away from the center point) does not explore the edges of the space at all.
If you are able to afford a larger experiment (with more runs), how might you best utilize the additional runs?
Additional runs could be used to gain additional insight in a number of ways:
- Consider more control factors. This allows us to understand the impact of additional parameters under our control.
- Conduct trials at more levels of each control factor. This would allow more precision in tuning each control factor.
- Increase the number of trials in the DOE by using a larger fraction of the full factorial. This would allow consideration of some interaction effects.
- Conduct trials at more levels, types, or combinations or noise. This would help us understand the effects of noise.
TQ 15.2
When would you choose not to randomize the order of the experiments? How would you guard against bias?
Randomizing the sequence of experimental runs ensures that any systematic trend over the duration of the experiment is not correlated with the systematic changes to the levels of any factors. For example, the temperature of the room might slowly increase throughout the day, and the effects of this might be incorrectly attributed to the changes in control factor A since this factor (typically) moves through its levels once over the non-randomized sequence of the DOE.
However, for some experiments, changing certain factors may be so difficult that all trials at each level of that factor are run together and only partially randomization may be achieved.
One way to guard against this type of bias is by running an additional experiment at the end using the selected setpoints, to ensure that the performance is still superior. This is referred to as a "confirmation run".
TQ 15.3
Explain the importance of balance in an experimental plan?
In a balanced experiment, each factor level is used the same number of times. This not only facilitates analysis of means (using simple averaging, rather than weighted averages), but it also assures that the experimental space is explored in a balanced manner. An unbalanced experiment would explore one portion of the space more completely and other areas less so. For example, a center-weighted experiment (moving one factor at a time away from the center point) does not explore the edges of the space at all.