Statistical Power analysis is a critical part of designing experiments. It lets you balance the cost of an experiment with the anticipated value of the results. The R language has a module, pwr, which you can use to model these trade-offs.
To introduce the topic, real world experiments are a balancing act. Every experiment involves selecting a combination of the following three factors.
- Cost – often driven by sample size
- Ability to detect small effects
- Confidence that the outcome reflects the underlying process
These three factors give rise to a fourth: power.
The statistical power of an experiment represents the probability of identifying an effect that is present in the population you are sampling.
Power Analysis – The pwr package
We can use the pwr package to perform statistical power analysis in R.
This package has statistical power calculations for many experiment types. These have a common approach: enter three of the four parameters above (sample size, effect size, statistical significance, and power) and the package will calculate the forth.
The package has some defaults. The significance level is defaulted to a=0.05.
There is no simple answer to the question selecting a desired effect size. For those of you working in industry, there is an obvious solution: consult with your financial team and executive sponsors to identify what level of result is financially meaningful. For example, a 2% reduction in manufacturing waste could be perceived as a significant contribution to the organization. Whereas we might need a 15% to 20% lift in customer satisfaction to justify adding new features to a product.
Statistical Power Analysis Options
R’s pwr package supports the following:
|pwr.f2.test||General Linear model||pwr.f2.test(u =, v = , f2 = ,|
sig.level = , power = )
|pwr.r.test||Correlation Analysis||pwr.r.test(n = , r = , |
sig.level = , power = )
|pwr.anova.test||One Way ANOVA||pwr.anova.test(k = , n = , f = , sig.level = , power = )|
|pwr.chisq.test||chi-square test||pwr.chisq.test(w =, N = , df = , sig.level =, power = )|
|pwr.t.test||one sample t-test|| pwr.t.test(n = , d = , sig.level = , |
power = , type = c(“two.sample”, “one.sample”, “paired”))
|pwr.t2n.test||two sample t-test||pwr.t2n.test(n1 = , n2= , d = , sig.level =, power = )|
|pwr.p.test||proportions (one sample)||pwr.p.test(h = , n = , sig.level = power = )|
|pwr.2p.test||proportions (two samples, equal n)||pwr.2p.test(h = , n = , sig.level =, power = )|
proportions (two samples, different n)
|pwr.2p2n.test(h = , n1 = , n2 = , sig.level = , power = )|
# power analysis in r example library(pwr) # Election Example. Interpreting a poll result. # Assume a significance level of .05 and a sample of # 1000 voters. What effect size can be detected with # a power of .5? > pwr.2p.test(n=1000,sig.level=0.05,power=0.5) Difference of proportion power calculation for binomial distribution (arcsine transformation) h = 0.08764393 n = 1000 sig.level = 0.05 power = 0.5 alternative = two.sided NOTE: same sample sizes
For a second example, let us assume we’re trying to measure public response to an issue; a single sample test of proportions would be appropriate here.
# power analysis in r example > pwr.p.test(n=1000,sig.level=0.05,power=0.5) proportion power calculation for binomial distribution (arcsine transformation) h = 0.06196988 n = 1000 sig.level = 0.05 power = 0.5 alternative = two.sided
Which can be improved upon by the simple act of boosting the sample size.
# power analysis in r example > pwr.p.test(n=5000,sig.level=0.05,power=0.5) proportion power calculation for binomial distribution (arcsine transformation) h = 0.02771587 n = 5000 sig.level = 0.05 power = 0.5 alternative = two.sided