Statistical Power analysis is a critical part of designing a study or experiment. It lets you balance the cost of an experiment with the anticipated value of the results. The R language has a module, pwr, which you can use to model these trade-offs in a simulated data model called a power simulation.
To introduce the topic, real world experiments are a balancing act. Every experiment involves selecting a combination of the following three factors.
- Cost – often driven by required sample size
- Ability to detect small effects
- Confidence that the outcome reflects the underlying process
These three factors give rise to a fourth: power.
The statistical power of an experiment represents the probability of identifying an interaction effect on the dependent variable that is present in the population you are sampling.
Power Analysis – The pwr package
We can use the pwr package to perform statistical power analysis in R.
This package has statistical power analyses for many experiment or study types. These have a common approach: enter three of the four parameter options above (sample size, effect size, statistical significance, and power) and the package will calculate the fourth parameter.
The package has some defaults. The significance level is defaulted to a=0.05.
There is no simple answer to the question selecting a desired true effect size. For those of you working in industry, there is an obvious solution: consult with your financial team and executive sponsors to identify what level of result is a financially meaningful factor, or a statistically significant result. For example, a 2% reduction in manufacturing waste could be perceived as a significant contribution to the organization. Whereas we might need a 15% to 20% lift in customer satisfaction to justify adding new features to a product.
A statistical test like power analyses can help you make multiple comparisons across simulated data sets, finding the true effect of the hypothesis on the dependent variable and other statistical test measures. Even across unequal sample sizes, you can measure the mean, standard deviation, and confidence interval of the desired interaction effect, and perform a hypothesis test against the null hypothesis as if all of the simulated data had an equal sample size.
Statistical Power Analysis Options
R’s pwr package supports the following:
|pwr.f2.test||General Linear model||pwr.f2.test(u =, v = , f2 = ,|
sig.level = , power = )
|pwr.r.test||Correlation Analysis||pwr.r.test(n = , r = , |
sig.level = , power = )
|pwr.anova.test||One Way ANOVA||pwr.anova.test(k = , n = , f = , sig.level = , power = )|
|pwr.chisq.test||chi-square test||pwr.chisq.test(w =, N = , df = , sig.level =, power = )|
|pwr.t.test||one sample t-test|| pwr.t.test(n = , d = , sig.level = , |
power = , type = c(“two.sample”, “one.sample”, “paired”))
|pwr.t2n.test||two sample t-test||pwr.t2n.test(n1 = , n2= , d = , sig.level =, power = )|
|pwr.p.test||proportions (one sample)||pwr.p.test(h = , n = , sig.level = power = )|
|pwr.2p.test||proportions (two samples, equal n)||pwr.2p.test(h = , n = , sig.level =, power = )|
proportions (two samples, different n)
|pwr.2p2n.test(h = , n1 = , n2 = , sig.level = , power = )|
# power analysis in r example library(pwr) # Election Example. Interpreting a poll result. # Assume a significance level of .05 and a sample of # 1000 voters. What effect size can be detected with # a power of .5? > pwr.2p.test(n=1000,sig.level=0.05,power=0.5) Difference of proportion power calculation test statistic for binomial distribution (arcsine transformation) h = 0.08764393 n = 1000 sig.level = 0.05 power = 0.5 alternative = two.sided NOTE: equal sample size
For a second example, let us assume we’re trying to measure public response to an issue; a single sample test statistic of proportions would be appropriate here.
# power analysis in r example > pwr.p.test(n=1000,sig.level=0.05,power=0.5) proportion power calculation for binomial distribution (arcsine transformation) h = 0.06196988 n = 1000 sig.level = 0.05 power = 0.5 alternative = two.sided
Which can be improved upon by the simple act of boosting the required sample size.
# power analysis in r example > pwr.p.test(n=5000,sig.level=0.05,power=0.5) proportion power calculation for binomial distribution (arcsine transformation) h = 0.02771587 n = 5000 sig.level = 0.05 power = 0.5 alternative = two.sided