Week 7: Sample Size

Sample Size Planning (Guest lecture by Winnie Tse)

Week Learning Objectives

By the end of this module, you will be able to

  • Describe the importance of having a sufficient sample size for scientific research
  • Describe conceptually the steps for sample size planning: precision analysis and power analysis
  • Perform power analysis for MLM using the PowerUpR application and the simr package
  • Understand the effect of uncertainty in parameter values and explore alternative approaches for sample size planning

Task List

  1. Review the resources (lecture videos and slides)
  2. Complete the assigned readings
    • Snijders & Bosker ch 11
  3. Attend the optional Tuesday session to learn about simulation in R
  4. Attend the Thursday session and participate in the class exercise
  5. Complete Homework 6
  6. Think about your project—Prospectus will be due in two weeks

Lecture

Slides

PDF version

Test yourself

Quiz on null hypothesis significance testing. We will cover the quiz solution in class.

Quiz pdf here

Supplemental reading

  • Open Science Framework (OSF) provides some general guidelines for pre-registering a study.

  • Kirley et al. (2020) provided a template for studies using one type of multilevel design—experience sampling methods, which is a type of intensive longitudinal method that collects participants’ self-reports of daily thoughts, emotions, and behaviors.

Practice yourself

Find the required \(J\) for \(SE(\gamma_{10})\) to be \(\leq .1\), when the treatment variable is at level-1 (i.e., \(X\)), with 10 individuals per cluster, ICC = \(\tau^2_0 = 0.3\), \(\sigma^2\) = 0.7, and \(\tau^2_1\) = 0.1.

Check your learning
In the examples in the video, what would change if the researcher assumes a larger effect under the alternative hypothesis?



Tools

PowerUpR Demo

Practice yourself

Using PwerUpR, verify that power = .678 when the treatment variable is at level 1 (i.e., Two-level MS-IRT, random treatment), with an average effect of 0.3 (standardized), no covariate, ICC = 0.3, \(\tau^2_1 = 0.15\) (i.e., omega2 = 0.5 in the program), with 30 clusters and 10 observations per cluster.

Now, include one covariate that has an \(R^2\) of .40. What is the power?

Check your learning
In a pilot study, we found \(\delta = .2\), with a standard error of \(.05\), and ICC = .1, with a standard error of .05. We use the information \(\delta = .2\) and ICC = .1 to determine the number of clusters, and the program suggests us to have 116 clusters. If we employ 116 clusters in our study, what will happen to the power?



hcbr Demo

One advantage of this program is it analytically solves for the number of clusters or cluster size you need. Using hcbr, you do not need to iteratively test different \(J\) and \(n\) until the power achieves 80%; you get the answer directly from the program.

Practice yourself

Given that \(\delta\) = .2 with a standard error of .05, and ICC = .1, with a standard error of .05. What is the required \(J\) if \(n\) = 2 and \(n\) = 20?

Notes on power analysis

Check your learning
A researcher originally planned to collect data from 30 classrooms with 5 students per classroom, but then found out the study was underpowered. He can collect data from 150 more students. Would it be more beneficial to make it 300 students from 30 classrooms or 300 students from 60 classrooms? Why?