Week 14: Prediction

Prediction in MLM

Week Learning Objectives

By the end of this module, you will be able to

• Describe the role of prediction in data analysis
• Describe the problem of overfitting when fitting complex models
• Use information criteria to compare models

Task List

1. Review the resources (lecture videos and slides)
2. Attend the Tuesday session and participate in the bonus class exercise
3. Attend the Thursday session for class presentations

Lecture

Slides

PDF version

Predictive Models in MLM

Think more

Think about a prominent theory in your area of research. What predictions does it make? Does it give precise predictions?

Example

Check your learning

In a multilevel model with students nested within schools and with student math achievement as the outcome variable, what is a cluster-specific prediction?

The predicted achievement for a student from one of the schools included in the sample.
The predicted mean achievement of a new school.
The predicted achievement for a student from a new school that are not part of the sample.

Prediction Error

Some information has been updated since the video was recorded. Check out the updated slides

Overfitting

Check your learning

Which of the following growth curve model would show the largest degree of overfitting, given a sample of 15 participants across 5 time points?

A linear growth model.
A piecewise growth model with two phases.
A piecewise growth model with four phases.

Out-Of-Sample Prediction Errors

Check your learning

Why shouldn’t we just choose a model with the lowest in-sample prediction error?

Prediction error should not be a criterion for choosing among models.
One should use a likelihood ratio test instead.
A model that fits one data set too well usually does not fit well with other data sets.

Cross Validation

Check your learning

Why does cross-validation, compared to in-sample MSE, give a better estimate of the out-of-sample prediction error?

It uses different subsets of the data for estimation and for prediction.
It is developed in the machine learning literature.
It uses less data for estimation, so the results are more stable.

Information Criterion

Check your learning

Which of the following two models are nested?

M1: mathach ~ meanses + ses_cmc + sector + (1 | ID)

M2: mathach ~ sector + (1 | ID)

M3: mathach ~ meanses + sector + (ses_cmc | ID)

M1 and M2
M2 and M3
M1 and M3

Model Comparison

Check your learning

Which of the following model is the best based on AIC?

M1: mAIC = 1203, cAIC = 1037

M2: mAIC = 1202, cAIC = 1000

M3: mAIC = 1210, cAIC = 1055

M1 M2 M3