[S] 1SE rule

Therneau, Terry M., Ph.D. (therneau@mayo.edu)
Thu, 1 Oct 1998 09:13:10 -0500


My paper with Beth should be more explicit that "Breiman et al recommend
the 1-SE rule, and show that it is successful in screening out noise
variables ....". Much of the methodology in the rpart routines is based
on their book and I got tired of prefacing every sentence.
If several trees have a cross-validated error within .1 of each other say,
and the se of that estimate is .2, then I think that we would all agree that
the evidence for any one of the models vs the others is slight at best.
For presentation purposes the principle of Occam's razor --- simplest model
consistent with the data --- suggests that one use the smallest of the
models.
For "automatic" model selection I would have to agree with Brian. For
prediction the larger models work better. But the best rule is problem
specific. The Poisson deviance, for instance, does not make a good measure
of fit for cross-validation, it has a bias towards pruning off any node with
a low event rate and thus favors trees that are too small. Any rule that
further shrinks the tree, such as the 1se rule, worsens the situation. Thus
for survival trees the table of xval errors must be read with scepticism.

I started looking at this once but then got sidetracked to other things;
preliminary work suggested that a variance stabilizing transform was a
better way to go. Let y_i = event/time for the "left out" data point and
yhat_i the prediction from a tree fit without observation i, and use
f(sqrt(y_i) - sqrt(yhat_i)) as the meaure of prediction error, where
f(x) = x^2 or abs(x). This is fairly easy to explore using the xpred.rpart
routine, which returns the matrix of yhat values, one row per obs and one
column for each amount of pruning. If this can be shown to work better
across several data sets it should perhaps become the default xval calculation
for the Poisson case.

rfit <- rpart(Surv(time, status) ~ x1 +x2 + x3, mydata)
yy <- mydata$status/mydata$time
yhat <- xpred.rpart(rfit)

new.error <- apply((sqrt(yy)-sqrt(yhat))^2, 2, sum)

Terry Therneau
Z
-----------------------------------------------------------------------
This message was distributed by s-news@wubios.wustl.edu. To unsubscribe
send e-mail to s-news-request@wubios.wustl.edu with the BODY of the
message: unsubscribe s-news