Pedro Domingos
Artificial Intelligence Group
Instituto Superior Técnico
Av. Rovisco Pais
Lisbon 1049-001, Portugal
E-mail: pedrod@gia.ist.utl.pt
Phone: +351-1-841-7479 / 7269
Fax: +351-1-841-7472
Methods to avoid overfitting fall into two broad categories: data-oriented (using separate data for validation) and representation-oriented (penalizing complexity in the model). Both have limitations that are hard to overcome. We argue that fully adequate model evaluation is only possible if the search process by which models are obtained is also taken into account. To this end, we recently proposed a method for process-oriented evaluation (POE), and successfully applied it to rule induction (Domingos, 1998). However, for the sake of simplicity this treatment made two rather artificial assumptions. In this paper the assumptions are removed, and a simple formula for model evaluation is obtained. Empirical trials show the new, better-founded form of POE to be as accurate as the previous one, while further reducing theory sizes.
Model selection, overfitting, error estimation, rule induction, classification.