Example of the Performance of a Classification Technique
Category # Objects Correct Class Correct Class (%)
1 112 105 93.8
Total 220 201 91.4/80.1
example, a possible classification matrix is the one shown in Table 5.9. From it, it can be seen that the 112 objects of category one were classified in the following way: 105 correctly to category one, none to category two, and seven to category three. In the same way, it can be deduced that all the objects of category three which were not correctly classified have been assigned to category one. Therefore, it is easy to conclude that category two is well defined and that the classification of its objects gives no problems at all, while categories one and three are quite overlapping. As a consequence, to have a perfect classification more effort must be put into better separating categories one and three. All this information cannot be obtained from just the percentage of correct classifications.
If overfitting occurs, then the prediction ability will be much worse than the classification ability. To avoid it, it is very important that the sample size is adequate to the problem and to the technique. A general rule is that the number of objects should be more than five times (at least, no less than three times) the number of parameters to be estimated. LDA works on a pooled variance-covariance matrix: this means that the total number of objects should be at least five times the number of variables. QDA computes a variance-covariance matrix for each category, which makes it a more powerful method than LDA, but this also means that each category should have a number of objects at least five times higher than the number of variables. This is a good example of how the more complex, and therefore "better" methods, sometimes cannot be used in a safe way because their requirements do not correspond to the characteristics of the data set.
Was this article helpful?