Second International ICSC Congress on Computational Intelligence:
Methods & Applications, June 19th - 22nd, 2001, Bangor, Wales, UK.
Computational Models: How do we know if they are any good?Vincent C. Arena, Wei Li, Sati Mazumdar, Nancy B. SussmanComputational models are used in human health risk assessment to predict the biological activity of potentially health-threatening agents. Structure-activity relationships (SAR) that relate chemical structure to biological activity are examples of such models. Once a model is developed, it is used to predict the activity of new chemicals. The success or failure of an SAR model lies in its ability to correctly predict the true activity of a chemical. The validation of an SAR model is traditionally based on a single empirical data set where the prediction measures may only be germane to the individual data set and do not represent the true performance of the SAR modeling approach. This paper presents a simulation-based scheme to address this problem. Data are simulated, mimicking chemicals with a set of physico-chemical features and biological status (active or inactive) spanning over a broad range of different associations between covariates and outcome with varying signal strengths that might potentially be found in real world data. A validation procedure is then used to estimate the prediction measures of an SAR model. By repeating this process, the empirical distributions of the prediction measures are obtained which provide information about the "goodness" of the model. Prediction measures used in this paper are sensitivity, specificity, and accuracy. The simulation-based scheme allows various forms for the underlying relationship between physico-chemical features and biological activity. Thus, the prediction performance of a modeling approach is evaluated under a broad range of data and not just that found in a single empirical data set. We illustrate this scheme for SAR modeling approaches (e.g., traditional, bagging, and hierarchal Bayes modeling) and different validation procedures (e.g., train-and-test, k-fold cross validation) with flow diagrams. |