A Comparison of Machine Learning with Human Judgment

Abstract:

This paper compares human judgment with machine learning in a check processing context. An experiment was conducted comparing teams of subjects with three commercially available machine learning algorithms: recursive partitioning, ID3, and a back-propagation neural network. Also, the statistical technique discriminant analysis was compared. The subjects were allowed to induce rules from historical data under ideal human conditions, such as adequate time and opportunity to sort the data as desired. The results on multiple holdout samples indicate that human judgment, recursive partitioning, and the ID3 algorithm were equally accurate and more accurate than a back-propagation neural network. Subjects who chose mixed strategies of judgment were more accurate than those using noncompensatory strategies, while no subjects chose compensatory strategies. Large decision trees were not more accurate than smaller ones. There appeared to be a time threshold for humans to form accurate decision rules. Holdout sample accuracy tended to increase with primary sample accuracy. 103 built larger trees than did either humans or recursive partitioning. The conclusion of this research is that the knowledge engineer faced with available historical data concerning a classification problem should not waste his time discerning rules, since he will only take longer and be no more accurate than a good learning tool. Knowledge of these tools will be the requisite skill for the knowledge engineer of the 1990s. Implications for IS design and further research are discussed.

Key Words and Phrases::

• artificial neural networks
• discriminant analysis
• expert systems
• ID3
• machine learning
• policy capture
• recursive partitioning

Additional information

Notes on contributors

Michael W. Kattan

Michael W. Kattan received his M.B.A. in 1989 from the University of Arkansas, and is a doctoral candidate in management information systems at the University of Houston. His primary research interests are in the areas of machine learning and expert systems. His articles have appeared in the Journal of Computer Information Systems, Computer Personnel, Proceedings of the Decision Sciences Institute, and Proceedings of the ACM Special Interest Group on Computer Personnel. Prior to entering the doctoral program, Mr. Kattan conducted research in the food science area, which he published in Cereal Chemistry and Journal of Food Science. He also has a paper forthcoming in Leukemia and Lymphoma.

Dennis A. Adams

Dennis A. Adams received his Ph.D. in 1987 from Texas Tech University. He is an Assistant Professor of Management Information Systems in the College of Business Administration at the University of Houston. His research interests include management of information communication technologies, the uses of information technology for organizational competitive advantage, and the analysis and design of parallel systems. He has published articles in journals such as Data Base, MIS Quarterly, Information and Management. Information Systems Research. Journal of General Management, and Advances in Accounting Information Systems.

Michael S. Parks

Michael S. Parks received his Ph.D. in 1973 from the University of Georgia. He is an Associate Professor of Management Information Systems at the University of Houston. His current research interests are natural intelligence and cybernetics. His work has been published in Cybernetica, Management Science, Kybernetes, and Operations Research Quarterly.