Verification and validation (V & V) of knowledge remains a major bottleneck in developing knowledge-based software components and systems. Anecdotal reports on currently fielded systems as well as those under development describe efforts on the order of man-years to perform knowledge V & V. As part of the typical life-cycle of such knowledge-based components, validation is performed in the following fashion: (1) field testing, (2) both correct and incorrect results are collected, (3) these results and the knowledge-base are handed to the expert, (4) the expert repairs the knowledge-base appropriately, and then the whole precess is repeated again until the expert is satisfied with the results.
V & V for the knowledge-base is different from V & V for the software system. Knowledge-base V & V primarily addresses the question: ``Does my knowledge-base contain the right answer and can I arrive at it?'' This involves issues such as exploring the completeness of the knowledge-base, accuracy of information, and the more sophisticated problem of dynamic interactions between knowledge chunks in the decision-making process.
One of the major limitations of current approaches is in dealing with uncertainty (information confidence): uncertainty in the expert's themselves about their knowledge, uncertainty in the engineer trying to translate the knowledge, and finally, uncertainty that is inherent in the domain itself. Although it seems that all knowledge can be encoded in logical ``if-then'' style rules exceptions to the rule quickly explode the number of rules necessary. Unfortunately, this further leads to questions on completeness. Unless all the exceptions have been identified, the original rule being as general as it is will produce an incorrect result in the situations where the un-identified exceptions occurs.
This tutorial covers the fundamental problems and approaches of uncertainty and considers methods in handling uncertainty during V & V. Example frameworks for uncertainty include Bayesian networks and fuzzy logics among others. We will also consider the concrete problem domain of performing diagnosis on the Space Shuttle Engines.
Biography: Eugene Santos, Jr. received B.S. ('85) in Mathematics and Computer Science from Youngstown State University, M.S. ('86) in Mathematics from Youngstown State University, Sc.M. ('88) and Ph.D. ('92) degrees in Computer Science from Brown University.
He is currently an Associate Professor of Computer Science and Engineering in the Computer Science and Engineering Department at the University of Connecticut. He is also an Adjunct Professor in the Department of Electrical and Computer Engineering at the Air Force Institute of Technology where he also served as an Assistant Professor. He has over 60 refereed technical publications in journals and conferences. His areas of research interest include intelligent information systems, decision support, human-computer interaction, intelligent user interfaces, uncertainty, probabilistic reasoning, knowledge engineering, verification and validation of knowledge-based systems, large-scale knowledge-bases, neural networks, intelligent agents, training and tutoring systems, numerical analysis, optimization, and natural language processing. He is a member of IEEE, AAAI, ACM, SIAM, Sigma Xi, and IEEE Computer Society.