An Automated Evaluation Procedure for Artifical Imagination Systems

Artificial imagination systems are systems which perform tasks that, if performed by humans, would require imagination. Evaluation is a pervasive methodological challenge for artificial imagination systems research. Consequently it is difficult to compare imagination systems and to generalize results. Methodological challenges associated with imagination system evaluation can be addressed through the implementation of artificial assessors. The goal of the present work is to demonstrate that well-motivated artificial assessors for imagination systems can be developed and to make a case that an evaluation methodology incorporating such artificial assessors will be an improvement on current practices in the field. I establish a theoretical framework for the development of such automatic assessors based on the philosophy and psychology of imagination, and I report on an experiment investigating the performance of a pilot automatic assessor in the domain of texture synthesis.