CogSketch: Software for studying and enhancing spatial learning

Sketching is a powerful means of reflection and communication. Whether drawing a map, the structure of a complex system, or how a process unfolds, sketching allows us to naturally externalize and communicate ideas. As a central effort of SILC, we plan to create a sketch understanding system, CogSketch, which will be used both to explore spatial learning and as a platform for sketch-based educational software. CogSketch will allow a learner to draw sketches and diagrams and to label them conceptually. It will be able to compare diagrams (for example, to assess the correctness of the learner's depiction). It will also be able to critique a learner's diagram, based on its model of the domain. Thus, it could have profound effects on basic cognitive science research. It would facilitate the creation of more precise simulations of spatial learning and cognition (cf. Forbus, 1995; Forbus, Nielsen, & Faltings 1991; Kuipers, 2000; Regier, 1996; Regier & Carlson, 2001), by providing a natural way to gather data about people's spatial representations that could be used directly in computational models.

Further, we believe that CogSketch has the potential to revolutionize spatial education. The most straightforward way to monitor a child's spatial representation is to ask her to depict them at various times in the course of learning--e.g., to draw a map or to sketch a geological feature such as a fault and instantly receive feedback. Currently, the difficulty of scoring children's drawings makes this impossible in classrooms. But what if children could sketch a map or a diagram, and their sketches could be automatically interpreted by the software, which would note the ways the child's representation differed from the correct target representation, as well as any internal inconsistencies in the child's sketch? Such a system could help teachers monitor a child's progress, and see which aspects of a topic they did not understand.

Our vision is that, in ten years or less, sketch-based educational software can be as widely available as graphing calculators are today.

This is the kind of advance that can only be accomplished through the intense, concerted long-term effort possible in a Center, drawing upon all facets of SILC's research.

Although creating CogSketch will require significant research, it will exploit a solid foundation of existing technology. It will be based on the sketching Knowledge Entry Associate (sKEA), the first open-domain sketching system (Forbus & Usher, 2002). sKEA uses a new architecture for sketch understanding, the nuSketch architecture (Forbus, Ferguson, & Usher, 2001). This approach to sketching differs from most prior sketch understanding systems, which rely on the system recognizing speech and gesture (cf. Alvarado & Davis, 2001; Cohen et al., 1997). Today's recognition technologies limit such systems to operate only in narrow domains. To escape these limitations, sKEA's interface enables users to draw and conceptually label elements of their sketches, bypassing the need for computerized recognition. Coupled with a library of tens of thousands of formally represented concepts, sKEA allows users to create sketches quickly and naturally by drawing and labeling as they go, with the system using its stored knowledge to interpret the diagram. Part of this interpretation process relies on structure-mapping simulations of analogical mapping (SME; Falkenhainer, Forbus & Gentner, 1989), similarity-based retrieval (MAC/FAC; Forbus, Gentner & Law, 1995), and generalization (Kuehne et al 2000), tying it closely to the cross-cutting project on analogy and similarity in spatial learning. sKEA has been used in a number of cognitive modeling experiments and AI experiments, including

• Modeling human performance on the Miller Geometric Analogy Test (Tomai et al 2005)
• Modeling human judgments and recognition memory errors (i.e., Feist & Gentner, 2001) involving spatial prepositions (Lockwood et al, 2005).
• Simulating automatic categorization of spatial prepositions, with SEQL (Lockwood et al, 2006).
• Learning visual/conceptual relationships via user interactions, with MAC/FAC (Forbus et al 2005)
• Learning to solve everyday mechanics problems from the Bennett Mechanical Comprehension Test, using MAC/FAC (Klenk et al 2005).

Thus our architecture for sketching can provide a basis for cognitive simulation of spatial learning and reasoning, and for creating educational software that understands sketches in a human-like way.

The evolution of CogSketch from sKEA will be guided by SILC research. We will develop CogSketch in two overlapping phases. In the first phase, CogSketch models of spatial learning and reasoning will be built and tested, drawing on basic research in spatial cognition. In the second phase, we will use CogSketch as a platform for education research on enhancing spatial learning. We will conduct studies of spatial learning using CogSketch. We will aim to identify key points where learning can be facilitated, with the goal of creating educational sketching systems. The products of this effort will include a suite of open-source software for cognitive science research and for education, along with the science base that will facilitate their widespread use.

Point of Contact: Ken Forbus

Download Page

• Download CogSketch

Relevant Background Publications

• Alvarado, C., & Davis, R. (2001). Resolving ambiguities to create a natural sketch based interface. Proceedings of IJCAI-2001, August 2001.
• Cohen, P. R., Johnston, M., McGee, D., Oviatt, S., Pittman, J., Smith, I., et al. (1997). QuickSet: Multimodal interaction for distributed applications. Proceedings of the Fifth Annual International Multimodal Conference (Multimedia '97), (pp. 31-40), Seattle, WA. ACM Press.
• Falkenhainer, B., Forbus, K.D., & Gentner, D. (1989). The structure-mapping engine: Algorithm and examples. Artificial Intelligence, 41, 1-63.
• Feist, M.I., & Genter, D. (2001). An influence of spatial language on recognition memory for spatial scenes. In J.D. Moore & K. Stenning (Eds.), Proceedings of the 23rd Annual Conference of the Cognitive Science Society (pp. 279-284).
• Forbus, K. (1995). Qualitative spatial reasoning: Framework and frontiers. In J. Glasgow, N. Narayanan, and B. Chandrasekaran, Diagrammatic reasoning: Cognitive and computational perspectives, AAAI Press.
• Forbus, K., Ferguson, R., & Usher, J. (2001). Towards a computational model of sketching. IUI'01, January 14-17, 2001, Santa Fe, New Mexico.
• Forbus, K.D., Gentner, D., & Law, K. (1994). MAC/FAC: A model of similarity-based retrieval. Cognitive Science, 19, 141-205. (Abridged version to be reprinted in T. Polk & C.M. Seifert (Eds.), Cognitive Modeling. Boston: MIT Press.)
• Forbus, K., Nielsen, P. and Faltings, B. (1991). Qualitative Spatial Reasoning: The CLOCK Project. Artificial Intelligence, 51 (1-3).
• Forbus, K., & Usher, J. (2002). Sketching for knowledge capture: A progress report. Proceedings of IUI'02, San Francisco, California, January 13-16, 2002.
• Klenk, M., Forbus, K., Tomai, E., Kim,H., and Kyckelhahn, B. 2005. Solving Everyday Physical Reasoning Problems by Analogy using Sketches. Proceedings of 20th National Conference on Artificial Intelligence (AAAI-05), Pittsburgh, PA.
• Kuehne, S., Forbus, K., Gentner, D., & Quinn, B. (August 2000). SEQL: Category learning as progressive abstraction using structure mapping. Proceedings of CogSci-2000. Philadelphia, PA.
• Kuipers, B. (2000). The spatial semantic hierarchy. Artificial Intelligence, 119, 191-233.
• Lockwood, K., Forbus, K., & Usher, J. (2005). SpaceCase: A model of spatial preposition use. Proceedings of the 27th Annual Conference of the Cognitive Science Society. Stressa, Italy.
• Lockwood, K., Forbus, K., Halstead, D. & Usher, J. (2006). Automatic Categorization of Spatial Prepositions. Proceedings of the 28th Annual Conference of the Cognitive Science Society. Vancouver, Canada.
• Regier, T. (1996). The human semantic potential: Spatial language and constrained connectionism. Cambridge, MA: MIT Press.
• Regier, T., & Carlson, L.A. (2001). Grounding spatial language in perception: An empirical and computational investigation. Journal of Experimental Psychology, 130, 273-298.
• Tomai, E., Lovett, A., Forbus, K., & Usher, J. (2005). A Structure Mapping Model for Solving Geometric Analogy Problems. Proceedings of the 27th Annual Conference of the Cognitive Science Society, Stressa, Italy, 2190-2195.