Understanding how designers think is core to advancing design methods, tools, and outcomes. Engineering researchers have effectively turned to cognitive science approaches to studying the engineering design process. Empirical methods used for studying designer thinking have included verbal protocols, case studies, and controlled experiments. Studies have looked at the role of design methods, strategies, tools, environment, experience, and group dynamics. Early empirical studies were casual and exploratory with loosely defined objectives and informal analysis methods. Current studies have become more formal, factor controlled, aiming at hypothesis testing, using statistical DOE and analysis methods such as ANOVA. Popular pursuits include comparison of experts and novices, identifying and overcoming fixation, role of analogies, effectiveness of ideation methods, and other various tools. This paper first reviews a snapshot of the different approaches to study designers and their processes. Once the current basis is established, the paper explores directions for future or expanded research in this rich and critical area of designer thinking. A variety of data may be collected, and related to both the process and the outcome (designs). But there are still no standards for designing, collecting and analyzing data, partly due to the lack of cognitive models and theories of designer thinking. Data analysis is tedious and the rate of discoveries has been slow. Future studies may need to develop computer based data collection and automated analyses, which may facilitate collection of massive amounts of data with the potential of rapid advancement of the rate of discoveries and development of designer thinking cognitive models. Two examples are my recent implementations of the Problem-Map ontological framework.
We have developed a computational aid to support problem formulation in which designers input problem definition fragments into different categories as free form text. We used natural language processing to determine if designers had misplaced problem fragments in inappropriate categories. We looked for keywords in design descriptions and extracting knowledge from text ontologies using these keywords. We collected data from a group of students who used our web-based testbed. For each of the six categories in the Problem Formulator ontology, we identified classes, which were hypernyms from WordNet, relations from K-parser, and modals extracted using the Stanford parser. We first parsed the user inputs to extract the keywords depending on the category in which they were entered. We then associated these keywords to the previously identified classes and categorized them into the appropriate entities. The objective of the other implementation is to be able to automatically grade conceptual design expressed as a series of state models in our web-based testbed. We make use of a clustering technique and an aggregated inventory to achieve this task. Inter-rater agreement and correlation analysis of the text category suggestions and automated grading compared to expert human judgment shows promising results.