Asking Questions and Accessing Knowledge

 This is mainly a very concentrated and rather cryptic list of the many problems !

There are more practical specifics in the paper on one technique - Inferring from verbal reports to cognitive processes.

Topics :

Distortion :

- in all reporting situations,

- knowledge elicited by a route which is not used during the task.

- - verbal / spatial,

- - reports away from doing the task.

Knowledge types :

- predicates, 

- pictures and other non-verbal knowledge.

Elicitation :

- verbal reports,

- inferring knowledge from what a person can do,

- suggestions for relating elicitation method to knowledge type.

Note - as usual the word 'skill' means having an ability at some level between novice and expert, not a specific type of task processing.

Asking Questions and Accessing Knowledge

LISANNE BAINBRIDGE 

Department of Psychology, University College London

Future Computing Systems, 1986, 1, 143-149.

Introduction

People use several types of knowledge in doing complex tasks. These require different methods of description, and different techniques are needed to minimise distortion while the knowledge is elicited [to find its content and format]. This brief note will introduce methods of elicitation, with references to more information. For a fuller discussion of some points made here without fuller reference see Bainbridge (1979).

Psychologists are interested in mechanisms of behaviour. The majority of psychological testing techniques, however, are concerned with the effect of knowledge on behaviour, not with the content of that knowledge. There are some techniques which do identify knowledge, which will be described briefly below. There are very few studies which quantify the extent of knowledge and its effect on behaviour.

This note is in three sections: 

- the ways in which elicitation techniques can distort the knowledge being investigated; 

- the types of knowledge; 

- some techniques available which may elicit different knowledge types with minimum distortion.

- - -

Distortion

At the philosophical limit no data obtained from a person is valid, whether it is the reason someone gives you for liking your new hat, or the time at which they press a button they have been asked to press immediately they see a red light. There is no way of validating independently that people are actually thinking what they report themselves to be thinking. However, speed of button pressing is a measure much used by psychologists, because the results obtained are consistent with the assumption that the button pressers are following instructions. The same is true of verbal reports. Several studies indicate the circumstances under which people do give a valid report of their thought processes (Nisbett and Wilson 1977, Smith and Miller 1978, Ericsson and Simon 1980).

Potential sources of distortion can be described in two groups. Some occur whatever the task being described, while some are specific to particular types of knowledge.

1.  Sources of distortion in all reporting situations

1. Reporting at the same time as doing a task may interfere with it.

2. Some techniques, because of time or opportunity constraints, limit the extent of the knowledge elicited.

3. Knowledge reporting involves self-presentation, and therefore can be influenced by social biases. What people say may be influenced by what they want people to think, rather than being a true report. People talk more freely in a non-threatening situation. On technical subjects, they talk more extensively to someone who understands what they are talking about. On the other hand, they may not mention points which they think are obvious.

4. Some testing techniques encourage people to develop a standard strategy for answering the questions. Such a technique does not illustrate the knowledge which would be used if each question was treated as a separate problem.

II.  Distortion because the knowledge elicitation technique accesses the knowledge by a different route than is used during the task

A. Reports which require translation between verbal/ spatial movement representations:

1. People are not consciously aware how they do perceptual- motor skills, such as bike-riding, skating or swimming. People with high levels of skill in more cognitive tasks also work automatically. When asked to report their methods they may revert to using inexperienced methods, which are slower, more verbal, and less integrated, so can be reported more easily.

2. Words, pictures, and the feel of movements are each dealt with by different parts of the brain. Images and movements may not be accurately represented when they have to be reported in words.

Asking singers to give a verbal report of what they are doing would obviously raise many of these problems !  Singing teachers can talk about what they are doing, but they have to stop actually doing it and focus on memories for sounds and movements of many parts of the body.

B. Distortions which arise from reporting knowledge when away from the real-task situation:

1. Memory is much poorer if no memory aids are provided. The person reporting is unlikely to remember all the details which affect decisions in real tasks involving interaction with a complex outside world.

2. Verbal reporting imposes a sequence on what is reported, and encourages rationalisation.

3. Reports of tasks which are usually done under time pressure may not represent the decisions which are made when time constraints are added to other task criteria.

4. The language used in reporting while doing the task (on-line), and when away from the task (off-line), suggests that knowledge is accessed in a different way in each case.

5. Some question types can be answered by people who have general knowledge of a task but have never actually done it; other questions can only be answered on the basis of task experience (Nisbett and Wilson 1977).

6. In off-line and part-task testing methods the person does not develop the working memory which usually provides the context for skilled decisions, including dealing with real-time pressures.

7 Actions which are not made in the actual workplace, or on a spatially and dynamically accurate (hi-fi) simulator, are likely not to be accurate representations of perceptual-motor skills.

- - -

Knowledge types

The following general categories of knowledge seem to be represented and best accessed in different ways:

I.  Knowledge which may be represented by predicates, or built up into more complex structures such as networks, frames, prototypes, scenarios

A. Knowledge of structures and causals:

1. Classes or categories: represented by hierarchies of categories and members (is-a hierarchies) with attribute-value descriptors.

2. Components/ mechanisms: represented by hierarchies of wholes and components (part-of hierarchies).

3. Causals, conditions on actions and events, changes between phases of operation.

In people this knowledge seems to exist in two forms: knowledge of the existence and direction of causal relations, for example, ‘when a increases b decreases’, and knowledge of the numerical parameters of the change, for example, gains and time lags. 

The first can be represented by predicates and can be verbally reported. 

The second is part of a learned skill (see III below), and the existence of this type of knowledge can only be inferred from actions. 

(It may or may not be coincidental that this distinction maps the difference between basic and applied sciences, for example, physics (basic) - engineering (applied) or psychology-human factors. The basic disciplines identify causal mechanisms, but before this knowledge can be applied it has to be quantified.)

B. Routines: including specific strategies, and general strategies for search, problem solving, analogical reasoning. Heuristics and other cognitive skills are automatic and cannot be reported, but must be inferred from behaviour while doing a task.

C. Criteria, goals, probabilities.

II.  Specific memory structures, knowledge which can be represented as predicates, but may not be represented in the brain in this way

1. Episodes: memories of specific events or instances, rather than general classes.

2. Working and context memory: knowledge of the present state of the external world, and of one’s expectations and plans in relation to it, including the structures built up during ‘understanding’. This temporary knowledge structure provides the context for decision making in any complex task, but may not be 'known' in an easily accessible way.

III.  Knowledge which is probably represented in the brain in a non-verbal way

1. Motor skills: movements and control actions, location coding.

2. Perceptual unities and figure-ground relations: visual images, mental images, maps, including changes over time. 

3. Skills in which several parts of the brain are operating simultaneously and automatically.

Descriptions of knowledge:

This will be mentioned only briefly. The approaches used by psychologists are related to those used in artificial intelligence. On the whole psychologists have been most concerned with describing the knowledge used in understanding, and use structures suggested by Anderson (1976), Schank (1982) and Rumelhart and Ortony (1977). There are few studies of how other types of knowledge are represented, for example Reiser et al (1985) suggest that autobiographical knowledge is mainly in the form of action descriptions.

Knowledge types and distortion:

The main problem is in accessing nonverbal knowledge types. Unconscious material cannot be reported, but people are very good at rationalising, at inventing an account which they believe to be the case but which is not supported by independent evidence (Nisbett and Wilson 1977). Visual and kinaesthetic knowledge must be translated into words to be reported, and skills which are done in parallel must be reported in sequence. Within the verbalisable types of knowledge, the information obtained can be incomplete if the wrong questions are used.

- - -

Techniques of knowledge elicitation

Knowledge is either obtained from direct report, or inferred from behaviour.

I.  Reports of knowledge

Techniques for asking questions to obtain verbal reports are of two main types:

1. Interviews, with spoken questions and answers.

2. Questionnaires, with written questions or drawings, and replies made by writing, drawing, or choosing one from a list of alternatives. The optimum methodologies for both have been thoroughly investigated by psychologists, see for example Brenner, Brown, Canter (1985), Oppenheim (1966).

In either case, the person reports their knowledge when not doing the task, either immediately afterwards or with some delay. Memory for specific events decays over time, and may be distorted (Wells and Loftus 1984). Reports are more likely to be useful if reporting is supported by recall cues. Suitable recall aids are video recordings of actual task activities, photos or drawings of workplaces, information about critical states to be dealt with, and asking the questions in the language used by the person being studied (Cuny 1979). The reporting situation can be made more socially natural by asking experts to discuss together what they would do, to give instructions to a novice, or to describe a difficult or dangerous situation they have been involved in.

II.  Techniques from which knowledge is inferred

These are all methods in which the person being studied does something for which they need to use knowledge, which is inferred from their behaviour. All these methods obviously depend on techniques for inferring the knowledge from the performance data. The following list covers most of the techniques which can be used without knowing beforehand the knowledge space the person is working in.

1. Methods of recording the work while it is being done, for later analysis, using : video recordings, eye-movement recordings, audio recordings (Bainbridge 1985), ethological recording (Hutt and Hutt 1970):

(a) during the actual work, or working in a hi-fi-simulation of the workplace;

(b) working with a static stimulation of the work (Leplat and Bisseret 1965);

(c) asking someone to deal with a test situation from memory (Bisseret 1970).

Audio recordings in this context are usually called ‘verbal protocols’. Interviews may cover more situations, but will not include details of particular task contexts.

2. Asking the person to do part of the task: for example predicting,  anticipating,  estimating, decision-making, reasoning, understanding, navigating, problem-solving, diagnosing, designing.

3. Tasks which are not the same as those done during the work, but which use the working knowledge, for example:

(a) category or mental-scale identification: various techniques (including response times, confusion errors, sorting into categories) for rating psychological similarity or distance between pairs or triads of items, and then analysing the ratings to identify clusters of items, or in terms of a multidimensional space: see reviews by Reitman and Reuter (1980), McDonnell (1969), Seaver and Stillwell (1983) (this contains an error on p. A-7) and Shaw (1982);

(b) drawing;

(c) making a simulator represent the situation, instead of describing it in words (Duncan and Shepherd 1975).

4. Errors provide interesting data for working backwards to identify gaps in knowledge.

Suggested elicitation techniques for types of knowledge

These are provisional suggestions, many based on observation and experience rather than formal research.

Number labels refer to sections above.

Knowledge type : elicitation method 

I.(a)

i. Classes, categories, is-a hierarchies : grouping methods. 

ii: Mechanisms, part-of hierarchies : interview; part-tasks:

draw, diagnose, design. 

iii: Causals, dynamics :

Qualitative : interview, questionnaire, reasoning tasks.

Quantitative, including criteria : actions, predictions, probability estimates, scaling techniques, questionnaires with spatial rather than verbal replies.

(b) Routines/ strategy/ heuristics, also goals : verbal protocols, or interviews.

II.

i: Specific memories: interview.

ii: Working memory: inferred from real-task decisions - may also be tested by drawing, doing task from memory.

III. 

i, iii: Skills: actions in workplace or hi-fi simulator.

ii: Perceptual/ mental images: draw, make simulator represent, judge representation true or false.

Mental maps: above, plus navigation tests.

Acknowledgements

This note was prepared while the author was Visiting Professor in the Department of Industrial Engineering, University of Toronto. The author would like to thank Professor Neville Moray for his comments.

References

ANDERSON J. R. (1976). Language, Memory and Thought (Erlbaum, Hillsdale).

BAINBRIDGE L. (1979). ‘Verbal reports as evidence of the process operator’s knowledge’, International Journal of Man-Machine Studies 11, 411-436.

BAINBRIDGE L. (1985). ‘Inferring from verbal reports to cognitive processes’, Uses of the Research Interview (eds. M.Brenner, J.Brown, D. Canter,  Academic Press, London). pp. 201-215.

BISSERET A. (197~71). ‘Memoire operationelle et structure du travail’, Bulletin de Psychologie 24, 280-294.

BRENNER, M., BROWN, J., CANTER D. (1985). Uses of the Research Interview (Academic Press, London).

CUNY X. (1979). ‘Different levels of analysing process control tasks’, Ergonomics 22, 415-425.

DUNCAN K. D. and SHEPHERD A. (1975). ‘A simulator and training technique for diagnosing plant failures from control panels’, Ergonomics 18, 627-642.

ERICSSON K. A. and SIMON H. A. (1980). ‘Verbal reports as data’, Psychological Review 87, 215-251.

HUTT S. J. and HUTT C. (1970). Direct Observation and Measurement of Behaviour (Charles C. Thomas, Springfield).

LEPLAT J. and BISSERET A. (1965). ‘Analyse des processus de traitement de l’information chez le controleur de la navigation aerienne’, Bulletin du C.E.R.P. 1-2, 51.

MCDONNELL J. D. (1969). ‘An application of measurement methods to improve the quantitative nature of pilot rating scales’, IEEE Transactions on Man-Machine Systems, MMS- 10, 81-92.

NISBETT R. E. and WILSON T.D. (1977). ‘Telling more than we can know: verbal reports on mental processes’, Psychological Review 84, 231-259.

OPPENHEIM A. N. (1966). Questionnaire Design and Attitude Measurement (Basic Books).

REISER B. J., BLACK J. B. and ABELSON R. P. (1985). ‘Knowledge structures in the organisation and retrieval of autobiographical memories’, Cognitive Psychology 17, 89-l37.

REITMAN J. S. and REUTER H. H. (1980). ‘Organisation revealed by recall orders and confirmed by pauses’, Cognitive Psychology 12, 554-581.

RUMELHART D. E. and ORTONY A. (1977). ‘The representation of knowledge in memory’, Schooling and the Acquisition of Knowledge (eds R. C. Anderson, R. J. Spiro and W. E. Montague, Erlbaum, Hillsdale).

SCHANK R. C. (1982). Dynamic Memory (University Press, Cambridge).

SEAVER D. A. and STILLWELL W. G. (1983). ‘Procedures for using expert judgement to estimate human error probabilities in nuclear power plant operations’, U.S. Nuclear Regulatory Commission NUREG/CR-2743, Sand82-7054.

SHAW M. L.  G. (1982). ‘PLANET: some experience in creating an integrated system for repertory grid applications on a micro computer’, International Journal of Man-Machine Studies 17, 345-360.

SMITH E. R. and MILLER F. D. (1978). ‘Limits on perception of cognitive processes: A reply to Nisbett and Wilson’, Psychological Review 85, 355-362.

THORNDYKE P. W. and HAYES-ROTH B. (1982). ‘Differences in spatial knowledge acquired from maps and navigation’, Cognitive Psychology 14, 560-589.

WELLS G. L. and LOFTUS E. F. (1984). (eds) Eyewitness Testimony (University Press, Cambridge).

Access to other papers via Home page

© 1997, 2022 Lisanne Bainbridge