2. the Basic Element

Topics in this section :

2.1. From the verbal protocol to the nature of the element.

2.1.1. Analysing the verbal protocol.

2.1.2. Identifying the underlying processing.

2.1.3. What needs to be included in a representation of the processing.

2.2. The basic element.

2.3. A more complex example.

2.4. Other mechanisms which will need to be added.

2.5. Summary of main points in Section 2, indicating the need for a contextual model of cognitive processing.

Section 2 : the Basic Element

Building Up Behavioural Complexity from a Cognitive Processing Element

Lisanne Bainbridge

This Section is an introduction to the proposed element, to the reasons why it was devised, and to the ways in which elements are linked to carry out cognitive processing and support working storage. Later sections in this review will discuss wider properties of the element.

2.1. From the verbal protocol to the nature of the cognitive elements

What was the origin of the proposed constituents of a cognitive element ? This Section illustrates how the processing element originated from the protocol analysis.

The first sub-section gives a protocol example.

The second sub-section shows how the processes underlying this protocol were inferred.

The rest of this section discusses why the processing has been represented as in the box element.

2.1.1. Analysing the verbal protocol

In the furnace power allocation task (e.g. Bainbridge, 1974) the operators were asked to speak their thoughts out loud, and this verbal report was recorded and transcribed. Table 2.1 shows a short section of protocol from an inexperienced controller (university student). Five furnaces are involved in the task, named A, B, C, D and E. Each furnace goes through a sequence of stages in the steel making process - charge, melt, half-melt, oxidise, reduce, tap, fettle - at its own rate, depending on its size and what type of steel it is making. The protocol fragment in Table 2.1 shows the operator predicting when furnace C will end oxidising. This will be an important event relative to the control task, as the furnace will change from using much power to using none, a step change in power demand.

Table 2.1 : A sample protocol fragment(Bainbridge 1972, 1985) - transcription of a short segment of a verbal protocol recording. . . . indicates a pause in speaking.

C is on oxidation now that’s something you can make an estimate for it’s a quality so I must leave it alone. . . oxidation average length is one hour 30 minutes for C and started at time zero no it didn’t it started at time 33 minutes how confusing of it so it’s got nearly one and a half-hours to run . . . I’d better check that oxidation for C one hour 30 minutes started 50 minutes ago so it’s got 37 minutes to go. . .

This section of protocol can be divided into distinct phrases, on the basis of natural language understanding. Figure 2.1 shows the same protocol, with each successive phrase written on a different line. The analyst has also judged, on the basis of natural language understanding and knowledge of the task, which of the phrases are linked together. These links are shown by the connecting arrows.

Figure 2.1 : The protocol in Table 2.1 divided into phrases, with cross references between them (Bainbridge 1974, 1985)

2.2.2. Identifying the underlying processing

To make a simulation of the thinking expressed in the protocol, it is necessary to identify the processes underlying the protocol. Phrases 5-14 in Figure 2.1, for example, fall into two groups, which differ considerably in detail, but are both concerned with predicting the time when a given furnace stage will end. So the analyst's aim was to find what common processes are carried out whenever the person predicted the end of a furnace stage, and to describe these processes in such a way that they could be used for any furnace or stage of steel making. The aim of the paper-and-pencil simulation was to represent the underlying thinking processes, not the details of the language in which they were expressed. It is of course a major assumption that the same processes underlie each time the operator predicts the end of a stage, rather than that the operator has a separate set of mental instructions for predicting the end of each stage on each furnace. In this case, however, the protocol evidence did not suggest that there were major differences, and also the prediction process had been described in the task instructions as a general strategy.

The protocol does not mention explicitly all the processing which must have been involved in any given piece of thinking. The underlying processing was therefore identified by combining two types of evidence :

- by drawing together information from all the occasions on which given processing was done (in the furnace operator study, a cognitive simulation was only suggested for processing which had been done at least twice),

- by the analyst inferring any processing which must have been done in order for the operator to be able to say what he did, but which he had not explicitly mentioned.

The process of combining these two types of evidence is illustrated in Figure 2.2.

Figure 2.2 : The processing underlying the protocol in Figure 2.1 phrases 5-14 (Bainbridge 1974, 1985).

It is obvious there is a large gap between what is in the protocol and what is given in Figure 2.2. This gap has been bridged by the analyst's assumptions :

This process of simplification is done partly by human categorisation - what is the general type of content of this phrase ? What categories are used depends on the purpose of the analysis. The validity of the categorisation could be tested by using a number of independent judges. These aspects of protocol analysis are discussed in Bainbridge (1985, 1990).

Secondly, when the speaker does not mention something, the analyst has added what the operator must have thought about in order to be able to say what he did - added in lower case in Figure 2.2.

I thought there were several important things missing in Figure 2.2 as a representation of human cognitive processing. Figure 2.2 is somewhat like the form of computer programmes at the time the protocol analysis was done, c1970.

1. It does not mention the 'cognitive goal', what the person wanted to know, although that was mentioned in the verbal report. And each later statement is in the form of what information is needed and then its value. Though there is no information in the verbal report about how the operator is finding this information (it was in the task instructions).

The wording of a protocol often suggests that the person speaking knows what they want to know or do, and then (particularly in the case of an inexperienced operator) works out how to do it, e.g. phrases 2 or 11 in Figure 2.1. A representation is required which makes explicit the need met by the processing, rather than leaving it implicit in the final operation of the processing as in Figure 2.2.

The reasons for using a goal (or cognitive need) oriented account are discussed further in Sections 4-5.

The 'cognitive' goals (what the operator wants/ needs to know about) are not the same as the 'task' goals (control the electricity supply). Meeting the cognitive goals is the thinking the person must do to meet the task goals.

2. I thought it was important to make explicit the use of, and cross references within, working storage. In human cognitive processing, memory can be limited in capacity, so it can be important as part of cognitive task analysis to identify what the person needs to keep in memory.

The format of representation in Figure 2.2 does not make the use of working storage explicit. In trying to understand the protocol, I found that working storage was a key underlying factor. It could for example be involved in the pronominal and other content links between phrases indicated in Figure 2.1. (This will be discussed further in Sections 4-6). It was therefore useful to have a representation which made the working storage explicit. In the element diagrams a box has been used, with arrows to represent cross references. In Figure 2.3 below, cross references within the group of phrases are shown in an abbreviated form, by arrows in the boxes. Figure 2.7 below is an example in which the cross references in working storage are shown in full.

3. It later became obvious that :

- people can use different methods for meeting the same 'goal', so the goal and means are independent, and there needs to be an indication of this choice point. This choice point is represented in the box element diagrams by a stepped arrow. I chose to represent this choice as being done using meta-knowledge about the alternatives. This is not represented explicitly in the element but the stepped arrow is a shape code to indicate this mechanism, see discussion in Section 7.

- The methods of meeting a cognitive goal can include :

- attending to some feature of the environment : the interface, task instructions, another person, etc.

- accessing a stored knowledge base.

- doing some processing,

See Figure 2.3 and 3b.1 in Section 3b, also Sections 3a-d.

- The activity in Table 2.1 is done as part of building up an overview of what the furnaces are doing, what will happen in the future, when an action may be needed, and what that action should best be, see more on these wider activities in Sections 4-5 .

2.1.3. The processing element representation

The next Figure shows the processing in Figure 2.2 in a way which emphasises the cognitive goals of each thinking step, what the person wants to know, and what is stored, as well as how the processing steps relate together.

Figure 2.3 : The processing in Figure 2.2 described in terms of goals, means of finding what is wanted, and stored data (Bainbridge, 1974).

Figure 2.2 shows a possible conventional representation of the cognitive processing in the protocol fragment in Table 2.1 and Figure 2.1. Figure 2.3 shows the same processing represented in a way which makes cognitive needs and working storage explicit, also the link to the method of meeting the 'cognitive need'. This representation was chosen after trying out many possible alternatives, to make explicit several important aspects of the processing which are not shown in Figure 2.2.

2.2. The basic element

There are other features of cognitive behaviour in complex tasks which need to be included (and which will be discussed later), so the element was eventually expanded into the general form shown in Figure 2.4.

Figure 2.4 : Main components of the cognitive processing element (Bainbridge, 1972, 1992)

Figure 2.4 shows the components of the basic processing element and its links. (The element was previously called a 'module', but this was confusing as it does not distinguish the basic element from the structures it can be built up into.)

There are five main parts of an element, in two groups :

A. a need, and working storage :

A1. a statement of a cognitive need. In some papers and here this may be called a cognitive 'goal', but the word goal causes problems. Another possible term would be cognitive 'function'. This is shown as the focus of the processing.

A2. the associated working storage, indicated by boxes. The left hand box (not shown in all Figures) represents the input parameters for the element, the context which describes what specifically is needed. The right hand box is a 'place holder' for the result of the processing to meet the cognitive need.

B. there are three types of link from these pairs of boxes to others (three different types of arrow in Figure 2.4) :

B1. the link to the processing for meeting the cognitive need. This link is indicated by a stepped arrow (arrows may be stepped left or right in the Figures, for convenience of visual layout).

An operator may use several different ways of meeting a given cognitive need, at different times (see Section 3), so a representation is needed which makes explicit that the goal of (or need met by) the processing and the means of meeting it are independent. The reason why this link is represented by a different type of arrow is that it represents a proposed feature of the processing which is not explicit in this diagram - deciding between alternative ways of meeting the goal/need. This link plays a special part in the choice and organisation of behaviour, which will be discussed more in Sections 7-8.

B2. the links between cognitive needs, by which they are built up into larger structures or 'routines', which meet other cognitive needs. These links are represented in Figure 2.4 by dashed lines (and are discussed further in Section 3c.)

B3. the cross references between different parts of working storage. These are represented by solid lines in Figure 2.7 below, and more compactly by double lined boxes and arrows in boxes in Figure 2.3. (These cross references, and the nature of the context for behaviour, will be discussed further in Sections 4-6.)

One descriptive problem is that this element is sufficiently complex, that its components can’t all be included in all Figures. Different figures emphasise different features of the processing described, and so need to include different components.

For example, the basic unit used in Figure 2.3 is a simplification compared to Figure 2.4. Yet Figure 2.3 contains some extra features needed when elements are grouped together.

A missing item in Figure 2.3 : there is only one box associated with each 'cognitive goal', the 'results' box, to reduce clutter in the diagram. The omitted 'left' box specifies the context in which this processing is being done, in this case :

- the furnace about which the operator wants to know ’time to go' (this would be transmitted down to the missing context boxes for lower elements),

- the reason why he wants to know the time to go - the higher level cognitive goal that this 'routine' is being used to meet.

The majority of my diagrams just include one box, the right one, but the left one is needed for completeness.

There is also some extra information in Figure 2.3 :

- the main cognitive goal of the ’routine’ is shown by a double-lined box. This relates together the point where it is found with the point where it is needed.

- when goals are met by direct sensor/effector activity rather than by working through a routine of subsidiary cognitive goals, this is labelled in italics. So italics indicate where details of interface design may be important.

And the use of sensors is as active as the use of effectors. The operator seeks information as part of doing the task, he does not simply react passively to information. This is top-down behaviour.

In fact this whole little segment of behaviour is about anticipation of, not reaction to, external events (also the whole example in the next section is about anticipation).

- an arrow in a box refers to information which is 'stored' in previous elements in the cognitive processing.

These elements were at first used simply as a notational convenience. Locations in the digram provided me with a place to note :

- the purpose of a segment of behaviour,

- the result of carrying it out,

- the method(s) used to find the result.

The cognitive processing element thus originated as a form of notation which captured, or made explicit, various key aspects of the nature of the processing. I later found that this representation could provide answers to questions about mechanisms for more complex aspects of behaviour, so it was expanded into being claimed as a 'model’. Hopefully, why these are key aspects of processing and how they are involved in the organisation of behaviour will become clear in the rest of this review.

2.3. A more complex example

Table 2.2 : Prairie Island Nuclear Power Station Incident, Time 1414
(Bainbridge, 1992, Table 1, adapted from Operator Decision/ Action Summary in Pew et al, 1981, p.B-12.

This table is not based on analysing verbal protocols during the task, but taken from interviews with the operators after the event.)

And a few minutes later :

Table 2.3 : Bainbridge 1992, Table 2, also Adapted from Pew et al (1981), time 1421

These further examples illustrate more detail about some basic aspects of what needs to be in a representation of complex cognitive activity - what the element needs to be able to do. The behaviour in these examples is the cognitive activity of a team of nuclear power station operators reacting to a high radiation alarm. The operators did not just react to the alarm as an alarm, but used their knowledge of the process to infer what had happened. Some of the possible explanations for the alarm, such as a pipe rupture, are not instrumented so cannot be displayed directly. So they have to be inferred by the operators from other information. In this example, the operators went straight from reacting to the alarm (bottom-up processing) to actively looking for information about the reasons for the alarm, thinking generated from their knowledge of the process (top-down processing).

These two excerpts each have 4 phases :

- an alarm brings the operators’ attention to the plant,

- they identify what is happening on the plant, with less or more certainty depending on the amount of information available,

- they predict what may happen next,

- they choose an action in relation to the predicted event.

This sequence might superficially described be described as input-action, but that would ignore that in both cases the operators are choosing their actions relative to predicted not environmental events, they are reacting to the result of cognitive processing rather than to the current situation. The first excerpt also involves active looking for information (top-down processing) and the second includes feed-forward control.

infer state of process by

1. find possible causes of displayed information.

2. select one of these by

i. find which one has the highest frequency.

3. check interpretation by

i. find nature of distinguishing evidence

ii. look for/ wait for distinguishing evidence

iii. compare actual with expected.

Figure 2.5. Possible generalised account of the state identification processing underlying the first sections of Tables 2.2 and 2.3.

In this description of what they are doing, what the operators do is expressed in terms of their cognitive needs, which are met by carrying out more detailed cognitive processing, also described as cognitive needs. In the Figure 2.5 list, level of detail is indicated by indentation and type of number.

The above list could also be represented by simplified 'elements' as in Figure 2.6.

Figure 2.6 : Element representation of the processing in Table 2 (Bainbridge, 1992). See also Figures 3b.1 and 3b.2 in Section 3b.

This simplified diagram has been made more clear by omissions, which it’s important to note, i.e. by :

- leaving out the 'left/context’ boxes, which signal the details about what will be looked for in the 'lower level’ processing. The results of those are shown in the right-side 'results' boxes. For both boxes, and cross references of data, see Figure 2.7.

- not indicating the means by which these items are found. Nearly all those hooked arrows are fulfilled by finding information in the operator’s knowledge bases about their task. This important feature of cognitive processing has not been mentioned before : the knowledge the operators have to have to be able to do what they report doing. For a version of Figure 2.6 which includes the knowledge base, see Figure 3b.1. Knowledge bases are discussed more in Sections 3b and 3d.

The cognitive processing by these nuclear operators could be modelled in many ways. Figure 2.6 shows how it could be described using the processing element. Two main aspects of the links between elements will be mentioned in this section : the transfer of control between parts of the processing, and the use of working storage.

Transfer of Control

Two types of transfer of control are represented, within and between the levels of organisation of the behaviour. Figures 2.5 and 2.6 show three levels of detail, The highest level element could be called the 'originating cognitive need', and in some diagrams is indicated by a double lined box called a 'head' box. (The need for a special term will become clear later). Passing control from one level of detail down to another is shown by a stepped arrow. Within one level, the elements are linked by dotted lines. So two types of transfer of control are represented, within and between the levels of organisation of the behaviour.

a. Links between levels are represented by the stepped arrow, meaning 'the above is implemented by doing the following'. The stepped arrow also indicates a goal-means separation : the cognitive need is independent of the means by which it is met. And also that this can be a point for choosing between different ways of meeting this need. This gives the possibility of great flexibility in behaviour, as discussed in Section 7.

To make Figure 2.6 simple, it does not show explicitly how each of the cognitive needs is met, there is not something at the end of every stepped arrow. For more on this, see Sections 3a-c. (In this case there was only one method for finding the information needed, these were experienced operators.)

b. The arrows made of dashed lines represent transfer of control between cognitive needs within the same level. Items linked together by dashed arrows make up a 'routine' which meets a higher level cognitive need. In this example, the 'routine' is somewhat like a conventional programming routine, but this is misleading as these 'routines' have some different properties which are discussed more in Section 3c. (Elements are also linked together into a 'routine' by cross references in working storage, see next.)

Working Storage

Working storage ('working' storage in the sense that these items only need to be stored while the task is being done) is represented by the boxes associated with the cognitive needs. The left-hand box is for the input parameters, so that what the lower processing finds is specific to the given circumstances. The right-hand box is for the results of carrying out the processing which meets the need (e.g. in the nuclear example, what is found as a result of looking for possible causes). Figure 2.7 is an attempt to make all the cross references explicit.

Figure 2.7 : Cross references in working storage (Bainbridge, 1992).

In this diagram the arrows do not represent transfer of control, or sequencing of processing, they represent cross references of information used or needed.

Left box : input parameters for meeting goal, usually specified previously in other processing.

Right box : output of the processing, box may be linked downwards to a 'routine' for finding the information needed, upwards to supply the result needed earlier.

See Figure 3b.1 for how the main cognitive needs (finding information, filling the right box) are met.

The cross references between items in working storage are not represented in Figure 2.6 explicitly. Figure 2.7 does show these local cross references. The left hand box receives the inputs, which are either passed on from the inputs of previous cognitive needs or come from the result of meeting previous cognitive needs. The right hand box represents the result of meeting the cognitive need. This result may then be fed 'down' the levels of processing, for use in later cognitive processing. Or it may be fed 'up' the levels of processing, as a result which gives the result needed at a 'higher' level.

Figure 2.7 suggests that elements may be structured together into a 'routine' by these cross references in working storage.

The boxes and cross reference arrows are not meant to imply actual storage or transfer of data from one place to another. It might be more appropriate to think of the arrows as type-token links referring to data stored elsewhere, and the box as a symbol/ place holder for a point at which the results of processing could be available. But we don’t know how neurones do these things. Again I do not at any point wish to limit the realisation of these elements to the computer programming mechanisms available in a particular era.

Evidence from longer segments of behaviour (examples in Section 3c) suggests that the data found by a main 'routine', for the originating cognitive need, may be available for a longer period of time than just during the 'routine', and may be referred to by other 'routines'. This will be called continuing working storage (or the ’overview'), and may be represented by a double-lined 'head' box. The working storage items cross referenced within a 'routine' appear to be available for reference only during that 'routine', so this type of local storage is temporary. As cross references are only either within a 'routine', or to continuing working storage, this might suggest that a 'routine' is an independent processing module. This is discussed more in Sections 3c and 6.

Other aspects of behaviour (see Section 5) suggest that the contents of working storage are available in parallel. It is possible to think of these boxes as a special type of blackboard, in which the contents and the format of this blackboard are inherently structured by being related to the cognitive needs and the links between them. And only the 'head' data is available to later processing (see Section 5). (Though again I don’t want to suggest this mechanism might be implemented by specific computer-based mechanisms which happen to be available as concepts to researchers working at a particular period.)

The term working 'storage' is used in this paper, rather than working 'memory', because this working storage is not simply a memory for items which are replicas of the input information. Working storage here is a task-related temporary structure of data which has been transformed and interrelated as a result of task thinking to meet the cognitive needs. For example, the person does not necessarily remember the value of a process variable, but what this value implies for action, and what the action should be. They may not remember the specific times of events, but use these data about times as part of working out what will happen next, when it will happen, what this will imply for action, and what to do about it. (More detailed examples of this in Section 6a.) So this working storage is a cognitive structure, and it may not be possible to infer what is in it simply from observing what information the person takes in and what actions they make. The cognitive processing elements make this use of working storage explicit (which is not usually done in descriptions of cognitive processing), because working storage is such an important mechanism, both in linking together local processing, as outlined in this Section, and in providing the context for the organisation of more complex behaviour, see Sections 4-6.

The rest of this review will discuss in more detail the properties of the element outlined here, and its potential for accounting for complex behaviour.

2.4. Other points which will be discussed later

Later sections show how the properties of the basic 'element', in :

- building up the overview,

- using knowledge bases to provide information and

- meta-knowledge to decide how to do the task,

can build up into complex behaviour.

As well as the elements and 'routines', there are two aspects of the processing which have been mentioned in passing but not yet in detail :

- the context within which the processing is done (see more in Sections 4-6).

- the knowledge base providing information about both the 'routines' and what they refer to (see more in Sections 3b and 3d).

And there are some major features which are discussed later :

* the combination of conditional elements into 'sequencer' structures which retain a contextual overview of the state of the task, and which determine the sequence of behaviour, what to do next. This is discussed more in Sections 4-6.

* there are often alternative methods for meeting a goal (at the stepped arrow). In this account of the nature of cognitive processing, I decided to represent this, and the method of choosing between them, by suggesting that different methods of doing a task are chosen between on the bias of meta-knowledge associated with the each method (what task qualities it meets, what personal qualities it needs) which constrain the best working method at this moment (given the task qualities needed, and personal qualities available), and so how best to do something at this time. This is discussed more in Sections 7-8.

Superficially, this element might just look like an unnecessarily complicated if-then element. However, later sections of this paper will show the power of these properties of the element :

* the cognitive needs act as the focus for the organisation of behaviour (most remaining Sections).

* the contents of working storage are built up by cognitive activities. The contents are not simply a representation of unprocessed input information, and the contents are structured by their relation to the cognitive needs (Section 6a).

* the links between the working storage items build the working storage up into a contextual overview, which is cross-referenced by other cognitive processing, is structured by its relation within the task processing, and is the basis for choosing both what to do next and how to do it (Sections 4-7).

* the goal/need and the means for meeting it are independent. It is suggested that the link between the two (represented by the stepped arrow) is a flexible and powerful decision point using meta-knowledge, which is involved in mental workload and learning (evidence for this claim comes later, see Sections 7-8).

2.5. Summary of main points in Section 2

* There are many parts to a processing element :

- a statement of a cognitive need,

- the input parameters which the result must meet.

- associated working storage for the result of carrying out the cognitive processing which meets this need,

- links to operations for meeting the cognitive need. This may be of three types :

- more cognitive processing, also described as goals and elements.

- referring to a knowledge base.

- using sensors or effectors.

* The method ('routine') for meeting a cognitive need may be itself made of linked cognitive needs.

* Cross references between working storage items within a 'routine' also link the 'routine' together.

* The cognitive needs and the cross references also structure the working storage.

* Working storage contains the result of the thinking which meets a cognitive need, so contains transformed data, not simply a representation of the outside world.

* The result found by a main 'routine' is maintained in working storage for a longer time, and is available for reference by other processing, it is part of the context in which later processing is done (evidence for this comes later).

Later sections suggest how the properties of the basic 'element' - in building up the overview, using knowledge bases to provide information, and meta-knowledge to decide how to do the task - can build up into complex behaviour.

Other papers on this site can be linked to from the Home page

In this review, there are 3 main groups of topics.

1. Introduction.

Basic element, sources of data which meet cognitive needs.

2. The cognitive processing element.

Meeting an information need :

3a. by finding it in the environment.

3b. from a stored knowledge base.

3c. by working through a 'routine',

or referring to the result of using a routine elsewhere.

3d. more on knowledge bases.

Choosing what to do.