Analysis of verbal protocols from a process control task

This paper describes some of an operator’s task-related thinking when controlling a slow complex industrial process, as inferred from analysing the sequence of items mentioned in verbal protocols.  The task was controlling electricity supply to electric-arc steel-making furnaces.

Much of the detailed protocol analysis which this paper summarises is in my PhD Thesis, An analysis of a verbal protocol from a process control task, Bainbridge, University of Bristol, 1972.

Topics :

1. Introduction

2. The control task

3. Method of analysis

4. Summary of findings from protocol analysis : Continuity of control, Basic sequences of activity, Flexibility of sequences, Prediction of future events and actions, Interrupts, Limitations of the flexible sequencing strategy, Summary of effects of experience and individual differences, Possible sources of mental load.

5. Models of human process control : Size of control actions, Other aspects of the process control task, The controller's 'mental model' and 'mental picture' of the process.

6. Conclusions

Analysis of verbal protocols from a process control task

Lisanne Bainbridge

Department of Psychology, University of Reading

published in Edwards, E. and Lees, F.P. (eds.) (1974) The Human Operator in Process Control, Taylor and Francis Ltd, London, pp. 146-158.

1. Introduction

Most studies of human control performance are concerned with the tasks of the pilot and car driver, and investigate them using laboratory simulation. In those studies the controller has to make continuous movements at a speed such that the dynamic limitations of the human motor system are a major determinant of performance and this is the part of the human system component which is usually modelled.

Human control of slow industrial processes has received less attention (see Cooke 1965, Beishon 1966, 1967, 1969, Bainbridge et al 1968, Bainbridge 1971, 1972). An industrial process is usually complex, with many input and controlled variables whose interactions and response characteristics may be little understood; also alternative control strategies may be available. 

Control may be needed simply to keep the process running at a steady state to produce a particular product by compensating for fluctuations in the input materials, 

or the process may go through a sequence of different phases in each of which the correct conditions must be attained and maintained, 

and in some cases the process may have to be changed to make a product of different specifications. 

Events which require control responses are usually rare, for instance in the task discussed here 9-10 control actions were made per hour on average. Obviously, motor dynamics are not a factor limiting performance and this aspect of these tasks is trivial.

More important aspects of process control are the mental skills of organising serial attention to several parallel continuous variables and integrating this information in making control decisions.

The sampling aspect of fast tracking tasks has been widely studied but the results cannot necessarily be extrapolated to industrial tasks in which there are long system response times or periods between actions. In a paper-mill studied by Beishon (1966, 1967) changes made to the first stages of paper-making appeared in the finished product 4 minutes later and 50 yards away. One might expect that the operator will have problems in correlating input and output changes together, and so learning about the process dynamics, when these changes occur widely separated in time, and also that he may not follow an optimum sampling strategy when there are physical and time costs on sampling some of the process variables.

As many complex industrial processes are now controlled by analogue controllers or digital computers one might say that human process control is rapidly disappearing, and becoming a craft skill of only academic interest. In these automated plants, however, human operators may still be retained to monitor control performance and for their flexibility in un-programmed situations. At the moment we do not know enough about how people do these tasks to be able to design optimum interfaces and task allocation, or selection and training programmes, or to know in what situations manual performance would be adequate.

This paper describes part of a study of operators controlling the distribution of electricity to a group of steel-melting furnaces. The task existed because the electricity board only allowed the steel works to use a certain amount of electricity in each half-hour period. While there were severe financial penalties for using more than this, the most steel would be made by using all the power available and by distributing it to the furnaces in a way which disturbed the steel-making process as little as possible.

2. The Control Task

The power controller has to direct power to five furnaces, each of which goes through a sequence of steel-making stages in a total cycle taking about 5 hours. (Another operator controls these furnace stages.) At any one time each furnace will probably be at a different stage of the cycle. The power controller's task has a shorter cycle of half-hourly periods in each of which the furnaces must not use more than 50 MW of power in total.

The operator has two displays related to present power usage which can be used for compensatory tracking; if these could be maintained at a given value the total power used in the half-hour would be correct. There are several problems in doing this. One is that each stage [of the steel making process] requires a different amount of power; some stages use a large amount, others 1/6 - 1/3 of this, and some use none. Consequently the power required at one time may be greater than the average amount which can be used throughout the half-hour. If, however, a furnace changes later in the half-hour from using a large amount of power to using none and this change compensates for earlier over-usage, a cutback of power in response to the earlier over-usage is unnecessary. Optimum control therefore consists of balancing current power usage against possible future events in the half-hour. Only if this predicted effect is unacceptable need the controller change the power supplied to the furnaces. The task of predicting the timing and effect of changes is complicated by differences in the timing and power usage of stages on the different furnaces, which differ in capacity and design; also the timing of some stages is more predictable than others. In addition, the power supplied to a furnace can only be set at discrete values (0, 50, 75, 100 %) of the amount required at that stage, and the quality of steel made is affected by reducing the power supply to some stages but not others.

-

3. Method of Analysis

A preliminary report of the experiment and data collected appears in Bainbridge et al op cit. Both experienced operators and university students controlled a digital simulation of a melting-shop. (The computer also logged present display and control values once a minute.)

Measures of subjects' control behaviour showed little difference between the two groups and, as so few control actions are made, attempts to correlate display variables with control actions do not provide much information about the way in which the subjects made their control decisions. To obtain more data on this, the subjects were asked to 'think aloud' while doing the task; these verbal protocols were tape-recorded and the recordings transcribed. This study concentrates on the protocol data. More detailed reports are available in Bainbridge (1968, 1971, 1972).

While a protocol does not give complete, or necessarily reliable, data on the operator's thoughts, it is a source of much interesting information which could not be obtained in other ways. It is necessary to assume that there is some non-distorting mapping from the underlying thought processes to the verbal protocols.

The protocol is divided [by the person doing the analysis] into a sequence of phrases, each phrase can for convenience be identified as a statement about one of the following [these are types of phrase content, as identified by the analyst, not the specific numbered protocol phrases in Figure 1.a] :

(1) Present rate of power usage (P).

(2) MW used so far and time in half-hour.

(3) Power usage prediction.

(4) Comment on power usage and whether action required.

(5) Choice of action.

(6) Making action.

(7) Furnace name and its current state.

(8) Furnace state parameter.

(9) Furnace state prediction, i.e. future state.

(10) Furnace state parameter prediction.

(11) General comment.

The phrases can be grouped into blocks, in which all the phrases have a common referent. The sequences of items in these blocks can be represented by algorithmic flow diagrams, as used for the description of computer programs. 

Figure 1.a. Section of protocol divided into phrases. (Lines indicate connections between content of phrases [as identified by the analyst from pronouns and knowledge of the task. C is one of the furnaces, oxidation is one of the steel making processes].)

Figure 1.b shows the development of a flow diagram to describe phrases 5-14 in Figure 1.a : upper case letters indicate items which appear in the protocol, lower case describe operations which the operator must have carried out [to able to say what he says later]. 

Figure 1.c. The same 'routine' described in 'box notation'.

Figure 1.c describes the behaviour in a notation which makes explicit the working storage, the result of each routine is 'stored' in a 'box' at its head. 

This can describe the way in which routines are not repeated unnecessarily, since if a box already contains a result this can simply be noted, if the box is empty or its contents are unreliable the routine can be carried out. Flexible behaviour can arise if there are alternative methods, e.g. analogue judgments or digital calculations, for obtaining the same data item. Where these diagrams are similar for several blocks of phrases these blocks are assumed to be different instances of the same behaviour, and the flow diagrams to be somewhat similar to the 'subroutines' of computer programs, as the same sequence of operations described may deal with different furnaces and stage parameters on different occasions.

Having identified these routines [in the protocol], it is then necessary to account for the sequence in which these routines, i.e. the blocks of protocol phrases, occur. The protocols contain very few phrases describing the operator's strategy. Analysing what determines the sequence of different types of behaviour cannot therefore be based on explicit evidence in the protocol phrases, as is done in analysing the routines. Instead, the analyst takes all the transitions from one type of behaviour to another, and identifies by trial and error a minimum number of variables which could define the contexts necessary to account for these transitions.

4. Summary of Findings from Protocol Analysis

Figure 2. Simplified flow diagram underlying sequences of activity in protocol from S22, a melting shop manager. (P = total present power)

About a dozen routines were developed to account for the majority of phrases in the protocol. These routines obtained the data items listed in Table 1(A-left). 

The variables which determined the sequences of behaviour are listed in Table 1(B-right).

The following major points can be made about the way in which this task is carried out, based on this type of evidence.

4.1. Continuity of Control

In controlling this type of slowly changing system the human operator makes control actions intermittently rather than continuously. Monitoring the control variables is also intermittent. Control variable phrases (types 1-4 in list in Section 3) alternate with blocks of other phrase types, in which the operator reviews the current states of the furnaces (phrase types 7-11) or selects a control action (phrase types 4-6). This is a true alternation, he evidently does not 'keep an eye on' the control variables while mentioning other parts of the task, as there are occasions when a significant change in a control variable occurs during a block of other phrases and is not mentioned until the other part of the task has been completed, and then in terms which suggest he has just noticed the value rather than delayed consideration of it.

4.2. Basic Sequences of Activity

The operator shows two major types of activity; which of these occurs at a particular time depends on whether or not the process is within acceptable limits.

When the control variable values are within his tolerance limits the operator reviews the general state of the system [left of Figure 2]. He notes, for each furnace, its present stage, some of its parameters. and the stage it will go to next. If this stage change will involve a significant change in the furnace's power usage, he predicts the effect on overall power usage. If this future power usage will be unacceptable, he chooses a control action to be made when the stage change occurs. In this way he apparently maintains a mental picture of the present and future stages of the furnaces. There is evidence that he often makes use of these stored data rather than reading values from the display panel each time they are needed.

Once this review is complete he makes general comments, chatting with the experimenter. During this time he apparently monitors the control variables, as he immediately returns to the task when there is a significant change. This is the only context in which he appears to monitor the control state continuously, rather than sampling it at particular points in his sequence of activity. 

When the control variables are not within tolerance limits, control action choice occurs [right of Figure 2]. The operator first checks whether a change in furnace stage will occur which involves a change in power usage which will compensate for the present error. This may involve recursive use of the control action choice routine; if the next predicted change will lead to an unacceptable control state, the operator may consider what will be the best action to make when the change occurs. If there is no such event, he selects a furnace on which to change the power supply, on the basis of priority rules. If a furnace has changed stage since the last state review, this new stage is used in control choice, but the 'present stage picture' is not updated. When all priority rules have been considered he continues to reiterate his control choice.

4.3. Flexibility of Sequences

The operator's behaviour can be described in general as in Figure 2, though it is actually more flexible. Although a wide variety of different sequences of activity appears in the protocol, it is possible to suggest a fairly simple method by which they might be generated. During each of the main behaviours described above, the operator checks the values of the control error variables at the end of each of its sections. It would have been confusing to show this in Figure 2. If this control check shows the same value as before, he continues with the main routine from the point at which he made the control check. If the control check shows a change in control state, he changes his activity appropriately, as shown in Figure 3.

Figure 3. Direction of activity according to control state : sequence of activity in protocol from S22.

Each of the two basic behaviours [control state review or action choice] continues by investigating further parameters of the stages, or of the control choices, in order of priority [see Figure 2]. The result of such a strategy is that even if only the first part of one of the basic behaviours has been used, an overview of the situation is available. This can be compared with a less efficient strategy in which each furnace would be fully investigated in turn.

4.4. Prediction of Future Events and Actions

A large part of both behaviours is concerned with predicting. The operator appears to maintain two main types of prediction: of the next event which will occur which involves a significant change in power usage, and of the action to make when a control action is next needed. Although these predictions are not maintained perfectly, this strategy means that identification of a new system state, or choosing a control action, is often not made from scratch at the time, but by drawing on anticipations made earlier. Beishon (1969) also presents evidence from a continuous baking oven study, in which the operator had several concurrent tasks, that the operator maintains a list of events and activities covering a time span of 1/2-1 hour ahead and that this list is updated by a special scanning procedure.

In the furnace control task these two types of prediction [next important event - left of Fig.2, next best action - right of Fig.2] seem to be independent as they are related to the two main routines in different ways. 

The next event prediction is made during stage reviewing, and these data are used by the control choice routine; the prediction is not made during control choice if the data are not available. 

The control choice prediction is made during control choice behaviour, it seems to draw minimally on data obtained during stage reviewing and does not update the stage review. Control choice is concerned with stage parameters such as the time the stage started, and how much power it is using now, while stage reviewing notes mainly when this stage will end, and how much power it will use after that time.

4.5. Interrupts

It seems that external interruptions which are irrelevant to the task, such as drinking a cup of tea, or saying how many spoonfuls of sugar he takes, and digressions, such as to ask for explanations of an imperfectly understood display or part of the task, do not have a disruptive effect. After the interrupt, or digression, the operator returns to the point he was at previously. 

This is in contrast to the changes in direction of activity as shown in Figure 3. Here the changes in direction occur only at particular points in the sequence, as determined by the operator himself, rather than at any time as with external influences, and the new routine is followed without any reference to the previous one. The effect of interrupts is also discussed in Beishon (1969) and Bainbridge et al op. cit.

4.6. Limitations of the Flexible Sequencing Strategy

This strategy for dealing with a changing control state by using the routine with highest control priority is an efficient mechanism, but can also be a source of error. For instance, when a significant change in overall power usage occurs, the operator immediately considers control action. He only investigates the cause of the change later if control was not urgent, so the 'next event' prediction may not be up-to-date when he chooses his control action. As another example, he may have predicted that when a certain stage change occurs he needs to make a given control action. Another stage change occurring after making the prediction and before the event may make a control change unnecessary, but if he has not repeated the prediction and changed his assessment he may continue to make the change. (The protocols can be the only source of explanations for apparently inappropriate actions such as these.)

4.7. Summary of Effects of Experience and Individual Differences

Studies of protocols from inexperienced operators doing this task suggest that inexperienced controllers also show the two main types of behaviour, choosing control actions or sampling the general furnace states; however, at first they use feedback rather than predictive control, and organise their data search less efficiently. For instance, they react to errors in power usage by making control actions, and only slowly learn to take future events into account in making control decisions. Similarly, at first they review the stages of the furnace in alphabetical order of furnace names, rather than grouping together furnaces which are in the same [steel making] stage, as the experienced operators do.

Efficiency at updating stored information and passing data between main routines seems to be one of the main areas in which the effects of individual differences and experience with the task appear. 

Inexperienced subjects seem to be more limited in their ability to use data obtained by one routine in another routine, it seems that data remain local to one routine unless a 'common' store is explicitly established. 

One type of poor performance from experienced subjects is given by operators who are poor at updating their predictions and remembered state information as discussed above.

Another type of poor performance, independent of ability to think about the task, is shown by the subjects who can choose the action to make, considering more and more refined dimensions of this choice, but have difficulty in committing themselves to making it.

One major area of individual differences and experience lies in the length of the two main routines, e.g. in the number of parameters which may be considered, or in the complexity and accuracy of any assessments which may be made. For instance, inexperienced subjects work entirely by feedback at first, making no predictions about future events or actions and having little knowledge of the available control strategies, while the more sophisticated of the experienced operators may calculate exactly in kWh what the effect on power consumption of a given future stage change or control action will be.

It can be suggested that this 'depth of consideration' is related to the amount of processing time available, as well as to the knowledge of the task. As mentioned in Section 4.3, each behaviour starts with a global assessment, and continues to greater refinement. If the amount of time available for the routine is limited, then an inexperienced subject who is establishing strategies, or an experienced operator who thinks slowly, will not get to such depth in the routine as an experienced or quick operator.

4.8. Possible Sources of Mental Load

The same notions can be applied speculatively to the problem of the 'mental load' imposed by a task. Under the type of increasingly refined strategy outlined above, in any task where there are time limitations on performance, routines will only be used to a certain depth, so that this type of task will show a tradeoff between speed and accuracy, as has been shown to occur in the control of rapid movements.

Also, when the task loading is such as to cause stress reactions, this might impair the flexibility of the processes involved in transferring and updating data, as has been shown in studies of the effects of stress on memory.

-

5. Models of Human Process Control

[This paper was written nearly 50 years ago.  Obviously some of this section is dated, but it does indicate some of the issues.]

Models of human control behaviour are usually concerned with describing the input-output relationships in control of a single-variable system with time lags of a few seconds. In the task analysed here this 'control error - size of control action' decision is a small part of the operator's total activity; he is otherwise concerned to keep up to date with the state of a multivariable system. It is relevant to consider whether different models are appropriate to these two parts of his task.

5.1. Size of Control Actions

In this study the protocol phrases are mostly concerned with qualitative decisions, with identifying various aspects of the system, and these decisions are described in some detail. When quantitative decisions occur, such as identifying the present need for control or choosing the size of control change to make, the results are stated but reasons are rarely given. Cooke (op. cit.) and Beishon (1966) have found the same phenomenon in protocols from subjects controlling other processes with slow dynamics. This might suggest that [some of the] the models used for fast dynamic tasks are appropriate, either of the transfer function type (see e.g. review by McRuer and Jex, 1967), or of the finite-state machine (FSM) type (see e.g. Angel and Bekey 1968). In these models the solution is obtained either by plugging values into an equation or by entering a matrix. Beishon (1969) presents evidence that the operator uses a number of permanently stored look-up tables of control information. Both Cooke and Beishon (op. cit.) suggest FSM models for control of slow-response systems, and call this 'system state/action state' control.

On some occasions the control is described in detail in the protocol. In these cases the controller calculates the effect of a given size of control action on power usage, and chooses one with the required effect. This is a form of predictive control. Sheridan (1966), Kelley (1967) and Smallwood (1967) have presented control models which include an equation representation of the 'controlled element', this is used in fast time to make predictions of future behaviour on which control is based. Fogel et al. (1966) have discussed FSM controllers which include internal models of the external environment. Bainbridge's (1967) model contains templates for system behaviour which are used for prediction. Craik suggested in 1943 that human beings form some kind of internal representation of the environment. Unfortunately, the power controller's task is proportional so the data do not give evidence on the existence or nature of a mental model in the control of systems with more complex dynamics. Cooke (op. cit.) gives protocol evidence that subjects use a mental model to predict future behaviour when controlling a slow-response system.

Using protocols to obtain evidence of this type is partly confused by the way in which human decisions which have been made frequently in similar circumstances become habitual, so would not be given in a conscious report. It might be suggested that the method of control changes as it becomes habitual. Andrew's (1967) finding that, for simple control tasks at least, operation with or without an explicit model of the controlled process may be mathematically equivalent is relevant in this context.

5.2. Other Aspects of the Process Control Task

From the data presented here it would appear that an equation or FSM is not the most appropriate type of model for the controller's activities during control of slowly changing systems in which there are several interacting variables. Instead one might use an 'information processing' approach. From the findings summarised in Section 4 it appears that :

(1) An operator uses flexible subroutines when thinking about the task.

(2) At least some of these can be used recursively.

(3) Identifying the present and predicting the future system states and choosing a control action are carried out by separate routines which are interrelated by common data.

(4) Ongoing data about present and future system states and control actions are stored [remembered], but updating and transfer of these data between routines is not completely reliable.

(5) Non-task external interrupts do not disturb the sequence of activity; this implies a mechanism for 'keeping one's place' while doing something else.

Discussion of program models for complex cognitive activity, e.g. Reitman (1965), Baker (1967), Beishon (1969), usually includes an 'executive' which organises the overall sequence of activity. Figures 2 and 3 show how, in this task, alternative sequences of activity could be determined at decision points built into the routines rather than by some overseer of the task activities. However, although the alternative behaviours are consistent, and permanently available in that sense, the actual overall sequence is more flexible than the simplifications presented in the figures suggest, and further analysis of this point is needed.

5.3. The Controller's 'Mental Model' and 'Mental Picture' of the Process

In equation and FSM models the controller's internal representation of the system is an independent entity to which the controller refers, to predict future events. This could represent the controller's knowledge of the process dynamics. The protocol evidence suggests that this is insufficient to describe the whole of the operator's knowledge; this [knowledge] also includes the routines used to do the task, which describe the operations of the process and the operations which the operator can carry out on it.

Statically the whole structure of the routines used for doing the task would represent the operator's knowledge of the system, dynamically the operator’s control activity consists of processing through this structure in sequence.

Comparison of Tables I (A) and 1(B) shows that the variables determining the sequence of behaviour are the same as those items which are stored in the head boxes [e.g. Fig.1.c] of the main routines. This suggests that any decision about the best next behaviour is determined by the data in the head boxes. These stored data items could therefore be considered as the operator's on-going 'mental picture' of the current state of the process, providing the context in which he works.

6. Conclusions

Data from this study of proportional control of one variable in a slowly changing process, many of whose parameters have to be taken into account in making control decisions, corroborate and expand some aspects in earlier studies by Cooke and Beishon of more complex systems with slow-response dynamics and several concurrent control tasks. 

The information processing type of model used to structure the data stems from work on computer simulation of human problem-solving (see e.g. Reitman op.cit.). Some new concepts are needed in extending these ideas to the description of mental activities in process control. This task involves working within a fairly well-defined set of aims, priorities and available strategies rather than generating creative solutions to problems, 'solutions' for control state errors are known after sufficient learning. Instead, the difficulties of this task lie in working with a complex independent dynamic system which changes in real time.

-

References

I am sorry, with apologies to the authors, that many of the references in this paper are not on this site.

They can be found in full in :

Edwards, E., Lees, F.P. (eds.) (1974) The Human Operator in Process Control. Taylor & Francis Ltd London.

Collection of the data described in this paper was supported by a grant from the Human Factors Committee of the British Iron and Steel Research Association, under the direction of Dr. R. J. Beishon, and using facilities in Sheffield provided by the then United Steel Companies Limited.

References database

Access to other papers via the Home page

©2021 Lisanne Bainbridge