The observation of real work involving high-level competencies is not always a practical option. For example, it is difficult for assessors to directly observe candidates generating designs and ideas, analysing and solving problems, leading teams and developing long term plans. In other cases, the presence of assessors may be intrusive (such as in counseling).

Other critical competencies, such as responding to unforeseen problems and emergencies, are even harder to assess because they are seldom used. Most candidates who work in high-level roles are not closely supervised. Therefore, there is also a question of who can know enough about what they do in order to assess them.

For institutional RTOs, the observation of candidates performing real work is even more problematic since many of their students have no access to a workplace.

Despite the practical difficulties involved, there are some circumstances where it is highly desirable for assessors to observe key aspects of high-level performance. To address this need, the following case study is used to illustrate the design of effective simulations.

The case study is workplace-based and involves assessing personnel who manage responses to major events that seldom occur, or where it is unsafe or too costly to conduct assessment in the real work environment. It describes how to design and write a simulation for the Unit of Competency PUASES006A: Manage emergency operations, from the Public Safety Training Package. It should be noted that while emergency management in the mainstream public safety sector has been chosen as an example, most managers and team leaders may be called upon to manage serious and unforeseen emergencies. For example:

• A retail store manager may have to evacuate customers and staff in the event of a fire
• A supervisor in a warehouse may have to implement emergency procedures after a container of dangerous chemicals has been ruptured by a forklift.

The use of simulations in high-risk situations arises out of the reluctance of assessors to rely on other forms of past and present evidence (such as third party reports, interviews and workplace documents). The simulations are designed to predict the ability of candidates to apply critical skills in pressure situations and in a real workplace environment. Prior to assessing their overall performance, assessors should be satisfied that candidates already possess the skills and knowledge to perform the individual tasks involved.

Simulating emergency management

Effective emergency management does not simply rely on the application of technical expertise and well–defined operating procedures. It also requires candidates to demonstrate non–technical skills, such as efficient communication and decision-making, sometimes under considerable pressure. Without adequate development of these non–technical skills, problems can arise during crisis management such as:

• Poor appreciation of an emerging crisis
• Inadequate situation analysis
• Weak leadership
• Faulty decision making
• Blind adherence to procedures
• Role ambiguity
• Lack of explicit coordination and communication.

Assessment strategies must be focused towards confirming these skills as well as an all round capability to deal effectively with emergency situations. Emergency management requires professionals who are able to combine a range of technical, human and logistical skills to ensure appropriate responses before, during and after an emergency occurs.

A simulation exercise provides a controlled opportunity to observe an emergency management performance. It also enables assessors to confirm the ability of candidates to respond to unfamiliar and unforeseen contingencies and manage emergency responses without close supervision.

The skill of assessors is crucial to the success of conducting large-scale simulations. The general difficulty of observing management roles together with the issue of subjectivity when assessing candidate responses to problems can lead to unreliability in assessment. One way to increase reliability is to employ subject matter experts to assist with the design and conduct of the exercise. Another way is to design tools that provide a consistent basis for professional judgment of competence.

What is a simulation?

Exercises, or simulations, fall into three categories.

1. Hypotheticals – where a facilitator asks participants a series of 'what if....?' questions as a way of eliciting ideas or developing a novel solution to a problem.
2. Table top exercises – where the participants work together to consider a problem and produce one solution (from a group of known solutions). While all the decision makers can be involved, the artificial format makes it difficult to represent what would happen in the real world.
3. Simulated or 'live' exercises – where the participants respond to a scenario using the resources provided. These exercises can be highly sophisticated, as in the case of war fighting, and involve the deployment of personnel and equipment. They may also involve the development of sophisticated computer programs. These large-scale exercises are prohibitively expensive for enterprises outside aviation and the defence, fire and police services. However, simulations can be successfully designed on a much more modest scale.

Next we provide information on how to design a large-scale exercise and relevant assessment tools to collect the evidence produced. A small-scale exercise will involve exactly the same approach. The challenge for assessors in all simulations is to design and document a scenario that is realistic in terms of the activities and resources required.

Managing a simulated flood event

The following case study describes the assessment of senior State Emergency Service (SES) personnel during the management of an emergency operation.

The relevant Unit of Competency is PUASES006A: Manage emergency operations.

The unit applies to the following personnel:

Emergency managers	Broad duties
Operations Controller (OC) [Unit Controller, Deputy Controller or other senior SES member]	Controls response to the event Responsible for planning, intelligence and strategic decision making
Operations Centre Manager (OCM) [May be combined with OC for small scale events]	Manages the Operations Centre Responsible for the efficient running of the Centre and tactical management of the operation [routine decisions, coordinating activities and logistics, managing information, assisting OC to prepare operational orders]

SES agencies throughout Australia are typically responsible for the combat of natural hazards such as floods and storms. They may also provide response teams for a range of other emergency activities, such as search and rescue exercises. SES agencies are highly decentralised with the establishment of units reflecting local government areas and the locations of greatest threat. It is common for the local Operations Controllers and all of their team members to be volunteers. The high incidence of volunteer members poses special problems for training, including issues relating to their motivation and availability.

The Unit of Competency PUASES006A covers the management of emergency operations, usually within a single agency command. The information within the unit provides valuable advice on relevant sources of evidence for assessment. A sample of this information has been extracted in the following table.

Identifying sources of evidence for the unit PUASES006A (only elements 1 and 2 are considered for this example)

PUASES006A	Manage emergency operations
ELEMENT OF COMPETENCY	PERFORMANCE CRITERIA
1) Prepare for operations	1.1 Operational information is obtained and hazards are identified to permit accurate assessment of the problem 1.2 Operational plans are activated in accordance with local emergency management practices and procedures 1.3 Operations Centre is activated and confirmed as ready for use 1.4 Operations Centre’s staff recall system is activated.
Possible sources of evidence It would be costly and impractical to directly observe candidates opening up an Operations Centre. Competence can only be confirmed over a period of time. Third party reports from managers (written reports, logs, etc) would be appropriate. Other evidence could be: questioning about roles, responsibilities, authority, information sources (flood plans) (1.1 – 1.4) response to scenarios to confirm knowledge of emergency management arrangements (1.1 – 1.4) development of action plans that outline preparations for storm and flood operations (1.1 – 1.4).
2) Manage an Operations Centre	2.1 Operations Centre tasks are allocated, staff rosters confirmed and briefings conducted in accordance with operational plans 2.2 Internal and external liaison networks are activated and maintained to ensure effective communication 2.3 Operations Centre staff work is supervised to ensure it meets relevant OHS and other regulatory requirements 2.4 Physical and financial resources are managed in accordance with organisation’s policies and procedures.
Possible sources of evidence While assessors may observe a briefing, it is more likely that competence would be demonstrated over time and in a range of contexts. Sources of evidence could be: observations during a simulated emergency operation (2.1–2.4) workplace documents (rosters prepared to meet a given scenario that reflect scale of operation, work and childcare requirements of volunteers, shifts, skills required) (2.1) workplace documents (a real/simulated briefing using SMEAC convention) (2.1) workplace documents (fax stream list to agencies) (2.2) OHS audit using a checklist for assessing centre layout, lighting, furniture, staff rotation, etc (2.3) third party reports (team members, manager) concerning interpersonal skills, personnel and resource management during operations (2.3–2.4).

 The Table above highlights the importance of simulated exercises in providing evidence of competence for this unit.

Writing a simulation

A typical simulation has five elements.

1. Creating a scenario. This is called the general idea. This should set the scope and context for the exercise and outline the types of resources that are available to the team or individual.
2. Designing additional information that participants must react to and complicated problems that they need to solve. These are called the special ideas. They must be realistic, but not 'over the top'. They should also have a logical sequence, mirror the likely occurrence of various unforeseen problems and be consistent with the overall time constraints of the exercise.
3. Providing instructions for the exercise manager(s). Their role is to:
4. rotate candidates through OC and OCM roles at simulated change of shifts
5. monitor the decisions that candidates make as the exercise unfolds
6. second guess consequences for the actions and decisions that candidates take and modify the exercise to suit
7. decide whether to intervene to keep the exercise safe
8. decide how much pressure candidates can take and whether to push them beyond their comfort zone
9. decide whether to rescue candidates (as in the real world) to prevent catastrophic failure and to keep the exercise flowing
10. decide when to introduce special ideas to increase complexity and pressure
11. role play interactions with the candidate(s) to draw out their reasoning behind key decisions and actions.

Decisions will also need to be made on the duration of the exercise, the location, the number of candidates and whether they will rotate through various roles. The manager must know what resources are available. For example, will all information be provided in print or can the exercise manager contact the group via the phone, fax or email to increase realism? The information could also be assembled in a kit.

1. Providing the assessors with tools that they can use to record the performance of candidates during the exercise. Depending on the budget, there may be more than one assessor if there are six or more candidates. In the SES case, a District Controller or the State Training Manager will be used as the second assessor to enhance quality assurance. Assessors could be allowed to ask questions during the simulation, although questioning may distract the candidates. In any event, there should be a 'post mortem' of the exercise and questions relating to the process can be left to this debrief.
2. Providing the participants with the information they need.

Case study: The flooding of the Retnuh Valley

General idea The Retnuh River valley is prone to flooding. If the water level rises more than 11 metres then one large and several small towns will be cut off. You have maps of the area that show the locations of a major town, a district hospital and high school, three primary schools, a retirement home, an airport, fire brigade and a defence base. The council has one major depot within the area. There are 12,000 people living in the valley. The main population centre has a ring levee. It is broken by roads in four places and by the railway track in two places. These gaps will need to be closed in the event of a major flood above 11 metres. Detailed information about the towns and their populations is provided in the exercise manual. You have also been provided with the relevant flood plans, flood intelligence cards, standard operating procedures, Controllers Handbook, and a manual dealing with Emergency Management Arrangements. Steady rain is falling and the Retnuh River is rising. The Bureau of Meteorology has issued a minor flood warning. The gauge reading at Girella Bridge at 8.00 am is 3.8 metres. The Division Controller has issued a media release that includes advice to farmers and equipment operators about moving stock and plant to higher ground.

 The following special ideas will be introduced during the exercise.

Special idea 1 Twelve hours have elapsed … the Bureau has issued a second bulletin that includes a moderate flood warning. Establish the flood Operations Centre. Assign tasks. Allocate roles. Set up flood boards. Develop a roster. Activate gauge readers. Do you advise other agencies and organisations? Determine the likely effects in the valley using flood intelligence cards, gauge heights and flood plans. The Local Emergency Management Officer attached to the Council is on the telephone (role play). Discuss what the unit needs and establish what the Council can offer. Determine mitigation measures, personnel and resources required. Estimate lead times for necessary actions (evacuations, closing levee, involvement of other services).

Special idea 2 A member of the Operations Centre staff does not trust the Bureau of Meteorology bulletin. The staff member is in a critical role. Do you counsel the member or assign them to other duties?

Special idea 3 The District Controller telephones (role play) and provides five readings from up–stream gauges that indicate the levels are approaching 6 metres. There are three reports from the public about road closures. There are six calls for help. Not all the information is congruent or equally reliable. Analyse the stream gauge data and take appropriate actions. Deal with each request for help.

Special idea 4 One of the volunteers reports that her boss wants her to return to work immediately. Your best flood plotter needs childcare backup tomorrow. These developments have immediate roster implications. Do you try and negotiate with her boss?  What do you do about childcare?

Special idea 5 One of your centre staff reports that a council supervisor refuses to call in staff to provide sand for sandbags, claiming a lack of overtime funds. What are your powers? Are you going to pay for the overtime? Which section of the Act applies? How are your centre staff holding up? Do you need to prepare a new roster?

Special idea 6 A volunteer telephones (role play) the Operations Centre and says that a major evacuation route is blocked by sightseers. The local police constables are on the scene but claim that they have no power to move the sightseers on. What are your powers? How should you react?

Special idea 7 A major flood warning is issued by the Bureau. The river level at Girella Bridge is now approaching 11 metres. Should this trigger a major evacuation response? What is your response? What involvement from the other major services is required? Do you need to advise the Division Controller, the Local Emergency Management Coordinator, others? Is a media release required? Is a door knock required? Where will the forward control points and evacuation centres be?

Special idea 8 The major town is now cut off. The Operations Centre receives a fax from the hotel to say that it has run out of beer and a telephone call to say that a resident is seriously ill and requires urgent medication. The school children will not be able to get to school for a week. Your resources are stretched and some priorities need to be set. How will you re–supply the town? Who pays for the supplies? Do you need a vetting committee? What would be the appropriate membership?

Special idea 9 The water has peaked. Announce an ‘All Clear’. What steps to do you take now? Develop a re–supply plan and roster a skeleton staff to manage the re–supply.

Special idea 10 Pool all your ideas from running the three shifts and prepare a debrief for the District Controller. Choose one person to conduct the debriefing.

Preparing assessment tools for the simulation

The design of an effective simulation should include preparation of relevant assessment tools to record the evidence that is gathered during the exercise.

As the simulation is intended to be a practical demonstration of a management task

The following table has been adapted to provide a checklist for assessors to use as they observe the simulation.

Record of assessment by observation/demonstration
Observation of: A simulation of management of flood emergency in fictional Retnuh River. Assessment of six OC/OCMs rotating through various roles during one–day exercise
During the simulation did the candidate: plan actions and resource requirements by appreciating topography, assessing likely conditions and allowing appropriate lead time? make informed decisions despite lack of adequate time and information? implement decisions and assess outcomes? keep options open when tackling problems? listen actively? remain focused, take considered action and avoid being sidetracked? give and receive constructive feedback? give concise briefings and directions that are readily understood? plan flexibly and prepare to modify plans as necessary? recognise own shortcomings and ask for help at appropriate times? identify strengths of team members and match tasks with individuals? maintain confidence of team members through honest and open communication and through being firm and fair? understand the roles and functions of team members and have empathy with other people's situations and pressures? prioritise tasks, take control of the situation, get the obvious things done before tackling the difficult challenges? make the time to support individuals with a minimum of fuss, recognise normal and abnormal behaviour of team members? maintain credibility with team through applying knowledge of emergency management effectively? avoid taking personal offence during negotiations and discussions? analyse own actions, learn from mistakes, strive for improved response?	Yes/No/NA