Part 2 Step-by-step plan

2.1 Step 1: Preparation

Start by formulating the purpose of the item bank. Consider the role of the item bank within the teaching process, phasing (where applicable) and ownership of the item bank. Who will you need? How can you gradually build an item bank that adds value to the teaching process and will stay ‘fresh’ for longer? See H3.1.

Consider whether the development of an item bank is in line with the policies of the institution and whether it supports the educational vision. Does it fit in with the digital assessment policy? What are the qualitative and quantitative goals that your institution wants to achieve through digital assessment? Make sure that the overall purpose of your item bank is in line with this policy. This will ensure that the item bank stands a greater chance of success. Commitment in the form of time and money will be required before you can elaborate an idea. Each institution has different procedures for obtaining this commitment. This often involves administrative procedures in which various people will discuss things or lobby at operational, tactical and strategic levels. There are usually guidelines and procedures available for launching a project within an institution. Examples include innovation funds that can be made available through teaching awards or incentive measures. For example, you may need to formulate a project launch memo or project description to help arrange the resources. An initial cost-benefit analysis may already be needed at this stage.

It is advisable to define and describe at least the following aspects while you are laying the groundwork:

  • The current situation and the desired situation.
  • The purpose of the item bank. What makes it a good idea? What is the benefit for students? Make clear how the item bank relates to other policy documents, such as the digital assessment policy and the teaching policy.
  • What outcome will be visible when the project is complete? What maintenance and management will be needed in broad terms?
  • What is the theory behind the development of the items? In the groundwork stage, decide on the theory you will be working on: classical test theory (CTT) or item response theory (IRT). What types of question do you want to use and how will feedback be processed? Decisions on feedback and question types will affect the final outcome. See also § 3.6.3.1, § 3.6.3.2, and § 3.7.3.
  • How much time, money and lead time will be needed for the project?
  • Work out how feasible the project is: if necessary, carry out a separate feasibility study or preliminary study .
  • Decide who the project initiator and/or the ultimate owner of the item bank is. Decide on who you will need to help develop the item bank. See also H3.5.

2.1.1 Feasibility study

A possible intermediate step in laying the groundwork for your item bank project is a feasibility study. This will explore and elaborate one or more options for organising an item bank. Choose the option that suits your situation best. Consider the following aspects: * Content: What do the items you want to include in the item bank have in common? Is there a common arrangement that can be identified? To what extent do lecturers believe they can reach a consensus on the course content and the items? Knowing this will enable you to decide on your approach. More time will be needed for this part if there are differences of opinion. See also H2.3.

  • Quality: how would you rate the quality of the tests at the moment? What are the p-values and trip-values? How acceptable is this? What improvements do we envisage making with the item bank?
  • The choice to buy in or develop the items yourself.
  • What item banks are already available in the market? Check solutions available outside the institution, such as buying items from an external commercial party. * Is there an option to join an existing partnership that is already developing an item bank?
  • The intended size of the item bank.
    • How many items are already available for use within the institution?
    • How many items will you need? See also H3.3.
  • Technology (assessment systems and item bank systems): Check what item bank systems are already available within the institution and item banks that are available for use. Find out which department is responsible for the management and use of ICT applications and, specifically, assessment applications. Is there anything that the institution needs to buy? See also H3.2.
  • Organisation: is there sufficient support for these option(s)? Talk to organisations that have experience with organising item banks.
  • Funding: detail the costs and benefits for each option and consider the following aspects.
    • Can sufficient time and money be made available for the development of the item bank?
    • To what extent will the item bank be able to improve the price-quality ratio (‘value for money’) of items?
    • Can you raise project funding or subsidies? See also H3.4.

2.1.2 Completion of groundwork stage

Process all the data you have in a written project description and obtain approval from the project initiator to start writing an action plan.

What will you have achieved by the end of this step?

  • Outcome: insight into the purpose, feasibility and overall approach of the idea.
  • A go/no go from your project initiator for the next step.

2.2 Step 2: Action plan

In this step, you will decide how the item bank should look once the project is complete. You will make an action plan to achieve this result.

Ask yourself: what will we have achieved by the end? The answer will depend on the purpose that the item bank will fulfil in the teaching process. Formulate the relevant requirements (hard and soft) that the item bank must meet. Hard requirements are those that absolutely must be met; soft requirements are those that people want to achieve, but are not strictly necessary to achieve a satisfactory outcome. Include both in your action plan.

In some cases, an organisation may facilitate project-based working and even a specific project management method, such as Prince II. Project leaders may even be available to oversee larger projects. In that case, there will be established procedures in place within the institution regarding working methods, phasing and decision-making. Check what the institution offers in terms of project-based working. Use standard documents wherever possible, such as a template for an action plan. Set out steps from the step-by-step plan in the action plan.

Make sure that your action plan describes at least the following aspects in as much detail as possible:

  • The purpose and outcome.
  • How you want to achieve the outcome.
  • The role of the project initiator and who this will be. Ideally, the project initiator will be the future owner of the item bank. Plan sessions to discuss the approach and the plans with this person. Decide what you will need his or her help with.
  • Think about who you will need during the project and once it’s complete. Do they already have the right knowledge and skills? Consider the following areas of expertise: client, project initiator, assessment expert, psychometrist, functional administrator, legal counsel and lecturers.
  • Decide how you will create support. How will you ensure that the item bank is actually used? How will you encourage lecturers and students to put the new working methods into practice? Talk to them about this. Consider how you can ensure the importance of the item bank is placed on the tactical and strategic agendas. Who are the stakeholders?
  • Costs and benefits: provide an estimate of what the development of the item bank will cost and what its benefits will be in the years ahead.
  • Quality: how will you ensure you produce an item bank of sufficient quality?
  • Timeline: what phases do you have in mind? What lead time will you need?
  • Consider how you can achieve the objective you have set, and how you can control the process of getting there and making adjustments as needed.

What will you have achieved by the end of this step?

The outcome of this step is an action plan in which the purpose, the outcome and how you intend to achieve this are defined as specifically as possible. Invite the project initiator to approve the action plan. The action plan ensures that the time that people will need has been agreed with their superiors. This will ensure commitment and support.

2.3 Step 3: Design

By the end of this step, you will have a design that makes it clear to the project initiator what deliverable they can expect to receive. What’s more, the employees involved in the next step, testing in a pilot, will know what needs to be done.

The following aspects are defined and described in the design:

  • Arrangement of items
  • Item development process
  • Item quality
  • System
  • Production and planning
  • Funding
  • Legal aspects that need addressing
  • Management

Clearly document the arrangements that are agreed for each focus item. The outcomes of each component add up to the overall design of the item bank and are therefore also interrelated.

2.3.1 Arrangement of items

Map out the extent of commonality between the disciplines. Invite the lecturers concerned to decide whether and, if so, to what extent they view a subject or discipline in the same way. They will work together in-depth to decide what is included in the item bank at the subject level and the question construction level. This phase is a difficult one because it is here where differences will come to light. It is important to remain pragmatic. Accept that some adjustments may be necessary in the teaching process, for example, you may have to choose a new teaching method or modify the use of specific names for variables or concepts.

  • Questions that come up: How do you describe the items? What do you consider relevant or programme-specific? What do the test matrices look like?
  • There may be differences in notation or illustration. How do you deal with this?
  • Decide what hierarchical structure the item bank will have and what metadata is desirable. See also H3.6.

2.3.2 Item development process

  • Decide who will run through which steps in the item development process to ensure the assessment quality.
  • Prepare an estimate of the expected development time per item.
  • Consider the current process steps. Fine-tune these or expand them if necessary.
  • Do existing items need to be reviewed and approved? How much? See also H3.5.

2.3.3 Item quality

  • Decide on and document guidelines that items must comply with. Consider also rules on language use and citation of sources.
  • Wherever possible, involve item experts (not everyone makes good items) or a psychometrician.
  • Plan review sessions led by a question expert/coach. Invite lecturers to submit a large number of existing sample items. Together, discuss the variation between the disciplines, the manner in which the questions are set and the construction errors in these items. This will give the stakeholders ideas on standardising the formulation of questions and question formats.
  • Consider how you can measure the desired quality improvement.
  • Start by thinking about the outcome: identify the data you want to be able to retrieve from the item bank for your educational process. See also H3.7.

2.3.4 System

  • For a simple item bank project: see which assessment system the institution uses. Does the system support the envisaged item development process, roles and privileges, hierarchical structure and metadata?
  • For a complex item bank project: if there are multiple systems in use, decide which system is most suitable for the purpose.
  • Does the envisaged item development process fit the system? You will no be able to calculate how much work it is to organise. If it does not fit, decide whether you can modify the process, or whether you would prefer to investigate whether the structure of the system can be modified.
  • State what is needed to ensure the safety of the items in the bank.
  • How scalable should the system be? Not just now, but in the future too – as far as you can tell? See also H3.2.

2.3.5 Production and planning

  • In the groundwork phase, you will have roughly determined how many items you will need in the item bank. Convert this number into time (short and long term). How much pressure will this put on employees?
  • How many items are already available from the various stakeholders, and can these be contributed with relative ease?
  • Think about how you will approach filling the item bank. Determine the period, any deadlines (when will the first tests take place?) and target production by scaling up (batches).
  • Decide when to conduct quality reviews based on assessment statistics and psychometric data. See also H3.3.

2.3.6 Funding

  • Further elaborate the cost-benefit analysis from the previous steps. Consider how the initial development will be financed.
  • Determine how you will fund the operation of the item bank once the project has run its course. Prepare an operating plan. Opportunities to generate revenues include:
  • Institutions take out a subscription to the item bank, which they then offer to students free of charge.
  • Students pay for access to the item bank.
  • Involve the project initiator of the item bank when preparing the operating budget. Ensure that he or she feels responsible for this. See also H3.4.

2.3.8 Item bank management

  • Think about how you can best support the users, such as lecturers and students, with getting the most from the final product. Talking to them and listening to their arguments will help here. Invite them to contribute their ideas on the design.
  • Decide on who or what you will need to manage the item bank. Who will you need to provide support and administration once the project is concluded?
  • Create instructional materials and plan instructional sessions.
  • Talk to the testing and management organisation, where relevant, to coordinate the working method. See also H3.5.

What will you have achieved by the end of this step?

A design for the item bank. All interim outcomes for each component will add up to the overall design of the item bank:

  • Structure: a map/table with the agreed subject structure and arrangements regarding notation methods, for instance.
  • Organisation: approach and documented organisational arrangements (description of the testing and administration organisation) on how the item bank will be managed, including governance of the item bank, including the necessary knowledge and skills of the parties concerned (a training plan).
  • System: organisational plan system; this concerns the workflow (of the item development process) and the meta structure (hierarchical structure and metadata structure).
  • Quality:
  • agreed arrangements on the envisaged quality of the items, including a description of guidelines and criteria that the items must comply with.
  • description of the working procedure for item development, including management, analysis and roles and privileges.
  • an indication of the time it will take to develop an item.
  • an indication of the increase in and consistency of the item quality.
  • Production and planning: approach and documented organisational arrangements on how to ‘fill‘ the item bank and keep it up to date.
  • Financial: a business case and operating plan.
  • Legal: agreed arrangements on the legal aspects of the items and the system

2.4 Step 4: Pilot

The design is complete. Now set up the item bank and prepare it for use. In this step, you will set up an item bank structure, enter metadata categories, create accounts, convert existing items and assign roles and privileges.

Run a pilot in which you are fully operational for an initial assessment period. In the pilot, you will be testing whether the chain of activities works. Create enough items so that you can cover the initial assessment period. Check whether the agreed arrangements are feasible in practice. Are the assumptions correct? Does everyone know what roles and privileges they have? Test whether the systems and the method work from a technical perspective. Are the items used by students as intended? Is formative assessment used? Evaluate and adjust as necessary.

The following activities will be carried out during the pilot:

  • Set up the item bank from a technical perspective and test it.
  • If old items are to be converted to the new situation: perform the conversion with a small number of items.
  • Train employees.
  • Enter a first batch of items based on the agreed working method
  • Prepare a test using the available items.
  • Offer the test in a course or module.
  • Evaluate the use and results of the test.
  • Establish an administration system for people and funding (for example, if a fee per constructed item or for coordinating role has been chosen).
  • Evaluate the use and procedures for creating items. Is there a better way to control quality? Are the assumptions correct? For example, the estimated time for question development?
  • Where necessary: make adjustments, fine-tune the arrangements made, modify the assumptions.

What will you have achieved by the end of this step?

  • A documented evaluation of the pilot in terms of the various design aspects and from the perspective of the users, and an improvement plan.
  • An established and tested item bank.

2.5 Step 5: Real-life deployment

In this step, you will use the item bank in real-life situations. You will also take action to increase the use of the item bank.

The pilot will have generated a number of lessons. Based on these, you will be able to scale up. During the real-life deployment, larger numbers of items will be created, and more and more tests will be deployed.

Points to consider:

  • Ensure that the item bank owner continues to follow and monitor the objectives and outcomes.
  • Regularly discuss the topic of embedding with the item bank owner. Are the item developers creating sufficient items? Are the items of good quality? How can you ensure the least possible variation in quality? Is everyone satisfied with the process? Is it easy for lecturers to create tests? Do students consider the formative items developed in this way (including feedback) appealing?
  • Check the ongoing viability of the business case. See also H3.7.

2.5.1 Aftercare

The item bank has been implemented. The process is operational, everyone is performing their tasks and using the item bank in the agreed way. There is a formal owner who monitors the purpose of the item bank. The item bank has found its place within the organisation: there are people who ensure the item bank’s continued existence and proper use. The added value for the teaching process is clear for now and in the future.

Points to consider:

  • Ensure that the maintenance of the item bank is carried out, for example, by a functional administrator and the editor(s).
  • Ensure that the financial administration functions as intended throughout the operating phase. Keep an eye on whether operation will provide an appropriate level of funding in the future.
  • Ensure proper documentation is available, such as:
  • a description of how the item bank will be used in a course.
  • a description for testing and management of how the item bank will be maintained.
  • a description for the coordination team, where applicable, and for each team of authors for each item bank of how the item bank will be maintained. See also § 3.7.2.3.
  • a standard instruction programme for use of the item bank system for new employees.
  • changes to the instruction programme when the item bank system is updated or amended.