Inter-assessor agreement of portfolio-based competency assessment for orthotist/prosthetists in Australia: a mixed-method study

Dillon, Michael P.2; Puli, Louise1; Ridgewell, Emily1; Anderson, Sarah P1; Chiavaroli, Neville3; Clarke, Leigh1

1Australian Orthotic Prosthetic Association, Camberwell, Australia, 2La Trobe University Australia, Melbourne, Australia 3Australian Council for Educational Research

Background
Internationally qualified orthotist/prosthetists who want to practice in Australia must pass a portfolio-based competency assessment. Testing the agreement between independent assessors is important to engender confidence in the assessment, and continually improve processes.

Objectives
The aim was to quantify inter-assessor agreement for all 68 performance indicators in the Australian Orthotic Prosthetic Association’s (AOPA) Entry Level Competency Standards (ELCS), and where there was significant disagreement between assessors, to explore the reasons why.

Study design
Mixed methods: explanatory sequential

Method
Fifteen portfolios were randomly assigned to two independent assessors. Assessors determined whether the evidence presented did or did not meet the requirements of each performance indicator. Inter-assessor agreement was calculated using Gwet’s Agreement Coefficient 1 (AC1). Semi-structured interviews explored the reasons for disagreement.

Results
Most performance indicators (87%) had moderate to substantial agreement (AC1 > 0.71) which could be attributed to a variety of factors including the use of a simple assessment rubric with supporting guidelines, and assessor training to establish shared expectations. The remaining performance indicators (13%) had fair to slight agreement (AC1 ≤ 0.7). There are opportunities to refine the wording of performance indicators to reduce ambiguity, as well as revise guidelines to help applicants carefully curate the most appropriate evidence for each performance indicator; among other recommendations to improve reliability.

Conclusion
While most performance indicators in AOPA’s ELCS were associated with moderate to substantial inter-assessor agreement, there are opportunities to improve agreement for a proportion of the performance indicators using the recommendations from this research.


Biography:

Louise Puli is an orthotist/prosthetist and public health professional holding a Master of Public Health (policy & management), Bachelors of Prosthetics and Orthotics (hons) and Graduate Diploma in Higher Education Curriculum, Teaching and Learning.

Louise has significant experience developing and implementing professional standards, including standards of; practice, certification and education, in the regulation of assistive technology professions.

Louise is also an experienced educator and has been involved in the education of orthotist/prosthetists in over 25 countries, contributing to a range of programs including specialised courses, bachelor, and master degree programs. She spent several years based in East Africa at the Tanzanian Training Centre for orthotist/prosthetists (TATCOT) and is passionate about increasing global access to assistive technology.

Currently, Louise is working with the Access to Assistive Technology team to improve access to assistive technology for people living with noncommunicable diseases and is contributing in other areas related to AT standards and education.

Print Friendly, PDF & Email
Recent Comments
    Categories