Are Your Training Effectiveness Methods Really Effective?
ISO 13485:2016 section 6.2c requires that with regards to training, an organization evaluates the effectiveness of the actions taken. The methodology used to check effectiveness is proportionate to the risk associated with the work.
I’m convinced that many 483s and Warning Letter stem from weak training programs. If done properly, evaluating training effectiveness will identify potential pitfalls within the training program. Let’s look at two common methods of verification—Witnessed Observation and Tests of Understanding. Although these methods seem relatively easy to implement, many organizations fail to provide sufficient objective evidence to demonstrate that training is effective and employees are qualified to perform their job functions.
Witnessed Observation
I reviewed training records during a recent audit. A QA associate provided group training for a manufacturing work instruction that was newly released. The content addressed cosmetic defects and the use of a TAPPI chart to measure the size of cosmetic defects. During our discussion, the trainer stated that he had reviewed the work instruction with the employees, demonstrated how to use the TAPPI chart and then verified that each employee used the chart correctly. The problem was that there was no objective evidence to support that he observed each employee using the TAPPI chart correctly. Reminder: If you didn’t document it, you didn’t do it.
If you use witnessed observation as a method for evaluation of training effectiveness, provide sufficient documentation. Here are some tips:
- Document the details:
- Name and title of the employee who was witnessed
- Name and title of who witnessed the activity
- What exactly was witnessed—was it a specific skill (i.e, soldering, pipetting) or was it the execution of a procedure? Provide the applicable information related to the procedure (title, document number, rev. level, and particular section if not the procedure in full).
- Date(s) of the witnessed operation
- Product-related information including lot numbers and quantities
- Equipment-related information including asset number and description
- Other applicable information (i.e., DHR/work order #, PO numbers, applicable standard, etc.)
- State the results
- Was training found to be effective or did the employee require re-training or coaching?
Let’s look at an example:
Mark Smith is hired as a Cleanroom Operator 1 by ACME Medical Device Company. On his first day of work, he reads the Cleanroom Gowning Work Instruction WI-005 rev. 4 and signs the training form. After this, his supervisor demonstrates the gowning/de-gowning and handwashing sequence. He then observes Mark performing these activities before they enter the cleanroom and after they exit. At the end of the day the supervisor checks off the Training Verification of Effectiveness box on the training form and forwards the form to HR for filing.
Was this sufficient? Not likely. I’ve written numerous NCs against cleanroom gowning and behavioral violations. The most egregious happened when I was in a conference room that had windows so you could see the cleanroom manufacturing area. While I was waiting for the QA manager to retrieve records I requested, I observed the following violations:
- Chewing gum
- Hair not properly covered
- Gown sleeves rolled up
- Gloves not worn
- A non-gowned employee entered the cleanroom
Getting back to our example, it’s important to note that training was “one and done”. Mark was observed immediately after training and there were no employees in the gowning area other than Mark and his supervisor. If you’ve ever been in a cleanroom gowning area during breaks or shift changes you know that it’s chaotic with a lot of distractions. This can be especially nerve-wracking for a new hire.
A better approach would be to observe Mark several times over an extended period (i.e., 30 days) in both the gowning area and the cleanroom.
Tests of Understanding
Some organizations seem hell-bent on making these tests as easy as possible to ensure passing grades for everyone. One memorable example is the company whose Test of Understanding consisted of a listing of quality system procedures and the employee was required to record the applicable document number. Each employee, regardless of job function, was administered this test.
“Best Practice” Points to Consider:
Use a test as a tool to accurately measure an employee’s understanding of the subject matter. Testing is meaningless if the employee is not required to think before responding. Here is an actual test question to demonstrate this point:
Question 3:
What document best defines supplier management requirements?
- Supplier Management Procedure QAP-003
- Employee Training Procedure QAP-007
- Packaging and Shipping Work Instruction WI-015
- Risk Management Procedure QAP-013
I offer that anyone who cannot answer that question correctly should not be working in the medical device industry. In any capacity.
- Define the passing score for each test before it is administered. This can be referenced in the training procedure or within each test record. Make it reasonable, 80% or higher is generally the rule. Depending upon the subject matter/risk, it may be appropriate to require a score of 100%.
- Make the test an appropriate length and comprehensive in nature. Again, go back to the risk. For example, 2 questions related to the ESD program is insufficient, given the potential impact on the performance of a medical device if procedures are not followed. For a cleanroom test, don’t focus on just gowning/de-gowning. The test should cover all aspects of cleanroom requirements/behavior.
- Define the required actions when a passing score is not achieved. Can the employee immediately re-take the test or is re-training necessary? Does the employee take the same test or a different one? How many times can an employee re-take the test?
- Make sure that the test includes the employee’s name, date, grader’s name, and score.