Products and Services - Law Enforcement - Validation Studies
Predictive Validation Study Process Model
The validation process includes the following major steps. We would work collaboratively with you to develop and refine this process to meet your needs while also offering our professional expertise and assistance in the collection and analysis of the data.
1. Job Analysis. The preferred method is that CWH Management Solutions creates a job analysis questionnaire to be administered within your department. We request that as many as possible of your incumbents and supervisors complete the questionnaire. While having all incumbents in the department complete the questionnaire is ideal, it may not be practical. As we understand that having incumbents complete the questionnaire may not be practical for your department, an alternative is that we have a committee of SMEs who volunteer to complete the survey. The actual number of incumbents/SMEs will be based on what is practical and achievable for your department while still meeting professional needs and standards. We highly recommend that this step be performed with some incumbents to ensure the legal defensibility and to demonstrate the content validity of the test. CWH Management Solutions will provide all instructions and the master survey. Your staff would be asked to copy and distribute the survey and (confidentially) collect the surveys and answer forms to return to us for analysis.
2. Test Administration to Applicants. CWH will provide either the current or our new version of our Law Enforcement test for administration to applicants. The newest version of the test has been validated statistically in one large Law Enforcement agency with excellent validity and adverse impact results. The advantage of using the new test is also that we will have other agency data to compare and include with the analysis for your department. We understand that the test data will not be used to determine hiring decisions at the time of the first administration(s) of the test. CWH will assist with the process of maintaining appropriate test data and helping to design a process for analyzing this data. CWH highly recommends that we include as many as possible applicants and new hires in this study over the time of the study (approximately 1 year). In other words, we recommend including multiple test administrations and academies in the process, if possible.
3. Collection of various types of performance data. All new incumbents who are hired from the time of the beginning of the study will be tracked throughout approximately 1 year of their employment. CWH can work with you, the department and the training department to design a process by which the department can collect various types of performance data. This data may include (but not be limited to):
a. Exit interviews (with any applicants who quit during the hiring, training, or probation period).
b. Scores or pass/fail rates throughout the remainder of the hiring process, such as Oral Board scores, Physical Ability scores, background checks, etc.
c. Academy performance, including various test scores and overall ratings on all new hires.
d. Performance ratings (supervisor preferred) after a reasonable time of employment and at the end of probation (9 – 15 months). CWH will assist you in designing the performance forms and process.
4. Analysis of Test Data and Performance Data (Validation study). We will use all of the various types of performance data collected to correlate with test performance (as collected in step 2) to provide statistical evidence that the test does in fact predict on-the-job performance. All test data and performance data would be completely confidential and would only be used to validate the test by correlating job performance and test performance. We would request that we have scores from the test that was used to hire applicants also in order to make a comparison between the two hiring methods or tests. CWH will work collaboratively with you to design this process and to assist in any way that we agree is helpful to you with the analysis and reporting of results.
If the new test version is used, it could then be modified if necessary, and we would recommend the components to be used in your department, as well as the appropriate scoring model. This test would then be ready to use in your candidate selection process in the future.
1. Job Analysis. The preferred method is that CWH Management Solutions creates a job analysis questionnaire to be administered within your department. We request that as many as possible of your incumbents and supervisors complete the questionnaire. While having all incumbents in the department complete the questionnaire is ideal, it may not be practical. As we understand that having incumbents complete the questionnaire may not be practical for your department, an alternative is that we have a committee of SMEs who volunteer to complete the survey. The actual number of incumbents/SMEs will be based on what is practical and achievable for your department while still meeting professional needs and standards. We highly recommend that this step be performed with some incumbents to ensure the legal defensibility and to demonstrate the content validity of the test. CWH Management Solutions will provide all instructions and the master survey. Your staff would be asked to copy and distribute the survey and (confidentially) collect the surveys and answer forms to return to us for analysis.
2. Test Administration to Applicants. CWH will provide either the current or our new version of our Law Enforcement test for administration to applicants. The newest version of the test has been validated statistically in one large Law Enforcement agency with excellent validity and adverse impact results. The advantage of using the new test is also that we will have other agency data to compare and include with the analysis for your department. We understand that the test data will not be used to determine hiring decisions at the time of the first administration(s) of the test. CWH will assist with the process of maintaining appropriate test data and helping to design a process for analyzing this data. CWH highly recommends that we include as many as possible applicants and new hires in this study over the time of the study (approximately 1 year). In other words, we recommend including multiple test administrations and academies in the process, if possible.
3. Collection of various types of performance data. All new incumbents who are hired from the time of the beginning of the study will be tracked throughout approximately 1 year of their employment. CWH can work with you, the department and the training department to design a process by which the department can collect various types of performance data. This data may include (but not be limited to):
a. Exit interviews (with any applicants who quit during the hiring, training, or probation period).
b. Scores or pass/fail rates throughout the remainder of the hiring process, such as Oral Board scores, Physical Ability scores, background checks, etc.
c. Academy performance, including various test scores and overall ratings on all new hires.
d. Performance ratings (supervisor preferred) after a reasonable time of employment and at the end of probation (9 – 15 months). CWH will assist you in designing the performance forms and process.
4. Analysis of Test Data and Performance Data (Validation study). We will use all of the various types of performance data collected to correlate with test performance (as collected in step 2) to provide statistical evidence that the test does in fact predict on-the-job performance. All test data and performance data would be completely confidential and would only be used to validate the test by correlating job performance and test performance. We would request that we have scores from the test that was used to hire applicants also in order to make a comparison between the two hiring methods or tests. CWH will work collaboratively with you to design this process and to assist in any way that we agree is helpful to you with the analysis and reporting of results.
If the new test version is used, it could then be modified if necessary, and we would recommend the components to be used in your department, as well as the appropriate scoring model. This test would then be ready to use in your candidate selection process in the future.
Department Resources Required
Your staff would be asked to help with project management by coordinating meetings and interviews, copying and distributing materials, and scheduling and administering test sessions and questionnaires. We estimate that this would take approximately 80 – 100 hours of staff time.
In addition, as many as practical incumbents would be asked to complete the job analysis surveys (Step 1). This requires 1 – 2 hours per person. All applicants would be asked to complete the test (Step 2). This would require about 3 - 4 hours per applicant. The collection of performance data would require approximately 1 – 2 hours for each supervisor, peer, or new hire who participates in the process.
In addition, as many as practical incumbents would be asked to complete the job analysis surveys (Step 1). This requires 1 – 2 hours per person. All applicants would be asked to complete the test (Step 2). This would require about 3 - 4 hours per applicant. The collection of performance data would require approximately 1 – 2 hours for each supervisor, peer, or new hire who participates in the process.
Information requested
This information helps put together an estimate for expenses and helps us plan the project and trips. This also allows us to help our clients plan how much time will be required for participation from incumbents, and how many incumbents should be involved in the project. We request this information in an Excel or other electronic file if possible.
• Number of current incumbents in each position and/or rank
• Badge No. or other ID # for each incumbent
• Social Security Number of all applicants
• Ethnicity and Gender of incumbents and applicants
• Tenure of incumbents / Start date of applicants
• Work Assignment of new hires
• Station of new hires
• Shift of new hires
• Performance data as described above for all new hires from selection process, academy, and on the job
All new hires (applicants who tested and were hired) will participate in the project. This enables us to have a sample size large enough to truly compare the results of the different testing methods (or tests). This should include representation from all of the categories above.
• Number of current incumbents in each position and/or rank
• Badge No. or other ID # for each incumbent
• Social Security Number of all applicants
• Ethnicity and Gender of incumbents and applicants
• Tenure of incumbents / Start date of applicants
• Work Assignment of new hires
• Station of new hires
• Shift of new hires
• Performance data as described above for all new hires from selection process, academy, and on the job
All new hires (applicants who tested and were hired) will participate in the project. This enables us to have a sample size large enough to truly compare the results of the different testing methods (or tests). This should include representation from all of the categories above.
