Comparing the Borich Model with The Ranked Discrepancy Model for Competency Assessment: A Novel Approach

In 1980, Borich presented a new model that allowed errors in an individual’s judgment of selfproficiency to be offset by considering the perception of a group. The model relied upon the calculation of means for competency items measured with ordinal scales, an approach subject to debate in modern times. The purpose of our study was to explore the use of a novel approach we developed, the Ranked Discrepancy Model (RDM), as an alternative method to the Borich model for determining training needs. Data obtained from an online survey of extension professionals employed by a land-grant university in the United States was used to compare the training needs identified by applying the Borich model with those identified by applying the RDM. A very strong and statistically significant correlation existed between the scores derived from using both models, demonstrating a high level of consistency between models. Researchers conducting competency research should consider adopting the RDM given its suitability for delivering results that closely resemble findings from the Borich model while providing improved rigor in methods and increased detail about training needs.


Introduction and Problem Statement
In 1980, Borich introduced a new model for assessing educator training needs. Borich's (1980) use of the group mean to describe the perceived importance of a competency and weight an individual's proficiency gap revolutionized how training needs were identified by accounting for errors inherent in an individual's judgment of what was important to know or do. Agricultural and extension education practitioners and scholars alike embraced the Borich model (e.g., Elhamoly et al., 2014;Umar et al., 2017;Waters & Haskell, 1988), and we count ourselves among its many adopters. However, usage of the Borich model is worth reflecting upon more than forty years later to determine its appropriateness for contemporary needs assessment research.
One reason to revisit the Borich model is due to an unsettled debate over the use of means to describe items measured on ordinal scales. Means derived from individual ordinal items are an inherent part of calculating the mean weighted discrepancy scores (MWDS) needed in the Borich model. Arguments can be found for and against using means of ordinally-scaled items (e.g., Boone & Boone, 2012;Norman, 2010). The controversy over using means for individual ordinal items impacts the potential acceptance of research conducted using the Borich model within the broader scientific community. A new analytical method is needed to help researchers identify competency training needs efficiently and avoid getting caught in the ordinal mean debate, while preserving the underlying rationale of the Borich model.

Theoretical and Conceptual Framework
The Borich model is primarily used to determine priority competencies for professional development (Borich, 1980) and is appropriate for use when assessing the needs of a sample, such as extension professionals or agricultural teachers. The Borich model relies on identifying gaps -called discrepancy scores -between a respondent's perceived ability to perform a particular competency and how important that competency is for job success. The discrepancy score is the difference between how a respondent rates their ability and importance using ordinal scales. In a Borich assessment, a discrepancy suggests an individual does not have sufficient ability to perform an important competency, therefore, a gap exists between the ideal and current conditions. A deviation between an ideal and current condition, or what should be and what is, represents the underlying nature of a need as described by Witkin and Altschuld (1995). For example, a respondent may rate a competency as having above average importance for their job success, but self-report having below average ability. This calculation is analogous to identifying gaps in a quantitative needs assessment process; a gap or discrepancy exists when a current condition is less than an ideal condition (Boyle, 1981;English & Kaufman, 1975;Witkin & Altschuld, 1995).
Next, the Borich model uses the perceptions of the sample to estimate a competency's actual importance (Borich, 1980) by calculating the sample mean for importance. This approach helps to overcome individual errors in judgment. Each respondent's discrepancy score is weighted by multiplying it with the sample mean for importance for a given competency, resulting in an individual's weighted discrepancy score. Finally, a MWDS is calculated by averaging the weighted discrepancy scores for the entire sample. The MWDS are then used to determine training needs, with positive scores indicating a need for intervention and negative scores indicating a need does not exist since ability exceeds importance.
Herein lies two problems with the Borich model. First, the weighted discrepancy score is dependent on the use of item means for importance. Recall that a respondent provides a single judgment of the importance of a competency based on an ordinal scale, resulting in the use of the controversial ordinal scale means (Kuzon et al., 1996). While suitable for reliable psychometric constructs consisting of multiple normally distributed ordinal items, assessing the mean value of a single ordinal item is generally not recommended by researchers (Dillman et al., 2014;Sullivan & Artino, 2013). Second, the MWDS for each competency does not follow an immediately clear and interpretable standardized range. A MWDS ranges from -4 to 20 when using a 5-point response scale. However, any time a researcher opts to use anything except a 5point response scale (e.g., a 7-point importance scale ranging from Not at all important to Extremely important), the MWDS range changes, creating difficulties in comparing competency needs across different studies. Valid reasons exist for using scales with varying numbers of response anchors. Dillman et al. (2014) stated a unipolar scale such as the type used for assessing competency training needs would be acceptable with only four scale points and this format would decrease the burden on the respondent. Conversely, Preston and Colman (2000) found the scores obtained from using 7-to 10-point scales were more reliable, had improved criterion validity coefficients, and better discriminating power as compared to scales with fewer points. If Borich findings can only be compared across studies when researchers use a 5-point response scale, then this is a limitation of the Borich model.
We propose use of the Ranked Discrepancy Model (RDM) as an alternative to the Borich model. Application of the RDM is only appropriate when certain conditions exist: (a) cross-sectional data (Ary et al., 2014) is gathered from a sample or census of a target population at one point in time, (b) data for each variable or item is paired on two ordinal scales with an equivalent number of response anchors, and (c) the objective is to assess discrepancies between two clearly identified states or conditions for each item. These conditions are also necessary for the application of Borich's (1980) model for determining training needs.
The RDM circumvents the two major drawbacks of the Borich model. As a descriptive approach, the RDM avoids the use of means for single items measured with ordinal scales (i.e., individual competency items). It also provides an intuitive standardized score that represents the discrepancy or gap in ability compared to a known state of equilibrium, which is consistent with early needs assessment literature, namely Lewin's (1939) field theory of motivation. There are three steps in the RDM. An illustrated step-by-step example is included after the findings. First, calculate the number of occurrences in the sample when respondents' ability ratings are: (a) less than respondents' importance ratings (Negative Ranks = NR), (b) more than respondents' importance ratings (Positive Ranks = PR), or (c) equal to respondents' importance ratings (Tied Ranks = TR). Second, convert the number of occurrences for NR, PR, and TR into percentages. Third, assign relative weights (W) to NR% (WNR = -1), PR (WPR = 1), and TR (WTR = 0). The resulting Ranked Discrepancy Score (RDS) is a standardized score ranging between -100 to 100. The RDS has an equilibrium of 0, with negative scores indicating a priority need or discrepancy in ability, and positive scores indicating the absence of a gap or need.
Like the MWDS, the RDS provides a snapshot of the professional capacity of an organization with respect to a competency area. Therefore, a negative, equilibrium, or positive RDS does not imply every individual of the sample has inadequate or adequate capacity to perform a specific competency. It assesses priority professional development needs of the sample as whole by accounting for the ability of all individuals within the sample. Like Borich (1980), the RDM does not take a deficit approach to identifying needs by only considering those with negative ranks. In addition, the RDM does not rely on a sample mean for importance, instead, it capitalizes on the frequency distribution of each item. This approach is widely regarded is an appropriate way to handle ordinal items (Sullivan & Artino, 2013), even with nonnormally distributed data. The frequency distribution of importance and ability ratings directly influences the RDS via NR%, PR%, and TR%. The RDM is intended to be an intuitive approach to handling paired needs assessment data. The RDS represents the severity of a need and allows for direct comparison and priority ranking between competencies.

Purpose
The purpose of the study was to explore the use of the RDM as an alternative method to the Borich model for determining training needs. The objectives were to: 1. Describe the unweighted rank frequencies and RDS for program planning and program evaluation competencies. 2. Compare MWDS and RDS for program planning and program evaluation competencies. 3. Describe the relationship between scores resulting from the application of the Borich model and RDM.

Methods
Our study used Borich-type data from a competency assessment conducted at the University of Florida in 2021. A convenience sample of county agents was taken by surveying those who were registered (N = 276) for an annual professional development symposium. With a 58.30% response rate, the sample consisted of 161 individuals (n = 161). However, an examination of the dataset revealed several incomplete responses and/or majority missing values. Partial responses were removed from the dataset, and the final usable sample was 122 county agents (n = 122, 44.20%). A small number of missing values (< 1% of observations) in the final dataset were determined to be missing at random. For each competency item, missing values were replaced with the corresponding sample mean for that item to maintain the initial distribution properties of the data (Dodeen, 2003).
Survey data were gathered using a researcher-made questionnaire (Ary et al., 2014). The questionnaire consisted of a list of Extension core competency items related to program planning and evaluation. Selected items were consistent with previous studies in the subject area (e.g., Harder et al., 2010;Lakai et al., 2012;Lindner et al., 2010;Maddy et al., 2002;Narine & Ali, 2020;Scheer et al., 2011;Suvedi & Kaplowitz, 2016). There were 17 competencies in program planning and 15 competencies in program evaluation. Following Borich's (1980) approach, respondents were first asked to rate their ability to perform each competency using a 5-point ordinal scale with the following options: 1 = None, 2 = Below Average, 3 = Average, 4 = Above Average, and 5 = Exceptional. Following, respondents were asked to rate the degree to which each competency was important to their job success using a 5-point ordinal scale with the following options: 1 = None, 2 = Below Average, 3 = Average, 4 = Above Average, and 5 = Essential. Data were analyzed using the Borich model and RDM to compare findings. Complete details about how to calculate training needs according to Borich (1980) can be found by visiting the original reference.
Negative Ranks (NR), Positive Ranks (PR), and Tied Ranks (TR) were calculated using IBM SPSS Statistics (Version 27) by performing the Wilcoxon rank test between paired responses (i.e., observations for ability and importance) for each competency item. The first variable entered in the option window was perceived importance, then paired with self-assessed ability. An automatic output table with NR, PR, and TR was generated. Then, rank values (i.e., number of occurrences for NR, PR, and TR) from the SPSS output table were exported to Microsoft Excel to perform Steps 2 and 3. After finding the number of occurrences of NR, PR, and TR for each competency item, the next step was to convert the three rank counts into percentages. The final step was to apply weights to NR%, PR%, and TR%. The RDS was calculated as follows: RDS = NR% (-1) + PR% (1) + TR% (0). In practice, the last expression in the equation (TR% [0]) naturally drops off, leaving RDS = NR% (-1) + PR% (1). However, TR is important to the model as it affects percentages for NR and PR.
With weights applied, the RDS equals -100 if all individuals have a negative discrepancy in ability relative to a competency's importance (i.e., 100% NR). A negative RDS trending to -100 reflects the magnitude of the discrepancy for one competency item and is directly comparable to the RDS for other competencies. In contrast, the RDS will equal (+) 100 when all individuals have a positive discrepancy in ability relative to a competency's importance. As mentioned, the RDS should be interpreted as representing the overall capacity of the sample to perform a competency; it indicates the needs of the sample as a whole. Table 1 provides the unweighted rank frequencies used to calculate the RDS for program planning competency items. After applying weights to NR (-1), PR (1), and TR (0), the RDS shows discrepancies in each item from the point of equilibrium (0). All items in Table 1 had a negative RDS, indicating a gap in ability to perform all program planning competencies. The RDS also shows the magnitude of the gap since all items are directly comparable on a standardized score between -100 and 100. Based on the results, the top three priority competency items for attention are to (a) develop long-term program objectives (RDS = -61), (b) conduct a needs assessment for your program (RDS = -60), and (c) use the results of a needs assessment for planning (RDS = -57).  Table 2 shows the unweighted rank frequencies and RDS for program evaluation competency items. Based on the RDS, there was a gap in ability to perform all competency items related to evaluation. The largest discrepancy identified by the RDS was to "conduct follow-up surveys to measure behavior change" (RDS = -58).  Table 3 provides a comparison between the Borich model and RDM for program planning competency items. Both models confirmed a discrepancy in ability for all program planning competencies. In Table 3, competency items were ranked based on the discrepancy in ability, which is translated as priorities for professional development; a positive MWDS and negative RDS indicate a need for professional development. In Table 3, the top three priority competencies were the same for the Borich model and RDM. Further, nine of the top 10 items were equivalent across models, with the only exception being "align program priorities at the local level with the Extension Roadmap." This item was ranked 11th in the Borich model and 8th in the RDM. Meanwhile, "develop short-term program objectives" was ranked 10th in the Borich model and 12th in the RDM. Lastly, the five items of lowest priority were equivalent across both models.  Table 4 compares rankings of program evaluation competency items between the Borich model and RDM. Like program planning, both models confirmed there were discrepancies in all competency items relating to evaluation. The top three priority items were similar across models, albeit with differed ordering. While "communicate evaluation information to stakeholders" was ranked as the highest priority in the Borich model, "conduct follow-up surveys to measure behavior change" was ranked highest in the RDM. Similar to program planning items, nine of top ten evaluation competency items were consistent across models. While "analyze findings from evaluation activities" was ranked 12th in the Borich model, it was ranked 9th in the RDM. Also, "design valid pre-and post-tests" was ranked 9th in the Borich model, but ranked 12th in the RDM.  Figure 1 illustrates the observed distances between MWDS and RDS. Absolute z-scores were used for comparison due to the inverted interpretation of scores between models; a positive MWDS in the Borich model and a negative RDS in the RDM represent a need or discrepancy. The figure shows a clear relationship between MWDS and RDS; scores followed a similar pattern across all 32 competency items. A Pearson's test revealed a very strong correlation between scores (r = 0.98), demonstrating a high level of consistency between models.

Analytical Steps in the RDM
Getting Started: Gather Borich-type competency data. Figure 2 provides a sample item from a competency assessment questionnaire.

Figure 2
Program Planning Item in a Competency Assessment Figure 3 shows sample raw data in SPSS for five respondents. On the left shows the value labels for each respondent (i.e., row), and on the right shows the coded value. The sample data view is the first place to observe the number of occurrences in the sample when respondents' ability ratings are: (a) less than respondents' importance ratings (Negative Ranks = NR), (b) more than respondents' importance ratings (Positive Ranks = PR), or (c) equal to respondents' importance ratings (Tied Ranks = TR). For example, Respondent 1 has a positive rank, Respondent 2 has a negative rank, and Respondent 4 has a tied rank.

Sample Competency Data Viewed in SPSS
Step 1: Calculate the number of occurrences for negative ranks (NR), positive ranks (PR), and tied ranks (TR) in SPSS (see Figure 4). In SPSS, run the test as follows: • Analyze → Nonparametric Tests → Legacy Dialogs → 2 Related Samples.
• For each competency item, enter responses for Importance (Variable 1) and Ability (Variable 2). • Repeat entries in Pairs for each competency item.

Figure 4
The Wilcoxon Test Window in SPSS (v. 27) The resulting SPSS Output will provide Negative Ranks, Positive Ranks, and Tied Ranks as shown in Figure 5.

Conclusions, Discussion, and Recommendations
We sought to determine if the RDM could serve as a suitable alternative to the Borich model. Our goal was to retain Borich's (1980) emphasis on using group perception to determine when or if gaps in competency should be considered priorities for training. Our findings support the utility of the RDM approach as a compatible alternative to the Borich model.
The comparison of RDS to MWDS showed a great deal of consistency in rankings, despite the RDM discarding the use of the group mean in calculations seen in Borich's (1980) model. Nine of the 17 program planning competencies had the same ranking when calculated using the RDM and Borich model approaches. No ranking was more than three places apart and five rankings were within one place of each other. For program evaluation, the same trend was observed. Four of fifteen competencies had the same ranking when calculated with either model while another four competencies were within one place apart. The remaining competencies were no more than three places apart. The visual comparison of absolute zscores and the results of the correlational analysis further confirm that a strong relationship exists between competency gaps identified by the RDM and Borich model. An implication of our finding is that calculating the group mean is not required for determining competency gaps. Researchers conducting competency research should consider adopting the RDM given its suitability for delivering results that closely resemble findings from the Borich model. Adopting the RDM allows researchers to avoid their work being scrutinized for the use of means for individual ordinal items (e.g., Boone & Boone, 2012;Norman, 2010). The standardization of RDS, regardless of how many scale items are used to measure importance and ability, offers researchers the improved ability to compare their work with prior studies of the same competencies to determine how closely their findings match others.
Another advantage of the RDM is that it decreases the complexity of interpreting results. The Borich model yields positive MWDS when training is needed. In our opinion, this is not intuitive given that many readers will have matriculated through a school system in which the goal was to score as close to 100 as possible to demonstrate mastery of a subject. We borrowed the same logic for the RDM. Instead of positive scores indicating a lack of competence, the RDM provides a negative RDS when training needs are greater (i.e., there are many individuals lacking sufficient ability and few individuals with an abundance of ability), which more clearly conveys that a problem exists that should be corrected. Therefore, the RDS demonstrates the magnitude of a discrepancy and maintains the underpinnings of a need as described by Witkin and Altschuld (1995), or a motivational force as discussed by Lewin (1939).
The use of a standardized range between -100 and 100 with the RDM is cognitively easier to interpret than a range that varies based on the number of response anchors, but often runs from -4 to 20 with the Borich model using a 5-point semantic scale. However, as seen in our findings, MWDS often range between 2 and 4, making it seem like the magnitude of a training need is quite small even when a score of 4 indicates a serious gap in proficiency. The example of developing long-term program objectives illustrates the difference; compare the MWDS of 4.04 (maximum possible score = 20) versus the RDS of -61 (minimum possible score = -100). The RDM does a superior job at showing the magnitude of the gap.
We want to be clear that the proper use of RDM approach requires the consideration of all three rank categories: PR, NR, and TR. The reason is that the RDS scores represent the capacity of the sample, inclusive of individuals who are excellent at a given competency and those who lack the necessary ability. For example, developing long-term objectives has a slightly higher RDS (-61) than conducting a needs assessment (-60), despite a greater percentage of NR for the latter competency. However, there is 9% of the sample that reported greater ability than necessary for conducting needs assessments while only 2% of the sample said the same for developing long-term objectives. Cumulatively, this sums up to a greater amount of capacity for conducting needs assessments in the overall sample. In practice, knowing what percentage of the sample has more ability than needed is helpful for assessing whether professional development strategies based on peer-to-peer learning, mentoring, or coaching may be effective. A skilled staff development professional should develop interventions that build upon existing assets, including human capacity.
Quantitative research in extension often relies on ordinal data; we commonly operationalize constructs to test theories and develop ordinal rating scales to measure psychological variables. As such, the Borich model has been widely applied in extension over the past forty years due to its ability to provide meaningful insights on the professional development needs of professionals. However, with ongoing philosophical and statistical debates in academia, we must be able to justify our analytical approaches to the wider scientific community. The RDM provides an innovative and defensible approach for researchers and practitioners interested in using needs assessment data to determine competency gaps when planning their professional development interventions.