Knowledge testing options in pre-test post-test evaluation design: Implications for Extension program evaluation

K. S. U. Jayaratne; Anil Kumar Chaudhary; John M. Diaz

doi:10.37433/aad.v6i4.659

Authors

K. S. U. Jayaratne North Carolina State University, USA https://orcid.org/0000-0001-5591-3905
Anil Kumar Chaudhary Pennsylvania State University, USA https://orcid.org/0000-0001-5809-8854
John M. Diaz University of Florida, USA https://orcid.org/0000-0002-2787-8759

DOI:

https://doi.org/10.37433/aad.v6i4.659

Keywords:

knowledge testing, objective knowledge test, subjective knowledge test, retrospective pre-test, extension evaluation, SDG 3: Good Heath and Well-being

Abstract

Knowledge gained is an important outcome indicator used in the evaluation of extension education programs. Three evaluation designs commonly used to document extension program participants’ knowledge change are: (a) objective knowledge pre-test, post-test design, (b) subjective knowledge pre-test, post-test design, and (c) subjective knowledge retrospective pre-test, post-test design. This study was designed to examine the relationship between measuring variables of objective knowledge pre-test, post-test, and subjective knowledge pre-test, post-test designs, and to examine the validity of subjective knowledge pre-test and post-test designs in assessing the knowledge gained by the Cook Smart Eat Smart extension participants. The researchers developed the survey instrument to document objective knowledge and subjective knowledge. The survey was administered before and after completing the training. The study received 71 responses. Paired sample t-test and correlation analysis were used to achieve research objectives. The findings indicate that all three designs are effective in documenting the changes in participants’ knowledge. Findings also verified the accuracy and validity of the subjective knowledge retrospective pre-test and post-test design. The evaluation of extension programs is strengthened by evidence demonstrating the validity of both objective and subjective measures for assessing knowledge gain, a key outcome indicator in extension education.

Downloads

Download data is not yet available.

References

Ary, D., Jacobs, L. C., Razavieh, A., & Sorensen, C. (2006). Introduction to research in education (7th ed.). Thomson Wadsworth.

Bennett, C. (1975). Up the hierarchy. The Journal of Extension, 13(2), Article 2. https://open.clemson.edu/joe/vol13/iss2/2

Brucks, M. (1985). The effects of product class knowledge on information search behavior. Journal of Consumer Research, 12(1), 1-16. https://doi.org/10.1086/209031 DOI: https://doi.org/10.1086/209031

Carlson, J. P., Vincent, L. H., Hardesty, D. M., Bearden, W. O., & Deighton, J. (2009). Objective and subjective knowledge relationships: A quantitative analysis of consumer research findings. Journal of Consumer Research, 35(5), 864–876. https://doi.org/10.1086/593688 DOI: https://doi.org/10.1086/593688

Chasteen, C., & Chattergoon, R. (2019). A comparison study of pre/post-test and retrospective pre-test for measuring faculty attitude change. 2019 PERC Proceedings, 93-98. https://doi.org/10.1119/perc.2019.pr.Chasteen DOI: https://doi.org/10.1119/perc.2019.pr.Chasteen

Diaz, J., Silvert, C., Gusto, C., Jayaratne, K., & Narine, L. (2021). Towards intercultural competence: Using consensus to identify essential personality traits for an inclusive extension education workforce. Advancements in Agricultural Development, 2(3), 83–95. https://doi.org/10.37433/aad.v2i3.135 DOI: https://doi.org/10.37433/aad.v2i3.135

Drennan, J., & Hyde, A. (2008). Controlling response shift bias: The use of the retrospective pre‐test design in the evaluation of a master's programme. Assessment & Evaluation in Higher Education, 33(6), 699-709. https://doi.org/10.1080/02602930701773026 DOI: https://doi.org/10.1080/02602930701773026

Ellen, P. S. (1994). Do we know what we need to know? Objective and subjective knowledge effects on pro-ecological behaviors. Journal of Business Research, 30(1), 43–52. https://doi.org/10.1016/0148-2963(94)90067-1 DOI: https://doi.org/10.1016/0148-2963(94)90067-1

Engel, J. F., Blackwell, R. D., & Miniard, P. W. (1993). Consumer behavior (7th ed.). The Dryden Press.

Farrell, T., Kohn, G. P., Owen, S. M., Meyers, M. O., Stewart, R. A., & Meyer, A. A. (2010). Low correlation between subjective and objective measures of knowledge on surgery clerkships. Journal of the American College of Surgeons 210(5), 680-683. https://doi.org/10.1016/j.jamcollsurg.2009.12.020 DOI: https://doi.org/10.1016/j.jamcollsurg.2009.12.020

Howard, G. S. (1980). Response-shift bias: A problem in evaluating interventions with pre/post self-reports. Evaluation Review, 4(1), 93–106. https://doi.org/10.1177/0193841X8000400105 DOI: https://doi.org/10.1177/0193841X8000400105

Joshi, A., Diaz, J., Kumar Chaudhary, A., Jayaratne, K. S. U., & Galindo, S. (2024). Enhancing effectiveness of Extension program evaluations by validating the trustworthiness of self-reported measures of Extension program outcomes. Advancements in Agricultural Development, 5(4), 59–71. https://doi.org/10.37433/aad.v5i4.519 DOI: https://doi.org/10.37433/aad.v5i4.519

Meghanathan, N. (2016). Assortativity analysis of real-world network graphs based on centrality metrics. Computer and Information Science, 9(3),7-25. https://doi.org/10.5539/cis.v9n3p7 DOI: https://doi.org/10.5539/cis.v9n3p7

Moore, D., & Tananis, C. A. (2009). Measuring change in a short-term educational program using a retrospective pretest design. American Journal of Evaluation, 30(2), 189–202. https://doi.org/10.1177/1098214009334506 DOI: https://doi.org/10.1177/1098214009334506

Nielsen, R. B. (2011). A Retrospective pretest-posttest evaluation of a one-time personal finance training. Journal of Extension 49(1), Article 4. https://doi.org/10.34068/joe.49.01.04 DOI: https://doi.org/10.34068/joe.49.01.04

O’Leary, J. L., & Israel, G. D. (2019). Capturing change: Comparing pretest-posttest and retrospectivee methods. IFAS Extension, University of Florida. https://doi.org/10.32473/edis-wc135-2013 DOI: https://doi.org/10.32473/edis-wc135-2013

Pratt, C. C., McGuigan, W. M., & Katzev, A. R. (2000). Measuring program outcomes: Using retrospective pretest methodology. American Journal of Evaluation, 21(3), 341–349. https://doi.org/10.1177/109821400002100305 DOI: https://doi.org/10.1177/109821400002100305

Raju, P. S., Lonial, S. C., & Mangold, W. G. (1995). Differential effects of subjective knowledge, objective knowledge, and usage experience on decision making: An exploratory investigation. Journal of Consumer Psychology, 4(2), 153-180. https://doi.org/10.1207/s15327663jcp0402_04 DOI: https://doi.org/10.1207/s15327663jcp0402_04

Skeff, K. M., Stratos, G. A., & Bergen, M. R. (1992). Evaluation of a medical faculty development program: A comparison of traditional pre/post and retrospective pre/post self-assessment ratings. Evaluation and the Health Care Professions, 15(3), 350-366. https://doi.org/10.1177/016327879201500307 DOI: https://doi.org/10.1177/016327879201500307

Spreng, R. A., & Olshavsky, R. W. (1990). Exploring the headwaters of the prior knowledge—search relationship. In A. Parasuraman & W. O. Bearden (Eds.), Enhancing knowledge development in marketing (pp. 220–224). American Marketing Association.

Taminiau-Bloem, E. F., Schwartz, C. E., van Zuuren, F. J., Koeneman, M. A., Visser, M. R. M., Tishelman, C., Koning, C. C. E., & Sprangers, M. A. G. (2015). Using a retrospective pretest instead of a conventional pretest is replacing biases: A qualitative study of cognitive processes underlying responses to thentest items. Quality of Life Research. 25(6), 1327-1337. https://doi.org/10.1007/s11136-015-1175-4 DOI: https://doi.org/10.1007/s11136-015-1175-4

Thomas, E. V., Wells, R., Baumann, S. D., Graybill, E., Roach, A., Truscott, S. D., Crenshaw, M., & Crimmins, D. (2018). Comparing traditional versus retrospective pre-/post-assessment in an interdisciplinary leadership training program. Maternal and Child Health Journal 23, 191–200. https://doi.org/10.1007/s10995-018-2615-x DOI: https://doi.org/10.1007/s10995-018-2615-x