All in A Moment: Continuous Response Measurement Method for Analyzing Dynamic Science Communication Content

Science communication strategy plays a crucial role in effectively conveying scientific information to target audiences. While various models exist for developing communication strategies, many focus on measuring the effectiveness of communication efforts and adjusting these based on audience feedback. However, traditional methods of evaluating communication effectiveness often measure one component at a time and do not consider how people make decisions in real-life situations. This paper proposes the use of continuous response measurement (CRM) as a method to evaluate science communication in agricultural social science research. CRM allows for real-time measurement of how individuals make decisions in response to dynamic communication content. This paper compares different types of CRM, including in-person and


Introduction and Problem Statement
Science communication strategy involves developing content that resonates with target audiences while emphasizing major tenants of science communication: timeliness, accessibility, trustworthiness, credibility, and usefulness.While models for developing a communication strategy vary, many models are cyclical in nature with an emphasis on measuring the effectiveness of communication efforts followed by adjusting communication based on feedback from target audiences (Percy, 2023;Villarreal, 2010).Primary and secondary consumer research play a vital role in developing a communication strategy and evaluating the strategy.Traditional methods for evaluating communication efforts include both quantitative and qualitative methodologies including surveys, sales data, behavioral observations, focus groups, interviews, and social media listening (Juska, 2021).Each of these methods has benefits in understanding how effective science communication is, but these tend to measure effectiveness in an artificial way, by focusing on measuring one component of a strategy at a time.However, modern behavioral economics indicates that people make decisions based on more than one factor at a time (Grayot, 2020).When you ask a question on a survey about a respondent's perceptions of a campaign, you are forcing them to think about one element of a message at a time, but in real life they are making decisions based on multiple components at a time.Dual processing theory proposes that people make decisions two different ways (a) intuitively and (b) deliberately (De Neys & Pennycook, 2019).When you ask a question in a survey or interview you force people to deliberately decide based on one element.In a natural setting, they would decide intuitively based on a blend of message, graphics, audio, logo, color scheme, speaker characteristics, etc.For these reasons, it is imperative that researchers establish and embrace new methods for gathering real-time data from participants.
Currently, many methods do not allow for real-time response or immediate feedback to change course in science communication campaigns.Continuous response measurement (CRM) offers a way to measure how people make decisions on dynamic, communication content in real time from moment-to-moment.CRM is commonly used in the communication industry for real-time measurement, but it has not yet been widely adopted in social science research (Maier et al., 2009).The purpose of this paper is to propose the use of CRM in agricultural social science research by describing the process and how to analyze data reaped from the method.Through these objectives, the authors will also compare types of CRM, including in-person dial testing and virtual dial testing, and provide resources for other researchers who want to apply this methodology in their own research.

Continuous Response Measurement
To effectively respond to public concerns and solve real problems, immediate feedback is needed.One of the public criticisms of scientists during the COVID-19 pandemic was a lack of transparent and immediate response (Hamel et al., 2020;Kenen & Roubein, 2020;Pollard & Davis, 2021).Particularly in the agriculture and natural resource sectors, immediate response is often necessary due to the essence and urgency of the industries, such as dealing with natural disasters (Mike et al., 2020), food safety issues (Opat et al., 2018), biosecurity risks (Sellnow et al., 2017), and animal welfare issues (Steede et al., 2018).CRM offers a way for researchers to

Methods
To address the research objectives of this study, research synthesis methodology was used to comprehensively review and synthesize existing literature on CRM methods to explain how to plan and conduct CRM studies, data analysis approaches for CRM, and best practices for CRM in academic settings.The authors' search strategy included a systematic search of relevant scholarly databases (e.g., PubMed, Scopus, Web of Science) to retrieve articles published up until January 1, 2023.Additionally, grey literature, offline sources, and popular press references were explored to minimize publication bias and ensure all relevant resources were found.

Search Terms and Inclusion Criteria
The search strategy employed a combination of keywords and Boolean operators to identify relevant studies.These keywords included the exact terms and variations of the following terms: continuous response measurement, perception analyzer dials, online/virtual dial testing, dial testing.Articles were included if these met the predefined inclusion criteria based on if the articles used CRM methods, explained CRM planning, or analysis of CRM data.

Study Selection
Two independent reviewers screened the retrieved literature based on title and abstract, followed by a full-text review of potentially relevant studies.Any discrepancies between reviewers were resolved through discussion or consultation with a third reviewer.Articles were included if they met the predefined inclusion criteria and provided pertinent information related to the research purpose.Studies were excluded if they did not meet the inclusion criteria, were duplicates, or lacked relevant data.

Data Extraction
A standardized data extraction form was utilized to extract key information from included studies, such as author information, publication year, study design, analysis methods, interventions/exposures, research stimuli, research setting.The methodological quality and risk of bias of included studies were assessed using the Critical Appraisal Skills Programme Systematic Review Checklist (CASP, 2018) by two independent reviewers.Discrepancies were resolved through discussion or involvement of a third reviewer.

Synthesis of Results and Reporting
An a priori qualitative narrative synthesis was conducted to deductively summarize the findings from the included studies within the framework of the two study objectives.This deductive coding approach matched the purpose of this research as recommended in qualitative coding procedures (Silverman, 2019).The synthesis methods in this paper adhere to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines to ensure transparency and comprehensive reporting of the review process (Page et al., 2021).

Findings
RO1: Describe the planning process for conducting a CRM study.CRM technologies allow for measurement in self-reported shifts in message processes using electronic technologies (Biocca et al., 1994).CRM has been used in public speaking, advertising, film and television, political and health communication (Biocca et al., 1994), and agricultural communication (LaGrande et al., 2021;Opat et al., 2022;Tarpley et al., 2020).Research designs within this method can use experimental, survey and quasi-experimental, time-series, focus groups, and coding content and behavior designs (Biocca et al., 1994).Further descriptions of type of research design can be seen in Table 1.The majority of previously published work, regardless of the type of research design, has focused on in-person use of CRM, but CRM technology providers have recently started introducing online CRM options (Paull, 2023).

Time series
This design works well for public opinion on scientific issues that changes over time in response to current events or legislation.It can also be used when purposeful intervention is used by the researcher to measure the effectiveness of an intervention over time.Choosing a research stimulus is one of the most important steps in collecting CRM data (Lawson et al., 2020).CRM can be used to evaluate any dynamic content from a persuasive campaign video, a political debate, a teaching presentation, or a podcast.Table 2 provides a list of recommended stimuli and unique considerations for each type that arose from the themes in the literature.Once a stimulus is selected, researchers must choose a response prompt.
Response prompts may include the trustworthiness of the content, emotion, or the persuasiveness of the message.Participants can only respond to one option at a time, so it is suggested that researchers limit the options they ask participants to respond to in a study.

In-person CRM
CRM is a measurement tool that allows researchers to pinpoint specific perceptions of communication messages and elements of communication in real-time (Izenson, 2016).Traditionally, CRM is conducted in-person using handheld dials (Figure 1) that participants use to respond to the research stimulus.It is recommended to have participants practice using the dials before responding to the research prompt.For most participants, this is the first time they have used a CRM dial.
Researchers should ensure the participants are comfortable using the dial and understand the prompt.For example, if the prompt is trustworthiness, the researchers should provide an example of how extremely untrustworthy and extremely trustworthy is rationalized in the study.As participants respond to the prompt, researchers should observe the participants and how they are using the dials.Researchers may need to remind participants to turn their dial continuously.If a participant is inactive, researchers may need to remove their data from the study.For a thorough step-by-step guide on how to use in-person CRM in an agricultural social science context, refer to Lawson et al. (2020).Quantitative data is automatically collected through CRM software, but to gain additional insight it is recommended that researchers pause at key moments in data collection and/or at the end of data collection to show results and discuss the reasons why participants felt the way they did about the critical moments in the data.Using a qualitative methodology for this questioning route is recommended.
While in-person dial testing offers many unique benefits, limitations do exist.Two of the most burdensome limitations are the cost and participant recruitment.Software and equipment, including the console and 25 handheld dials, necessary to collect in-person CRM data will cost ~ $25,000.Each participant must use a handheld dial when participating in a CRM study so if researchers have 25 dials, only 25 people can be included in each study session.To collect data from 120 participants, the session would need to be held at least four times.Even if researchers have access to more handheld dials, it is recommended to keep the study groups a manageable size to avoid nonresponse and technical difficulties with the dials.Additional costs will be incurred to recruit participants and record and transcribe qualitative data.

Virtual CRM
Online CRM has recently been introduced by CRM technology providers, such as DialSmith so, as such, fewer academic publications exist.However, online CRM provides researchers with more flexibility compared to in-person dial testing.Recorded video or audio is uploaded to the dial testing platform.Participants use a unique web link to view or listen to the content being tested, and participants use their own computer or tablet to indicate their moment-to-moment response.Respondents receive instructions and have an opportunity to practice rating content not related to the study.A slider appears on the screen and participants are asked to respond to the research prompt by moving the slider (Izenson, 2023).The feedback is automatically available to researchers through the reporting portal.Visuals of the data, including moment-tomoment video overlays, are available in the reporting portal but researchers can also download raw data to conduct additional analyses.
The online CRM platform supports recorded audio and video files, including .mp4 and .movfiles, but cannot be used to test live streaming content (Paull, 2023).Live stimuli must use inperson CRM.Since online CRM participants do not have to be physically present, a more geographically diverse sample can be obtained (Alzuhn, 2023).Researchers are not limited by the number of handheld devices when conducting online CRM so more responses can be collected over a shorter period (Alzuhn, 2023).Online CRM can also be more cost effective, particularly when researchers have a limited number of studies they wish to conduct.Costs vary depending on the type of content being tested, length of content, and number of desired participants.Costs range from $1,500 to $2,000 USD to upload the content and manage the project with an additional cost to recruit participants.One distinct disadvantage to online CRM is respondent engagement, particularly when testing lengthier content.When dial testing inperson, moderators can observe when participants are no longer engaging with the handheld device and can remind them to turn the dial or indicate when a response should be removed from data analysis.With online CRM, researchers cannot observe participants and help with focus, but a reminder is displayed on the screen to keep rating the content if a respondent stops moving the onscreen slider.DialSmith reports that online and in-person CRM results are similar in response sentiments (Paull, 2023).

Reliability and Validity
As with any type of research study, establishing reliability is an important step to gathering valid and useful data.Reliability is concerned with determining if the same results can be expected if measured the same way (Maier et al., 2009).This can be tested in CRM through test-retest, split-half, and parallel-test designs.Reliability can be a challenge in test-retest because of the spontaneous nature of psychological measurement within CRM studies, but some studies have obtained reliability this way.Split-half is a more common method for determining reliability in CRM studies, but parallel-test designs can also be used (Maier et al., 2009).External validity, or generalization, in CRM studies is a matter of properly designing the study regarding sampling procedures that are appropriate and representativeness of the target population.Additionally, it is important to match the measurement setting as close to a real life setting as possible and select the appropriate stimuli and data collection site.Like other experimental and quasi-experimental designs, pre-testing of CRM instruments can help establish validity (Maier et al., 2009).For focus group designs, researchers should focus on appropriate sampling for their research questions and consider transferability to other settings.Verbatim transcripts, audit trails, data triangulation, member checks, and multiple coders can be used to ensure data accuracy (Lincoln & Guba, 1985).

Limitations
The cost of in-person and online CRM can be a significant limitation.In-person CRM requires more software and equipment, which can cost an estimated $25,000 USD.Online CRM can be a more cost efficient alternative but still comes with a cost of $2,000 USD or more depending on recruitment costs.Limited training is available for CRM data collection and analysis, which can be a limitation for researchers wanting to implement CRM research.Some CRM software companies, such as DialSmith, offer training on their specific software at an additional cost.These companies also offer consulting services to provide input on overall study design, data collection, and data analysis.CRM data is continuous, resulting in a data point per second per participant.As such, this methodology results in a large amount of data, even when working with a short stimulus and a small sample size, which can be a limitation.Researchers must be extremely organized and attentive to details to effectively manage the large amount of data and tie participant responses to stimuli.

RO2: Describe how to analyze data gathered from a CRM study.
Output of data from CRM technologies allows researchers to interpret in multiple ways.When collecting data in-person, you can export data as: PowerPoint, Excel, or SPSS files.When conducting virtual CRM, the same data output as in-person CRM is provided.Data visualizations are more readily available in the virtual CRM interface and, when using DialSmith, a project manager can support the data output and analysis phase.

Analysis
Data can be analyzed in two different ways.We have divided this into real time and reflective analysis.Real time analysis can provide immediate feedback, course correction, and/or discussion within a focus group setting.CRM allows researchers to view data in real time during collection in aggregate, by a single participant, or by a specific demographic.Summary statistics for each study question are projected on the researcher monitor throughout the data collection and immediately following.Data are saved on the designated CRM computer within the CRM software for future analysis.A researcher can choose to show the results to participants to collect their perceptions of the results and to discuss peaks and valleys in the data (Figure 2).This will offer additional insight beyond the quantitative data analysis and fits into the focus group design described in Table 1.Peaks and valleys in the data allow for visual understanding of the data through identification of high and low points.Data can be grouped by other study variables to visually determine differences (Figure 2).Depending on the research goal, some science communication projects may be able to use the real-time data analysis to adjust communication campaigns without deeper analysis.However, to publish data in a journal, reflective analysis is needed.Researchers may choose to run crosstabs or mean comparisons initially to understand the data.
In the virtual CRM option, crosstabs will be run by the project manager at DialSmith.Because of the large quantity of data, researchers often chose to compress the data from 1 second increments to 5 or 10 second increments depending on the study size.Normalizing the data through Z scores is another step researchers may take before moving to deeper analysis, particularly when working to identify statistical significance of peaks and valleys in the data.
Researchers may select to identify Z scores greater than 1.96 or less than -1.96 (Biocca et al., 1994), or they may choose to run independent samples t-test with an a priori established level.
Once peaks and valleys are established as significant, then it is important to review the stimuli to determine what was happening at these critical moments.This will help researchers determine what is happening at these moments that may be affecting participants' responses and to understand patterns in responses.Researchers may decide to categorize what was happening at critical moments to interpret the data.For example, if a particular word is used in all peaks, or if a particular speaker is on screen during every valley.
From this point, the data are handled similarly to how you would handle any other quantitative data set based on the study design and research questions.Studies have used basic descriptive statistics and mean comparisons.However, with such large data sets, deeper analysis methods are available for use and can offer deeper understanding.Beyond means and peaks and valleys, researchers can understand group differences with ANOVA.They can also create alternative series to look at standard deviation series and absolute, mean, and percentage activity series (Biocca et al., 1994).Time series analysis may include autoregressive integrated moving average (ATIMA) techniques (Box et al., 2015).Confirmatory factor analysis may also be an appropriate method as an extension of regression models; spectral or Fourier analysis may be appropriate to understand complex problems that were periodic in nature within the CRM study design (Biocca et al., 1994).For those interested in a deeper understanding of data analysis related to CRM, Biocca et al. (1994)

Conclusions, Discussion, and Recommendations
The benefits of CRM in agricultural social science research are abundant, including the depth of data collected, real-time response and visualization of the data, reduction of recall bias, and limiting group think.CRM allows for researchers to quickly collect a large amount of data that can be analyzed in a variety of ways, including audience segmentations, experimental designs, qualitative research, and self-report items (Lawson et al., 2020).Even though data can be analyzed in multiple ways, data can be viewed in real-time as participants are responding and immediately exported into visualizations that can be shared with participants or relevant stakeholder groups.When asking participants to respond to a survey question or a question posed during an interview or focus group, participants must often reflect on a past event or experience and may not remember accurately.Since CRM is collected in real-time of the participants interacting with the stimuli, the need for recall is removed.Additionally, CRM offers a way to measure messaging in a dual process way while all elements of a message are considered at the same time rather than isolating one specific component.Groupthink is often a limitation in qualitative research and occurs when one participant or a small group of participants influences others' opinions (Janis, 1997).In CRM, you can collect individual opinions without the influence of others, but also have the option to collect qualitative data in a focus group using CRM to gain individual and group insight into specific points in the data.Participants respond to CRM individually without the influence of other participants, limiting groupthink.
CRM has not been used to its full potential in agricultural social science research, but it could take science communication evaluation to the next level with quantification of perceptions and response to communication stimuli in a moment-to-moment format.Multiple opportunities exist to enhance strategic science communication in agriculture and natural resources.In-person CRM use can offer real time testing of science and outreach products and allow for discussion with stakeholders on the reasons behind their views on science communication products.With the instantaneous nature of social media, CRM can render a similar environment and gather feedback.Moreover, the immediate response and course correction offers the ability to respond rapidly during a crisis or as a crisis develops.Because in-person CRM requires more equipment and a larger research team, it may not be an option for all researchers.However, online CRM has a lower entry point with costs allocated by online response and support for developing integration into surveys.This option also allows for help in the recruitment of participants.
In addition to being a unique and useful research tool in science communication, CRM also provides opportunities for Extension, teaching, and leadership.Notably, CRM can be used to determine preferred teaching styles, perceptions of presenters, and response to messages used.Utilizing CRM can provide real-time insight to effective messaging, while also providing insight on how to become better teachers, presenters, and leaders.

Table 2
Recommended Stimulus Types for CRM and Considerations for Each (O'Cathain et al., 2010)les, and strategies.Focus group designs typically combine the peak and valley analysis with qualitative data analysis methods (Williams & Moser, 2019) to compare quantitative with qualitative results in a mixed methods design(O'Cathain et al., 2010).Focus group results can offer insight into participants thought processes while viewing stimuli more than CRM results alone.