Duration: 10/01/2018 to 09/30/2023

Administrative Advisor(s):

NIFA Reps:

Non-Technical Summary

Statement of Issues and Justification

NEED, IMPORTANCE AND FEASIBILITY: The Expanded Food and Nutrition Education Program (EFNEP) was established in the 1960’s to assist low-income families gain the knowledge, skills, attitudes, and changed behaviors necessary for nutritionally sound diets (USDA, 2015). The program also seeks to contribute to personal development and the improvement of the total family diet and well-being. Obesity has become a hallmark of low-income adults, particularly women (Ogden et al, 2010) with a concurrent rise in chronic diseases including type 2 diabetes, hypertension, heart disease, and all cancers (Guh et al., 2009). Developing new dietary assessment methods to provide information relevant to today’s food intake and practices is essential to determine the most effective nutrition education.

In terms of personal development and well-being, improved diet and nutrition appear to have an impact beyond promoting good physical health. For the past 40 years, participants have reported improvements in self-esteem, moral direction, and sense of belonging in their communities from EFNEP participation (Arnold & Sobal, 2000; Auld et al., 2013). Research is needed to substantiate these findings and determine how the program effects positive emotional and social changes as well as the economic benefits to individuals, families, communities and society at large. Stronger evaluation of the wider influence of EFNEP participation will provide a greater understanding of EFNEP’s value.

For the next 5-year cycle, this multi-state effort is proposing substantial efforts in the following evaluation domains:

• 24-hour dietary recall
• Food and Physical Activity Questionnaire
• Retrospective pre/post
• Infant feeding
• Quality of life
• Cost effectiveness

24-hour dietary recall: The 24-hour dietary recall (24HDR) is generally considered to be the gold standard for dietary assessment (Van Staveren et al., 2012). In EFNEP, paraprofessionals are trained to conduct 24HDR with groups of participants. However, this participant-recorded group-administered procedure for collecting 24HDR has never been validated (Townsend & Wilson, 2016). Recent research has questioned the quality of the data obtained due to inconsistency in the protocols used to collect the 24HDR and variation in training given to personnel (Gills et al., 2017).

Currently EFNEP’s impact and effectiveness are shown using changes in dietary intake and selected health-related behaviors. As mentioned above, there are questions about the validity of the dietary intake results obtained by EFNEP paraprofessionals. Valid and reliable information about dietary intake is essential to determine program efficacy. Therefore, it is imperative to continue researching methodologies to improve dietary data collection.

As a step toward improved standardization of 24HDR procedures, a pilot, including a comparison group, study of the Automated Self-administered 24HDR (ASA24) recall will be conducted.

Food and Physical Activity Questionnaire: A Behavior Checklist (BC) has been utilized to evaluate the impact of EFNEP programming on changes in participants’ food-related behaviors for participants that are not well captured by the 24HDR. The BC was established in 1993 and had 10 core items, assessing Food Resource Management, Nutrition Practices and Food Safety.  New research findings in these areas and multiple updates to the Dietary Guidelines for Americans made revisions crucial. The Food and Physical Activity Questionnaire (FPAQ) was developed by NC2169 and replaced the EFNEP BC beginning in Federal Fiscal Year 2018. Item development, reliability testing and validity testing of the FPAQ were conducted by NC2169. This nationally administered pretest-posttest tool will evaluate the effectiveness of EFNEP by identifying changes in participants’ behaviors after exposure to EFNEP lessons. The FPAQ is composed of 20 core items (and 12 additional, optional items) to evaluate behaviors related to Food Security, Food Resource Management, Nutrition Practices, Food Safety, and Physical Activity.

FPAQ assesses priority behaviors related to the 2015 Dietary Guidelines that were not measured by the BC. These include the recommendation to choose vegetables from all the subgroups to ensure variety and to consume less than 10% of calories per day from added sugars (USDHHS, 2008 and USDA, 2015). The EFNEP curricula implemented by the majority of programs in the country provide information for these dietary behaviors, along with interactive instruction (Murray et al., 2015). The FPAQ assesses behavioral change related to these recommendations.

Based on the 2008 Physical Activity Guidelines for Americans (USDHHS, 2008), EFNEP has placed more emphasis on including physical activity in all lessons. FPAQ has items related to these specific changes that are quantifiable and promise to provide a clearer picture of the effects of EFNEP lessons on physical activity.

The 20 core items were selected in 2017 by the EFNEP National Office leadership after all 32 items were tested for face, content, and construct validity as well as reliability using cognitive interviews, expert panels, comparison of data to more rigorous measures, and test-retest procedures. One or more of the 12 optional items can be added at the discretion of participating institutions.

The FPAQ has only been tested with English-speaking EFNEP-eligible populations. In Federal Fiscal Year 2016, 40% of EFNEP graduates identified themselves as Hispanic. According to the US Census Bureau (2017), 22 percent of Hispanic families of any race earned incomes below the Federal poverty level, compared to 7 percent of non-Hispanic Whites. Furthermore, more than 20 states and 1 territory offer EFNEP classes in Spanish. Reliability and validity testing of the Spanish translation of the FPAQ is necessary to ensure the tool is equally effective with Spanish-speaking populations.

Retrospective pre/post:  The evaluation protocol employed by EFNEP is the pretest-posttest method commonly used in educational interventions. It requires that the evaluation is administered to the same group on two separate occasions, one before the intervention (pretest), and one after the intervention (posttest) (Campbell & Stanley, 2015). Improved responses between pretest and posttest indicate positive behavior change, presumably resulting from the intervention (e.g., EFNEP nutrition classes). Although this design is widely used, concerns exist regarding this design’s validity (Howard et al., 1979; Davis, 2002; Pratt et al., 2000).

The biggest and foremost problem with this design is that it does not account for response-shift bias, a change in the way respondents interpret and respond to a question pre-and post-program evaluation, due to a change in their understanding. This calls into question the internal validity of questionnaire responses and may lead to inaccurate interpretations of the program’s effectiveness (Howard & Dailey, 1979; Howard et al., 1979). Further, administering two evaluations (pretest and posttest) consumes time that may be better spent on program delivery or establishing rapport with the participants to reduce attrition rate (Marshak et al., 1998). 

An alternative approach that addresses the limitations of the pretest-posttest is the retrospective pretest-design. This method requires respondents to complete a single, self-reported measure in reference to their behaviors prior to the intervention/program (retrospective pretest) and after the intervention/program (posttest). It is presumed that the responses are based on the same understanding of the variables of interest and thus would eliminate response shift bias.

It should be noted, the retrospective pre-test might introduce other problems due to its greater reliance on memory (Sibthorp et al., 2007; Lam & Bengo, 2003) and potential social desirability bias (Lam & Bengo, 2003; Betz & Hill, 2003), defined as “the tendency to transmit a culturally accepted image, according to social norms” (Poinho et al., 2015). To date, few studies have documented the use of retrospective pretest in extension education programs (Davis, 2000; Shilts et al., 2008; Marshall et al., 2007).

Infant feeding: EFNEP has traditionally conducted evaluation of education for families with young children, with limited emphasis on families with infants. There is now troubling evidence that growth patterns are also not optimal in infants (Pediatric Survelliance, 2011), putting them at risk of overweight as they move into toddler and preschool years, with the accompanying risks of disease and adult overweight. It is important that nutrition education for parents begin during pregnancy and continue in the infant’s first year of life (Lioret et al., 2013; Horodynski et al., 2007).

There are no validated evaluation items for the content delivered related to infant feeding in EFNEP. Although a tool to evaluate the relationship of patterns of infant feeding to variables such as infant weight has been developed (Thompson et al., 2009), this tool has proven to be unsuitable for providing meaningful nutrition education program evaluation. (Olson and Horodynski, unpublished data.)

NC2169 employed a process to create and test the 32 evaluation items for FPAQ. These same processes will be used to develop and test infant feeding items to add to the additional question bank for EFNEP.

Quality of life: Maslow’s (1954) widely-accepted hierarchy of human needs states that basic physical needs (food, shelter, water) must be satisfied before a person can attain higher order psychological fulfillment. EFNEP focuses on low-income clientele who often have difficulty meeting the basic physical needs for food and shelter. The program helps families manage resources through menu planning, purchasing food and food storage. As a side benefit, practicing food resource budgeting skills can help in other household management areas. Meeting these needs allows clientele to move toward psychological fulfillment and improved quality of life.

Diener and Diener’s (1995) research determined that people do not adapt to long periods of extreme poverty and those living in poverty typically experience lower levels of subjective well-being or low quality of life. Individuals with very low incomes were highly dissatisfied with their personal quality of life; however, when homeless individuals experienced good social relationships, not only was their perception of well-being higher but the psychological costs of material deprivation were mitigated (Diener & Seligman, 2002). Because EFNEP hires paraprofessional educators from the same community who have experienced poverty themselves, a large part of the EFNEP experience relies on establishing good social relationships.

There is much anecdotal evidence that EFNEP participation results in improved quality of life. Arnold and Sobal (2000) found a 33% increase in community involvement at churches, food pantries and schools among program graduates and attributed this to encouragement from the nutrition paraprofessionals. They also found an increase in employment among graduates. As a result of earlier work, Auld et al. (2013 & 2016) found EFNEP graduates and educators reported an increase in positive views about themselves, e.g., self-acceptance and self-esteem, as well as improved sense of hope and belonging. Determining whether EFNEP participation provides a broader benefit of improved quality of life is needed for greater knowledge of the program’s impact. If EFNEP participation results in a perception of improved quality of life, program evaluation could expand understanding of EFNEP’s benefits, e.g., improved quality of life may predict sustained healthy behavior changes and, ultimately, improved health status. 

Cost effectiveness: Through NC1169 and NC2169, researchers focused on developing and testing of evaluation items for EFNEP related to food, physical activity and quality of life. These tools provide data for an assessment of the cost effectiveness of EFNEP. NC3169 proposes to seek funding to conduct an analysis of the cost effectiveness of EFNEP. This approach will rely on both objective health measures (blood pressure, Hemoglobin A1C, and body mass index) and self-reported data to forecast the benefits of EFNEP. USDA (NIFA) has funded the Expanded Food and Nutrition Education Program (EFNEP) for nearly 50 years and currently spends over $67,000,000 annually; an analysis of its cost effectiveness is needed.

ADVANTAGES OF A MULTISTATE EFFORT: A multistate approach is essential for this project because of the scope of EFNEP. Each state/region has access to unique groups of limited-resource EFNEP participants that would not be available from any single state. Variables that differ across states include: obesity rates, employment opportunities, education, age, income, proportion of urban versus rural households, and mixes of racial/ethnic groups, to name a few. NC2169 involves researchers from more than 20 states representing all four NIFA regions. With multi-state involvement and input, a more complete access to and understanding of demographic, cultural and other impacts on our diverse participant groups is possible. 

LIKELY IMPACTS: The target clientele of EFNEP consists of low-income families (income at or below 185% of the Federal poverty threshold). Self-reported race of participants was 54% White, 24% African American, 2% American Indian or Alaska Native and 3% Asian or Pacific Islander; 40% reported their ethnicity as Hispanic. EFNEP operates in 800 counties in all 50 states and six territories reaching 118,976 adults, and 365,369 youth in 2016. The total number of lessons taught annually to adults and youth is estimated to be 3.9 million. It is imperative that the value of this effort be maximized in terms of effectiveness of nutrition education and on wider potential benefits to quality of life.

The assessment of EFNEP impacts is critical to program success. This project will provide updated valid, reliable methods for measuring dietary quality and health-related behaviors including physical activity in the EFNEP population to document program performance and provide valuable needs assessment data to inform future planning and implementation.

This project will also lead to a better understanding of the non-dietary impacts of EFNEP. Both qualitative and quantitative methods will be used to assess the quality of life of participants and paraprofessionals, allowing EFNEP to quantify non-dietary impacts for the first time in the program’s nearly 50 year history.

 

Related, Current and Previous Work

RELATED WORK: A CRIS search of EFNEP identified two projects: one with the objective of evaluating the current operation of EFNEP and identifying targets for program improvement and expansion; and the second with the focus on using multidisciplinary methods for effective and sustainable evaluation of nutrition education programs. The second of these projects is led by a researcher who is a member of NC2169. These projects supplement the current work by informing the ways to improve participant retention, to expand program reach, and to promote effective use of dietary assessment tools. Other CRIS projects focused on childhood obesity. A search on dietary assessment showed 22 projects, including NC 2169. The projects used various established tools to assess dietary intake of different groups, evaluating nutrition education programs, or developing food frequency tools to measure specific nutrients. Two projects focused on dietary assessments using technology such as a diet tracking smart phone app and web-based digital imaging to measure fruit and vegetable consumption. 

 A related project in NIMSS is NC1196: Food systems, health, and well-being: understanding complex relationships and dynamics of change. NC1196 is investigating the interplay of food systems including food availability, seasonal supplies, consumption and utilization as a mechanism to improve food and nutrition security. It is assessing challenges for vulnerable populations to increase positive food decision making and also assessing strengths and weaknesses of public and private emergency food assistance programs. NC3169 is focuses on the assessment tools for EFNEP as a direct-education program for low income adults. NC1196 is taking a broader policy, systems and environments approach while NC3169 is about establishing research-based mechanisms to assess individual adult behavior changes. These two projects have potential to support and inform each other through their research efforts.

CURRENT WORK: Cognitive interviews have just been completed to determine face validity of the infant feeding (IF) evaluation tool items for the intended population of low-income mothers. Test-retest reliability will be performed to measure the reliability of the items over time.

In addition, a quantitative Quality of Life (QoL) assessment tool that can be used by EFNEP as a stand-alone outcome assessment or in conjunction with other measures including cost benefit analyses is being developed. Data from the pilot study (Auld, et al. 2013), interviews (Auld, et al. 2016) and focus groups are being integrated to develop a quantitative QoL questionnaire. The questionnaire will be tested to establish content validity and for further testing of various formats for items and response options.

An electronic, self-administered 24 HDR (ASA24) training manual is being developed to teach nutrition educators how to facilitate ASA24 data collection in group settings. This interactive manual was pilot tested with EFNEP educators, in the field. Feedback provided from this pilot study is currently being used to improve the manual.

Questions on physical activity in the new Food and Physical Activity Questionnaire (FPAQ) are being tested with participants along with the gold standard for physical activity measurement (i.e., accelerometers) to ensure the questions measure what they are intended to measure. The questionnaire and accelerometer data from participants are currently being analyzed by Rutgers University and the validity study is under development. Continued data analysis of the questionnaires and data collection from other states is underway.

Five doctoral candidates are presently involved with the projects related to ASA24, infant feeding evaluation, validity testing of physical activity questions, and QoL assessment. Materials have been and continue to be developed for training coordinators and paraprofessional educators in new data collection procedures.

Michigan State University has a small pilot study estimating the cost benefit analysis (CBA) and cost effectiveness analysis (CEA) of EFNEP. This study will determine the relationship of nutrition education, nutrition and physical activity behavioral changes, and biological changes to assess the effectiveness of the program. The EFNEP intervention and biometric measures will be used to estimate the health impact and to calculate CBA and CEA. First, we are investigating all relevant alternatives to CBA and CEA related to the nutrition education intervention and address the criticism of this methodology, and adjust the formula to capture all relevant direct and indirect costs and consequences of the training. These may include: current disease state, household demographics, socio-economic distribution, satisfaction with health, and QoL. The second step is to predict the obesity cases averted utilizing body-mass index (BMI) by a certain age as shown in Brown et al. (2007). Based on these estimates, the Life table (Peeters et al., 2003) will be used to estimate the health care benefits. Lastly, a BMI progression model and scenarios analysis will be used to determine the sensitivity of the CBA/CEA results.  In addition to demographic and economic data currently being collected, partnering organizations will facilitate the collection of key health status data of those participants. These may include: BMI, blood pressure, current disease state, and Hemoglobin A1c. These data will be used to develop, test and calibrate theoretical economic models that associate EFNEP training and demographic, economic, QoL, and health status indicators. Based on the outcome of the pilot project work, a national protocol will be developed to conduct an economic analysis of the EFNEP program.

To assess EFNEP participant perceptions’ of the 24-hour dietary recall (24DR) as delivered by educators in group settings, focus groups were held in five states. EFNEP participants described the 24HDR as a tool they found value in as a means of increasing personal awareness about the foods they eat. They also described challenges of remembering the amounts of food and the challenges presented by the classroom environment and form layout. Results revealed several opportunities for form redesign and training to enhance educator delivery to potentially improve participants’ ability to accurately complete the tool. A manuscript is in process to share these results.

PREVIOUS WORK: A comprehensive review of the literature examining the research basis for dietary assessment methods used with low-income adults was prepared through a collaboration of six states (CA, KY, NV, OK, WA, WY).

A searchable EFNEP research database was developed to assist practitioners, students, and researchers (Scholl, Paster & Jankowski, 2011). The database received the 2011 NIFA (National Institute of Food and Agriculture) Partnership Award for Effective and Efficient Use of Resource and the 2013 AgNIC (Agricultural Network Information Cooperative) Partnership Award. It identifies and categorizes all known research studies conducted about EFNEP since 1969 and currently houses 550 studies about EFNEP clientele, curricula, and staff. As part of NC2169, project investigator Jan Scholl updated this database with more than 150 additional research studies. In addition to EFNEP, other groups have benefitted from the searchable database as other articles have cited the work in various states. The awards received, including the Mary Nell Greenwood, from the National Evaluation Association in 2014, and in 2016, the Penn State Gamma Sigma Delta Extension Faculty award have drawn interest from those outside of the food and nutrition area that addresses the special needs of low-income people.

A survey of EFNEP coordinators’ perceptions of the current 24HDR method was conducted with 35 states and territories responding. Results suggest that the method is considered cumbersome and time-consuming. Coordinators stated concerns that the findings from the 24HDR have limited validity (Gills et al., 2017) due to the wide variation in reported collection, training, and coding practices. The authors recommend that standardized methods for training, data collection, and data coding for the 24HR might improve consistency in EFNEP.

A pilot study with EFNEP clientele in 2010 compared an electronic, self-administered 24HDR (an early version of ASA 24) with the current paper-and-pencil, group-administered 24HDR method at intake was conducted in seven states (AZ, CO, KY, OK, UT, WA, WY). No difference in number of foods reported or nutrient intakes were found. Although the majority of the participants were able to successfully complete the electronic 24HDR regardless of literacy level, some Hispanic clients who speak English as a second language reported challenges with the tools that was only available in English at the time. The study demonstrated that EFNEP participants were able to effectively use the ASA24. Challenges with consistent study protocols among states precluded publication of this study. Following the pilot study, an NIH proposal was submitted to determine the accuracy of reported energy intake using the ASA24 in a sample of low-income, low-educated Hispanic women and the accuracy of the group-administered written 24HDR.

A web based training manual on ASA24 was developed and pilot-tested with Expanded Food and Nutrition EFNEP Educators. Respondents had positive feedback for the content, reading level, appearance and design, and activities of the training. Results from this study suggest that the training is effective in preparing EFNEP educators to collect 24-hour dietary recall data using ASA24-2016 (Spruance, et al. 2017).

In addition, a white paper was written on validity of the group administered 24-hour diet recall as used by EFNEP (Townsend & Wilson, 2016). Following a systematic literature review, authors concluded that the group-administered 24HDR has not been validated and described the history of the 24HDR in EFNEP and the one validation study they found. The authors provided recommendations and strategies for meaningful evaluation. This white paper offers a basis for future research and offers guidance on best practices to improve data quality. (https://www.nimss.org/system/ProjectAttachment/files/000/000/091/original/24HDR.EFNEP.White.Paper.NOV2016.pdf). 

In preparation for QoL focus groups, specific recruitment and training protocols were developed and incorporated into a DVD filmed at Colorado State University and edited by (and funded by) Kansas State University. Seven states (AZ, CO, KS, NE, OH, SC, VA) participated in a training Webinar and interviews were conducted with EFNEP educators, supervisors and agency partners (Auld et al., 2016). QoL focus groups were also conducted with EFNEP participants. These studies established that EFNEP does influence QoL of both participants and educators. Participants improved (pre to post) in 5 of 9 tested QoL domains but partially regressed by 3 months. Future work will focus on designing and validating an EFENP QoL questionnaire (E-QoL) with the three primary EFNEP racial/ethnic categories of participants (non-Hispanic White, Hispanic, non-Hispanic Black) from multiple states. 

Work on NC2169 from 2013 to date has focused on the following:

(1) Development of time efficient, user-friendly, valid and reliable methods for measuring dietary intake and food-related behaviors among EFNEP clientele. Surveys or interviews of EFNEP coordinators, paraprofessionals, and clientele have been conducted in 20 states and one territory (CO, KY, NJ, OK, SC, AZ, KS, NE, OH, VA, ID, IN, ME, NH, NM, UT, WV, WY, NV, WA, and Guam). An electronic 24HDR has been tested for feasibility of use among EFNEP clientele. A systemic review to identify any validation studies of the group-administered 24HDR and the current practices in administering 24HDR in EFNEP was conducted and a white paper was published. A study comparing results from group dietary recalls conducted by trained EFNEP paraprofessionals and those conducted by Registered Dietitians found no differences between the results of the recalls (Gills et al., 2017). Research is ongoing on the ASA24 to determine the feasibility of its use with low-income participants of multiple race ethnicities;

(2) Thorough review and testing of the nutrition and physical activity domain behavioral checklist items. The new 20-item FPAQ is currently being implemented by EFNEP nationally (FY 2018). Next steps are to determine existing or new nutrition domain behavioral checklist items that best predict positive behavior changes from program participation.

(3) Documentation of major themes of QoL impacts from EFNEP coordinators, paraprofessionals, participants and community partners. Next steps are to determine the extent of positive changes in perception of QoL among EFNEP participants and to determine effective methods to measure change in QoL due to EFNEP participation. Also, the relationship of EFNEP indicators and economic benefits will be examined.

Objectives

1. To improve standardization of EFNEP 24HDR (24-hour dietary recall) methods a. Develop standardized training and administration protocols for educators for the paper-and-pencil, group administered 24HDR b. Conduct controlled pilot study of implementing the Automated Self-administered 24HDR (ASA24) in EFNEP, including comparison groups, in several regions
   
2. To complete and expand testing of newly developed EFNEP Food and Physical Activity Questionnaire (FPAQ) a. Complete further reliability testing b. Conduct Spanish reliability and validity testing c. Develop Retrospective pre/post questionnaire and examine feasibility of design on EFNEP impact reporting. d. Complete validity and reliability testing of infant feeding questions
   
3. To determine if participants’ changes in Quality of Life (QoL) are attributable to EFNEP, using a validated EFNEP-tailored questionnaire
   

Methods

Objective 1. To improve standardization of EFNEP 24HDR (24-hour dietary recall) methods.

a. Develop standardized training and administration protocols for educators for the paper-and-pencil, group-administered 24HDR 

Based on work completed in NC2169, investigators in University of Wisconsin-Extension adn University of Maryland, will lead the work to evaluate and recommend the best 24HDR training practices and 24HDR training protocols for EFNEP paraprofessionals. A critical examination of current practices and the objectives of the 24HDR will be conducted using data collected from EFNEP state coordinators in NC2169.

b. Conduct controlled pilot study of implementing the Automated Self-administered 24HDR (ASA24) in EFNEP, including comparison groups, in several regions

Based on work led by Utah State University, investigators will evaluate food-related behavior change by EFNEP participants using ASA24 in a geographically-diverse representative sample with a comparison group. Investigators at Utah State will lead the work of recruiting six to eight states to collect ASA24 recalls at pre and post from their participants. These states will also be recruiting an income-comparable group who will also complete the ASA24 twice at approximately the same time as EFNEP participants do, typically, pre and post. 

Objective 2: To complete and expand testing of newly developed EFNEP Food and Physical Activity Behavior Questionnaire (FPAQ)

a. Complete further reliability testing 

Investigators from New Jersey, Kentucky, and Idaho will take the lead to finalize reliability testing of the FPAQ, using the complete 32-item questionnaire. The data will be analyzed to test the reliability (i.e. reliability over time and internal consistency) of the 20-items currently used nationwide in EFNEP, as well as the 32-items available. Test/re-test reliability will be conducted with all 32-items at one-month intervals with EFNEP eligible participants with national/regional representation. To test for internal consistency, a secondary analysis on FY 2018 national data set will be conducted.

b. Conduct Spanish reliability and validity testing

Investigators in Florida and Washington will lead the testing of the Spanish translation of the FPAQ. The translated questions will be tested for face validity, internal consistency and reliability using cognitive interviews, expert panels (Lynn, 1986), and test-retest procedures. Reliability and validity testing will build on the previous work of NC2169 by following the adapted versions of the original protocols for reliability and validity testing of the FPAQ to the extent possible (Murray et al., 2017). Protocols of alternate studies which have successfully tested Spanish translations of validated health related pre-test post-test survey tools will also be consulted for guidance (Aragones et al., 2008).

c. Develop Retrospective pre/post questionnaire and examine feasibility of design on EFNEP impact reporting

A different survey administration method to assess program impacts (i.e., the use of a retrospective pretest in which both the pretest and posttest are collected at the end of the last class series) will be evaluated. At least one state from each of the four Cooperative Extension regions (i.e., North Central, Northeast, Southern, Western) will be recruited for testing this method. Participating states will continue to collect participants’ dietary, physical activity, and other behavioral data using the FPAQ before the start of the class series (pretest). At the end of the last class, participants will be asked to report their behaviors on the FPAQ again (posttest). They will additionally complete a retrospective pretest, in which they will be asked to recall and report their behaviors before they entered the program.

Two different formats of the retrospective pretest will also be evaluated: 1) a combined retrospective pretest along with the posttest on a single survey; 2) a separate retrospective pretest survey from the posttest with the posttest presented first. This method will be examined in terms of its ease of use; and in terms of any differences, when compared to the traditional measures employed, that are detected in participant responses and the amount of class time displaced by evaluation.

d. Complete validity and reliability testing of infant feeding questions

Infant feeding questions will be tested for test-retest reliability and congruent validity. 

Test-retest reliability. We will recruit 90 low-income mothers with infants who identify as African American, Hispanic, or white (30 for each group). Participants will complete the evaluation items two times, two weeks apart and will receive a $10 and $25 gift card for the first and second visit, respectively. We will analyze test-retest reliability using Spearman’s rank correlation, calculating each item separately. We will determine an average coefficient of stability among participants for each item; values above 0.7 will indicate reliable construct measures and those below 0.7 will be eliminated (Carter et al., 2013).

Congruent validity. Lacking a “gold standard” for evaluation of healthy infant feeding, we will further validate this measure with convergent validity. We will compare the remaining items to the Centers for Disease Control and Prevention Infant Feeding Practices Study II (IFPS II) questionnaire measurements (CDC). This instrument has been used with a nationwide sample, and has collected follow-up health outcomes (Grummer-Strawn et al., 2014). The items we will use measure the IF food frequency, food variety, age of introduction of solids, and formula and breastfeeding cessation. We will ask low-income mothers from the same race/ethnic groups to complete the IF items developed by the researchers and the IFPS II items during the same visit. Agreement between developed items and IFPS II items will be tested using correlation coefficients.

Objective 3: To determine if participants’ changes in Quality of Life (QoL) are attributable to EFNEP, using a validated EFNEP-tailored questionnaire (E-QoL).

NC3169 will follow similar protocols to validate the E-QoL as was done in NC2169 for the FY2018 national implementation of the FPAQ. Extensive rounds of cognitive interviews will be conducted with EFNEP participants from multiple states and the three primary EFNEP racial/ethnic categories of participants (non-Hispanic White, Hispanic, non-Hispanic Black). These interviews will establish face validity, i.e., the consistent and intended interpretation of items and response options. The next step will be to establish temporal reliability by following a test/retest protocol with EFNEP participants or EFNEP eligible participants – again from multiple states and the three racial/ethnic groups. The final validation step, construct validity, will compare results from the E-QoL with those from several established QoL tools. The established tools are typically longer, more expensive to use, and/or include assessment of some constructs not pertinent for EFNEP.

Measurement of Progress and Results

Outputs

• Valid and reliable assessment tools, administration protocols and training resources for use with EFNEP participants including the following: Paper-and-pencil 24HDR, ASA-24, FPAQ in English and Spanish, Pre/post and retrospective pre/post, Infant feeding, and Quality of Life
• Verification of E-QoL as a predictor of participants’ sustained behavior change
• Reports, articles and presentations reflecting work and findings throughout project
• Application for funding to plan and implement a cost effectiveness analysis of EFNEP

Outcomes or Projected Impacts

• NC3169 investigators will develop new skills to study and assess programmatic impacts in dietary and health-related behaviors.
• The national EFNEP system will have an enhanced understanding of and confidence in EFNEP program impacts.
• EFNEP coordinators and paraprofessionals will have access to protocols and training resources to implement valid and reliable tools in English and Spanish to assess program impacts.
• Educators will have an alternative approach for assessing EFNEP as retrospective pretest to potentially reduce response-shift bias.
• Educators will spend less time on evaluation with more time for program delivery and relationship building with program participants.
• At the Federal level, National Program Leaders and Congress will have a greater knowledge of the importance of EFNEP for the long-term health and wellness of individuals, families, communities, and the nation as a whole.
• Congressional stakeholders will have increased confidence in EFNEP's outcomes and therefore support continuation of EFNEP funding into the future.
• In the long term, the work of this project may have impact with nutrition education beyond EFNEP, including SNAP-Ed and the many nutrition education projects using EFNEP’s evaluation tool. Although the outputs, or assessment tools, are designed for low-income nutrition education, there may be applications to nutrition education for other income levels.

Milestones

(1):• Examine current practices and objectives of paper-and-pencil, group-administered 24HDR. • Develop methods for collecting income-matched control sample, standardized protocols using ASA24 and recruit states. • Implement English FPAQ reliability testing, recruit participants; collect data. • Analyze FPAQ reliability testing data. • Translate/back translate FPAQ 32 items; establish protocol and trainings for cognitive interviews for face validity testing of Spanish FPAQ. Complete expert review. • Develop a retrospective pretest version of FPAQ and protocols. Obtain IRB approval and recruit states for pilot study. • Completion of infant feeding validation and reliability work. • Conduct cognitive interviews for QoL • Submit manuscript(s) to peer-reviewed journals. • Submit grant proposal(s).

(2):• Establish and test protocols for paper-and-pencil, group-administered 24HDR. • Collect ASA24 data from participants and comparison sample. • Conduct cognitive interviews in Spanish for Spanish FPAQ. Revise questions. • Refine protocols for retrospective pretest. Recruit study for collecting retrospective pretest and pretest-posttest data from EFNEP. • Complete cognitive interviews, modifications to E-QoL. Conduct reliability testing. • Submit manuscript(s) to peer-reviewed journals.

(3):• Disseminate standardized training resources and protocols for paper-and-pencil, group-administered 24HDR. • Collect ASA24 data from participants and comparison sample. • Formulate FPAQ items per 2020 DGA; develop protocol for reliability, validity testing. • Finalize Spanish FPAQ. Develop study protocols for test-rest reliability, internal consistency. • Continue retrospective pretest and pretest-posttest data collection. Test different formats of retrospective pretest. • Conduct construct validity testing using established E-QoL tools. • Submit manuscript(s) to peer-reviewed journals.

(4):• Analyze, share results from ASA24. • Continue testing of revised FPAQ questions. • Complete validity, reliability testing of the Spanish FPAQ. • Analyze and share results from retrospective pretest data collection. • Assess participant, educator changes in QoL; establish baselines, size of changes. • Submit manuscript(s) to peer-reviewed journals.

(5):• Disseminate results from ASA24, FPAQ revisions, Spanish FPAQ and retrospective pretest data collection. • Begin program-wide data collection of E-QoL.

Projected Participation

View Appendix E: Participation

Outreach Plan

Results of this project will be widely shared with interested stakeholders in a variety of ways. Manuscripts will be published in peer-reviewed journals to strengthen the evidence base. Proposals for presentations and workshops at regional and national meetings will be submitted to share findings with professionals within EFNEP and other related programs such as SNAP-Ed. Potential meetings include the Society for Nutrition Education and Behavior, International Society for Behavioral Nutrition and Physical Activity, Experimental Biology, and the annual meeting of EFNEP Coordinators. Project personnel will also offer to serve in an advisory capacity to national EFNEP leaders who could utilize project outputs in the development of reports for Congressional leaders who annually vote on the continuation and/or potential increases in EFNEP funding. Clientele will be reached through the revisions made to materials/approaches based on project outputs.

Organization/Governance

The investigators; comprised of researchers and EFNEP Coordinators from participating land-grant universities, Regional Administrative Advisor, and USDA NIFA Representative; will be organized and governed as specified in the North Central Regional Association (NCRA) guidelines (2005) (http://nrca.info). Given the complexity of this project, the group is divided into two work groups, Dietary Assessment and Behavior (DAB) and Quality of Life (QoL). Each group will elect a chair, or co-chairs, and a secretary at the annual meeting. The entire group will elect a chair and co-chair. These elected positions will comprise the Executive Committee (EC). Efforts will be made to ensure that the EC will include representatives with research and EFNEP experience.

The EC will schedule conference calls (monthly for work groups and biannually for whole group); ensure project progress; complete annual reports to AES; and coordinate writing assignments (reports, manuscripts, recommendations, and papers or presentations at professional meetings). The entire group annually reviews, and revises as needed, procedural guidelines regarding cost sharing, authorship and publication, research procedures, and administrative functions.

Literature Cited

Aragones A, Schaefer EW, Stevens D, Gourevitch MN, Glasgow RE, Shaw NR. Validation of the Spanish translation of the Patient Assessment of Chronic Illness Care (PACIC) survey. Prev Chronic Dis 2008;5(4). http://www.cdc.gov/pcd/issues/2008/oct/07_0180.htm. Accessed November 8, 2017.

Arnold CG, Sobal J. Food practices and nutrition knowledge after graduation from the Expanded Food and Nutrition Education Program (EFNEP). J Nutr Educ Behav 2000;32(3): 130-138.

Auld G, Baker S, Bauer L, Koszewski W, Procter SB, Steger M. EFNEP’s impact on the quality of life of its participants and educators. J Nutr Educ Behav 2013;45(6): 482-489. 

Auld G, Baker S, Infante N, Inglis-Widrick R, Procter SB, Steger M, Yerxa K. EFNEP's impact on exemplary educators' quality of life. J Nutr Educ Behav 2016;48(9): 647-654.e1.

Campbell DT, Stanley JC. Experimental and quasi-experimental designs for research. Ravenio Books; 2015.

Carter R, Lubinsky J, Domholdt E. Rehabilitation research: Principles and applications. Amsterdam, Netherlands: Elsevier Health Sciences; 2013.

Centers for Disease Control and Prevention. Breastfeeding. Infant feeding practices II and its six year follow-up. Atlanta, GA: Centers for Disease Control and Prevention, US Dept of Health and Human Services; 2014. https://www.cdc.gov/breastfeeding/data/ifps/index.htm. Accessed November 8, 2017.

Centers for Disease Control and Prevention. Pediatric Nutrition Surveillance System: Summary of Trends in Growth Indicators by Age. Atlanta, GA: Centers for Disease Control and Prevention, US Dept of Health and Human Services; 2011. http://www.cdc.gov/pednss/pednss_tables/pdf/national_table6.pdf. Accessed November 8, 2017.

Davis GA. Using a retrospective pre-post questionnaire to determine program impact. J Ext [online]2003;41(4TOT4). https://www.joe.org/joe/2003august/tt4.php. Accessed November 8, 2017.

Diener E, Diener M, Diener C. Factors predicting the subjective well-being of nations. J Pers Soc Psychol 1995;69(5):851-864.

Diener W, Biswas-Diener R. Will money increase subjective well-being? Soc Indic Res 2002; 57:119-169.

Diener E, Seligman MEP. Very happy people. Psychol Sci 2002;13:81-84.

Drenowski A. Obesity, diets, and social inequalities. Nutr Rev 2009;67:S36-S39.

Gills SMH, Baker SS, Auld G. Collection methods for the 24-hour dietary recall as used in the Expanded Food and Nutrition Education Program. J Nutr Educ Behav 2017;49(3): 250–256.e1. https://doi.org/10.1016/j.jneb.2016.10.009. Accessed November 8, 2017.

Guh DP, Zhang W, Bansback N, Amarsi Z, Birmingham CL, Anis AH. The incidence of co-morbidities related to obesity and overweight: A systematic review and meta-analysis. BMC Public Health. 2009;9(8): http://www.biomedcentral.com/1471-2458/9/88. Accessed August 22, 2017.

Grummer-Strawn LM, Li R, Perrine CG, Scanlon KS, Fein SB. Infant feeding and long-term outcomes: Results from the year 6 follow-up of children in the Infant Feeding Practices Study II. Pediatr. 2014;134(suppl 1):S1-S3.

Horodynski M, Olson B, Arndt MJ, Brophy-Herb H, Shirer K, Shemanski R. Low-income mothers’ decisions regarding when and why to introduce solid foods to their infants: influencing factors. J Community Health Nurs. 2007;24(2):101-118.

Howard GS, Dailey PR. Response-shift bias: A source of contamination of self-report measures. J Appl Psychol. 1979;64(2):144-150.

Howard GS, Ralph KM, Gulanick NA, Maxwell SE, Nance DW, Gerber SK. Internal invalidity in pretest-posttest self-report evaluations and a re-evaluation of retrospective pretests. Appl Psychol Meas. 1979;3:1-23.

Howard GS, Schmeck RR, Bray JH. Internal invalidity in studies employing self‐report instruments: A suggested remedy. J Educ Meas. 1979;16:129-135.

Lioret S, McNaughton SA, Spence AC, Crawford D, Campbell KJ. Tracking of dietary intakes in early childhood: The Melbourne Infant Program. Eur J Clin Nutr. 2013;67(3):275-291.

Lynn MR. Determination and quantification of content validity. Nurs Res. 1986;35(6):382-386.

Marshak HH, De Silva P, Silberstein J. Evaluation of a peer-taught nutrition education program for low-income parents. J Nutr Educ. 1998;30:314-22.

Marshall JP, Higginbotham BJ, Harris VW, Lee TR. Assessing program outcomes: Rationale and benefits of posttest-then-retrospective-pretest designs. J Youth Dev. 2007;2:118-123.

Maslow AH. Motivation and Personality. New York: Harper & Row; 1954.

Murray EK, Auld G, Inglis-Widrick R, Baker S. Nutrition content in a national nutrition education program for low-income adults: Content analysis and comparison with the 2010 Dietary Guidelines for Americans. J Nutr Educ Behav. 2015;47(6):566-573.

Murray EK, Auld G, Baker SS, Barale K, Franck K, Khan T, Palmer-Keenan D, and Walsh J.  Methodology for Developing a New EFNEP Food and Physical Activity Behaviors Questionnaire J Nutr Educ Behav. 2017:49(9):777-783. http://dx.doi.org/10.1016/j.jneb.2017.05.341

Ogden CL, Lamb MM, Carroll MD, Flegal KM. Obesity and socioeconomic status in adults: United States, 2005–2008. https://www.cdc.gov/nchs/data/databriefs/db50.pdf. Published 2010. Accessed August 27, 2017.

Peeters A, Barendregt JJ, Willekens F, Mackenbach JP, al Mamun A, Bonneux L; NEDCOM, the Netherlands Epidemiology and Demography Compression of Morbidity Research Group.  Obesity in adulthood and its consequences for life expectancy: a life-table analysis.  Ann Intern Med. 2003;138:24-32.

Pratt CC, McGuigan WM, Katzev AR. Measuring program outcomes: using retrospective pretest methodology. Am J Eval. 2000;21(3):341-349.

Scholl J, Paster A, Jankowski B. Establishing a research base for the Expanded Food and Nutrition Education Program (EFNEP). Fam Consumer Sci Res J. 2011;39(3): 279-288.

Shilts M, Smith D, Ontai-Grzebik L, Townsend MS. Evidence to support the use of the retrospective pretest method to measure dietary behavior and self-efficacy in adolescents. J Youth Dev. 2008;3:080301RS002.

Spruance LA, Douglass D, Zimmerman TP, Guenther PM, Franck K, Head D, Henson T, Millerberg N, Moore CJ, Wilson-Sweebe K, Wood G, Durward CM. Online ASA24 training manual pilot-tested with Expanded Food and Nutrition Education Program (EFNEP) educators. J Nutr Educ Behav. 2017;49(7):S94-S95. http://dx.doi.org/10.1016/j.jneb.2017.05.167. Accessed November 14, 2017.

Thompson AL, Mendez MA, Borja JB, Adair LS, Zimmer CR, Bentley ME. Development and validation of the Infant Feeding Style Questionnaire. Appetite. 2009;53(2): 210-221.

Townsend M, Wilson M. Validity of the group-administered 24-hour diet recall as used by EFNEP. [White Paper]. 2016.  https://www.nimss.org/system/ProjectAttachment/files/000/000/091/original/24HDR.EFNEP. White.Paper.NOV2016.pdf. Accessed July 25, 2017.

US Census Bureau. American FactFinder. Poverty status in the past 12 months of families: 2011-2015 American Community Survey 5-year estimates. 2017. http://factfinder.census.gov. Accessed August 17, 2017.  

US Department of Agriculture. The Expanded Food and Nutrition Education Program policies. Updated 2015.  https://nifa.usda.gov/sites/default/files/program/EFNEP%20Policy%20Document%202015%20Update%20P1.pdf. Accessed August 27, 2017.

US Department of Agriculture. National Institute of Food and Agriculture. EFNEP 2016 National Data Reports.  https://nifa.usda.gov/resource/efnep-2016-national-reports. Accessed August 27, 2017.

US Department of Health and Human Services. 2008 Physical Activity Guidelines for Americans. Washington (DC): US Department of Health and Human Services; 2008. https://health.gov/paguidelines/pdf/paguide.pdf. Accessed August 22, 2017.

US Department of Health and Human Services, US Department Agriculture. 2015–2020 Dietary Guidelines for Americans. Washington (DC): US Department of Health and Human Services, US Department Agriculture; 2015. https://health.gov/dietaryguidelines/2015/guidelines/. Accessed August 17, 2017.

Van Staveren WA, Ocke MC, De Vries JHM. Estimation of dietary intake. In: Erdman J, Macdonald IA, Zeisel SH, eds. Present knowledge in nutrition. 10th ed. Hoboken, NJ: John Wiley & Sons, Inc; 2012.

 

Attachments

Land Grant Participating States/Institutions

AR, CA, CO, CT, FL, GA, ID, KS, KY, MD, ME, MI, NJ, NV, SD, TX, UT, WA, WI, WY

Non Land Grant Participating States/Institutions

University of Utah