2018 CE Outcomes Standardization Project (OSP): Just the Beginning …

By Brian McGowan, PhD, FACEhp, Co-Founder and Chief Learning Officer, ArcheMedX Inc.

There were two driving forces that led to a groundswell of momentum and support coming out of the Alliance Industry Summit meeting this past May. First, in conversations building off the morning’s keynote and in the midst of Greg Salinas’ session titled “Standardization of Outcomes: Lessons Learned from Attempts to Aggregate,” a small group of colleagues/attendees hatched the idea of an outcomes standardization project (OSP).

While these conversations were energizing, shortly thereafter the Regeneron CME team (including Karyn Ruiz-Cordell) presented a session titled “The Tower of Babel: Enough Already, Can We Please Speak the Same Language?”, and the groundswell rapidly grew. In the weeks that followed, the Outcomes Standardization Project (OSP) Steering Team took shape, and a plan of action was devised that relies fully on volunteer energies, galvanized around a passion to solve the following (working) problem statement:

“The community struggles to effectively understand and communicate the value of CE in part because the community lacks a standardized outcomes language and a set of standardized approaches to measuring impact … as a result, effectively comparing and aggregating outcomes data and insights remains impossible.”

Since May of 2018, the OSP Steering Team has been executing on an aggressive plan of action with a goal of delivering a workable draft definitions document by year’s end. This effort has included hundreds of hours of research, debate and outreach. The OSP Steering Team includes Anthia Mandarakas (Sanofi); Brian S. McGowan, PhD, FACEhp (ArcheMedX Inc.); Greg Salinas, PhD (CE Outcomes); Jason Olivieri, MPH (Med-IQ); Karyn Ruiz-Cordell, MA, PhD (Regeneron); Sue McGuinness, PhD, CHCP (Astra-Zeneca); and Wendy Turell, MS, DrPH, FACEhp (Platform Q). A special thanks should go out to our employers for seeing the value of this effort and allowing each of us the freedom to contribute to the project.

General Plan of Action

The general plan of action for the OSP that has directed our efforts over the past five months consists of the following six phases:

  1. Phase One: (completed, May-September)
    1. Steering Team will meet, frame mission and lock down the plan of action.
    2. Create an initial working document of terms, concepts and
    3. Author initial community outreach (background and planning).
    4. Prepare for direct interviews with community thought leaders.
  2. Phase Two: (completed, September-October)
    1. Implement and archive interviews.
    2. Collate community thought leader feedback and insights on terms, concepts and
    3. Evolve the working document through Steering Team consensus.
    4. Author ongoing community outreach (project status update, #CMEpalooza session).
    5. Prepare for focus groups with additional community thought leaders.
  3. Phase Three: (in progress, October-November)
    1. Implement and archive focus groups.
    2. Collate focus group feedback and insights on terms, concepts and
    3. Evolve the working document through Steering Team consensus.
    4. Author ongoing community outreach (project status update).
    5. Prepare for call for comments with broadest possible audience of CEhp professionals.
  4. Phase Four: (November-December)
    1. Implement and archive call for comments.
    2. Collate call for comments feedback and insights on terms, concepts and
    3. Evolve the working document through Steering Team consensus.
    4. Author ongoing community outreach (project status update).
    5. Prepare for final hack-a-thon working session.
  5. Phase Five: (December-March)
    1. Prepare for dissemination and community outreach.
    2. Ensure that the project output is fully open-source and accessible.
    3. Implement an ongoing dissemination and advocacy plan.
  6. Phase Six: (2019)
    1. Identify and prioritize the proper next set of terms, concepts and definitions to standardize and continue to extend the OSP dictionary over time.


Finally, to execute on this plan, the OSP Steering Team has chosen a path slightly less traveled, but one we hoped afforded us the ability to move quickly and to most efficiently respond to the community’s feedback. While past attempts to improve outcomes methods and reporting have depended on traditional approaches to secure funding through a research grant or working through formal committees and taskforces; instead, we committed to organizing this as a grassroots movement. And, to date, this decision has proven to have galvanized the Steering Team and accelerated our work. But, ultimately, the success of the Outcomes Standardization Project is dependent on the community’s support and application of the newly standardized terms, definitions and best practices. We believe this is best ensured by committing to two guiding principles, namely: broad community contribution — thus, the Phase Two interviews, Phased Three focus groups and Phase Four call for comments — and complete transparency — thus, this Almanac article (others will follow), the upcoming #CMEpalooza session, monthly progress updates through the CME group on LinkedIn, the OSP-focused Twitter account (@OutcomesinCE), and other emerging channels that document our efforts and progress. Once the project has progressed to Phase Five — and all the available feedback is collated and addressed — it is the shared vision of the OSP Steering Team that the project output is available without restriction, without copyright and without embargo. This is a project by the community and for the community.

While the current standardization efforts are in still a work in progress, the OSP Steering Team has agreed on the initial set of terms, concepts and definitions to be included. This list was narrowed down from nearly 100 proposed items:

  1. Learner Funnel
  2. Exposure
  3. Participant
  4. Learner
  5. Start
  6. Engagement
  7. Completion
  8. Completion Rate
  9. Moore’s Level 1, Participation
  10. Moore’s Level 2, Satisfaction
  11. Moore’s Level 3a, Declarative Knowledge
  12. Moore’s Level 3b, Procedural Knowledge
  13. Moore’s Level 4, Competence
  14. Moore’s Level 5, Performance
  15. Moore’s Level 6, Patient Health
  16. Moore’s Level 7, Community Health
  17. Pre-Test
  18. Post-Test
  19. First Post-Test Score
  20. Final Post-Test Score
  21. Evaluation
  22. Follow-up Assessment/Evaluation
  23. Summative Assessment/Evaluation
  24. Formative Assessment/Evaluation

Finally, as the title of this article suggests, this is just the beginning. The Steering Team is committed to revisit the full list of proposed terms, concepts and definitions once the general plan of action is complete and, as part of the community outreach and feedback we continue to collect suggestions on what should be included in the 2019 OSP effort

If you have ideas to share with the OSP Steering Team, or if you want to volunteer to support the effort in anyway, please contact us through email at OutcomesInCE@gmail.com or through Twitter @OutcomesInCE.

Alliance note: The Alliance’s CE Provider/Supporter Interactions SIG is working on an initiative, which includes identification, vetting, standardizing and collating professional vocabulary to improve communications across the full spectrum of learning. This will help develop a lexicon of terminology that will be a professional resource and disseminated globally across our interprofessional community of continuing educators. To learn more, click here.

Recent Stories
Alliance Podcast Episode 18: Spotlight on Leadership

Comparison of In-Person Versus Tele-Ultrasound Point-of-Care Ultrasound Training During the COVID-19 Pandemic

Rockpointe Presents Online CME Course on Treating High-Risk Patients With COVID-19