Developing Simulated Learning Experiences

Picture of Julie Scharton
Evaluation Tools
by Julie Scharton - Sunday, October 18, 2009, 5:42 PM

I am a graduate student at Bethel University in St. Paul, MN.  For my Master's project I am implementing an End-of-Life care simulation scenario with senior nursing students in the BSN program.  I am in need of a validated evaluation tool to have the student fill out upon completion of the excerise.  Does anyone know of an existing tool that I may use as a refrence?  The implementation will take place in Feb. and I'd like to have a tool designed by late January--at the latest.  Thanks in advance for any help you can offer.

Julie Scharton, RN

Picture of Suzie Kardong-Edgren
Re: Evaluation Tools
by Suzie Kardong-Edgren - Sunday, October 18, 2009, 9:31 PM
Many congratulations on being smart enough to not invent your own tool in a short period of time! There are many tools in print already including the tried and true NLN tools, which you might investigate. Two colleagues and I have a review of currently published  evaluation tools coming out in the January issue of Clinical Simulation in Nursing. Probably too late for you. Will students do the  scenario alone or in a group? This will impact the kind of tool you want to use also. There are some out there for groups and some for individuals.
Picture of Vicky Allen
Re: Evaluation Tools
by Vicky Allen - Sunday, November 15, 2009, 9:26 PM
I am also using critical thinking development through simulation use as my master's research project.  Is there a web site or a phone number or a specific article that has the actual tools in it? I would like to do a quantitative study and most of the research I have found has been on student perceptions rather than measuring performance. Thanks for the help. Vicky Allen
Picture of Cheryl Landis
Re: Evaluation Tools
by Cheryl Landis - Monday, November 16, 2009, 10:29 AM
Vicky, although i don't have information for you, I would love to read your study/research! Sounds very good!! Please let me know. And best wishes w/ your MSN. I just finally graduated w/ mine in May! Yahoo!
Picture of Suzie Kardong-Edgren
Re: Evaluation Tools
by Suzie Kardong-Edgren - Monday, November 16, 2009, 10:44 AM

as a matter of fact, there are many articles with tools in them published, and many more with information about the tools in them with a request to contact the authors for more information. Two colleagues and I have an article coming out in Clinical Simulation in Nursing in the January issue, cataloging 21 or 22 currently published tools available for simulation evaluation. You can start searching now for the many that do NOT focus on self reported competence and confidence or wait for our article. :)

Suzie Kardong-Edgren

Picture of Ruth Blomquist
Re: Evaluation Tools
by Ruth Blomquist - Tuesday, December 8, 2009, 6:30 PM
Dear Suzie,
I am about to formalize my graduate thesis using simulation to teach experienced NICU nurses how to institute induced hypothermia in hypoxic neonates which is a high risk, low frequency, random opportunity...( love these descriptive terms..but can't remember who to cite) anyway..I will need some good self-efficacy see if the program will give them enough confidence to be able to be able to do the admission.. Can you suggest a tool I could customize to my needs...Most Sincerely, Ruth Blomquist, BSN, RN (soon to be MSN) Brigham and Womens Hospital, Boston, MA
PS: Can't wait to get the January issue..
Picture of Suzie Kardong-Edgren
Re: Evaluation Tools
by Suzie Kardong-Edgren - Tuesday, December 8, 2009, 10:27 PM
Many congratulations Ruth! I would suggest looking at these articles along with the affective self confidence scales in the NLN tools. Having said that, we are very suspect of self report confidence. See our article for why. :)

McCausland, L., Curran, C., Cataldi, P. (2004).

Abdo, A. & Ravert, P. (2006).

Schoening, A., Sittner, B., Todd, M. (2006).

Mole, L. & McLafferty, I. (2004).

Arnold, J., Johnson, L.M., Tucker, S., Malec, J. Henrickson, S., Dunn, W. (2009).

Lambton, J., Pauly O’Neill, S., Dudham, T. (2008).
Picture of Julie Scharton
Re: Evaluation Tools
by Julie Scharton - Wednesday, December 9, 2009, 10:37 AM

Hi Suzie,

I'm planning on using the self-confidence tool from the NLN.  You say you are suspect of self-reported confidence.  Why is that?  Do students tend to overestimate or underestimate their confidence level? 

We are mostly interested in knowing how they feel about the simulation and whether they feel the knowledge they gain will "transfer" to the real world.  I wish your article was coming out a little sooner, but we'll make do!


Picture of Suzie Kardong-Edgren
Re: Evaluation Tools
by Suzie Kardong-Edgren - Wednesday, December 9, 2009, 12:18 PM

Read: Davis, D., Mazmanian, P., Fordis, M., Harrison, R.V., Thorpe, K., Perrier, L. (2006). Accuracy of physician self-assessment compared with observed measures of competence. Journal of the American Medical Association, 296, 9, 1094-1102.

I would not waste my time asking students how they feel about simulation anymore, we know they like it. (And I think you would have a very hard time getting published with this, if this is your aim.)That is the lowest form of evaluation, Kirkpatrick's Level 1. You are proposing then, to ask them to evaluate for themselves whether they will transfer knowledge to the workplace, Kirkpatrick's Level 3. I am not sure students are the right one's to ask, as they really don't know what our jobs entail, even in school. I would ask their teachers or preceptors if they see a difference in the students. But the BEST evaluation of transfer to the workplace would be if the clinical instructors did not know which students had been in simulation. Now you are on to something.

Coupling the tool you propose with something meatier is a good idea. Get two or three data points, and you have something useful. :)

Picture of Ruth Blomquist
Re: Evaluation Tools
by Ruth Blomquist - Wednesday, December 9, 2009, 2:29 PM
Susie, You are right about the need to focus more on external assessment techniques. After reading Davis et al. I see your point..Now in order to measure performance, one might need a skilled observer with a checklist? But is there a tool that can measure teamwork or collaboration as performed?
Picture of Susan Prion
More about direct assessment/evaluation tools
by Susan Prion - Wednesday, December 9, 2009, 5:02 PM
Ruth, the need for direct assessment (rather than self report and other types of "indirect" assessment) is the key to validating the instructional efficacy of simulation or any other type of teaching/learning strategy. In order to develop a meaningful, reliable and valid tool, one must first carefully and thoughtfully develop clear learner outcomes. Those outcomes, with their unambiguous descriptions of what the learner is expected to accomplish, then serve as a template for your direct assessment tools.

In the situation you describe, you could develop an observer checklist based on those aspects of performance you want to measure. Depending on your operational definition of "teamwork" and "collaboration," you can ask the observers to note whether or not the participants met those expectations. You could also videotape the interaction and ask reviewers to rate the participants on those defined behaviors. You could then triangulate those data with the self-assessment of the same defined behaviors by the participants, and perhaps ask them to rate each other or the entire team on the same data points. You could also look at objective data like how long it took the team to accomplish some critical tasks, and reevaluate later to see if those times decrease with more effective teamwork (according to your definitions) and collaboration. Your development of direct assessment tools and strategies are guided by your learner outcomes and limited only by your creativity!

I would also check the organizational behavior and leadership research, especially in the business discipline, as there are some terrific tools for demonstration/observation of these same behaviors.

Just like as 12-lead ECG, no one data collection tool will give you all the information you need. But using a multiple method, multiple perspective approach and adding the data together, you start to get a very good idea of what is going on.

I wrote an article providing a conceptual framework for evaluating simulations that is available online through Clinical Simulation in Nursing. The citation is:
Prion, S.K. (2008). A practical framework for assessing the impact of clinical simulation experiences in the nursing curriculum. Clinical Simulation in Nursing Education, 4, e69-78.

This is an area of interest and expertise for me, so please contact me if you have additional questions.
Picture of Ruth Blomquist
Re: More about direct assessment/evaluation tools
by Ruth Blomquist - Wednesday, December 9, 2009, 8:28 PM
Dear Susan,
Unfortunately my institutional library only has the journal for 2009.....In the meantime...until I can find the article...I will rethink my assessment tool design and work on some clear outcomes as you suggest...for individual performance and for group performance...I agree that evaluating multiple perspectives will give a much better result. Thank you for your comments! Ruth
Thank you so much for jumping in Susan! I almost quoted your article but ran out of steam. Way to go, thank you!!! I think this is very helpful to anyone  thinking about doing research in simulation.
Picture of Suzie Kardong-Edgren
Re: Evaluation Tools
by Suzie Kardong-Edgren - Wednesday, December 9, 2009, 9:02 PM

yes  there are several available in journals. The group at, the University of Aberdeen has done extensive work in this area, with ANTS tool. These authors below have also done work and produced a tool.

Malec, J., Torsher, L., Dunn, W., Wiegman, D., Arnold, J., Brown, D., Phatak, V. (2007).

Millward, L. & Jeffries, N. (2001).

Picture of Suzie Kardong-Edgren
Re: Evaluation Tools
by Suzie Kardong-Edgren - Wednesday, December 9, 2009, 9:06 PM
no, some are in Clin Sim, some are not. You have some citations now, you will have to do some exploring.