Revision Report for:  ms 111-v2_debriefing-as-learning

Reviewer’s comments

Authors’ response

Comment 1

“I have several concerns, but the overarching one is that what is being proposed is not new. The paper presumes that coaching while being an active participant in simulation based (or game based) training is a brand new idea. I have personally witnessed this approach dozens of times in military training. Just because it might be a less common practice in civilian settings does not justify the claim of discovery of a new approach for instructors. Instructors have changed scenarios and role played in training (as participants), and even sat in as worker-trainees, for a very long time.“

We agree that, within the context of military training (and, to some extent, live role-playing exercises in civilian contexts), the idea of an active instructor or coach is common practice. Our view, however, is that this practice is an implicit assumption that has not, by and large, been explicitly described and grounded in current theories related to game-based training and cognition/learning. In contrast, there are those (e.g. Klabbers) that assert that the instructor should not disrupt training, but instead take a step back and only observe how the game evolves.

We have softened our claim that this part of the framework is new and further clarified the contribution of the coaching cycle to the general serious game community (further described in our response to comment 6).

Comment 2

“The argument that instructors should play with their students so that they can relate to the trainee experience is also not new, and I would argue that every instructor should go through the training experiences they are going to provide to students prior to instructing others with it (and I think that this is also common practice).“

Even if this is common practice within the military domain, it is largely overlooked in other (civilian) digital game-based training contexts. At least, we have found very few accounts of instructors as game players in the literature. Thus, we see that the general GBL community could benefit from a theoretically grounded description of this practice.

The clarifications made for some other comment also relate to this comment.

Comment 3

“I also do not think that the presumption that playing with students will necessarily increase deliberate practice. Deliberate practice is something that students put into the experience. It is not something that is put into the students by instructional approaches. It is an age-old idea to employ instructional approaches that provide information, practice, and motivate students to try hard. Everybody knows that a knowledgeable, dedicated, exciting instructor can inspire student motivation. The interest in using games, gaming approaches or game characteristics is to attempt to crack the motivation nut without requiring an exceptional instructor to do so.”

We do not claim that playing with students directly increase deliberate practice. What we claim is that:

1.      Deliberate practice is essential for the development of expertise and coaching is an important part of increasing deliberate practice

2.      Games and simulations are good for deliberate practice because they offer continuous feedback and opportunities for repetition

3.      Only relying on in-game automatic feedback could, however, lead to a pattern of incorrect behaviors that are hard to unlearn

4.      Instructors involved in gameplay can give continuous feedback in a flexible and unobtrusive way (as a complement to automatic/game feedback)

5.      Coaching does not only occur during gameplay, but is also part of the debriefing (as summative feedback and reflective exercises), so that students know what to improve during the next gaming session.

We believe that coaching to increase deliberate practice has more to do with giving feedback and spotting weak spots in trainees’ performance (both on an individual and a group level) than extrinsic motivation. We don’t think that games can replace the instructors, neither in terms of feedback or motivation – at least not with current level of game AI. A game-based training approach still needs a highly competent instructor to succeed. Otherwise, there is a risk that the students learn how to win the game instead of picking up the intended learning goals. As serious games researchers we must take into account the situational aspects of serious gaming, i.e. studying not only the game, but also the physical, social, and organizational contexts in which it is used, including human actors.

We have clarified these points in the section labeled The power of game-based training.

Comment 4

“The goal is to inspire student motivation while reducing the use of costly human instructors, hence ICW, web-based instruction, and training games. The authors acknowledge that cost limits what is possible in training situations (only one instructor and no trainer operator for civilian fire fighting training) but they do not address the need to improve and/or maintain training effectiveness while reining in cost.”

As pointed out, we acknowledge the problem, but consider it outside the scope of this paper to fully examine the cost-effectiveness of the framework. Our focus is training quality.

We have added some text addressing the cost-effectiveness problem in the discussion.

Comment 5

“A paper that recommends leveraging standard military approaches to civilian training would be a more reality-grounded approach, but would probably not be worth publishing in Simulation and Gaming.”

We concur that the paper would be greatly improved if it was pitched more clearly towards a more general game-based training audience. We have revised the paper accordingly.

It is our hope that the editor of Simulation & Gaming finds it appropriate for this journal.

Comment 6

“It is obvious that the authors have read a great deal of literature related to the issues they discuss, but I don't feel that they have clearly articulated what is new and, importantly, what is the value (monetary and training effectiveness related) of their recommendations. An article with increased clarity of these two issues and data to back them up could be worth publishing.”

We have tried to improve the paper according to these issues (see above comments). To sum up, what is “new” about the framework is:

1.      Current training practices are explicitly articulated

2.      Increased focus on debriefing than current practices

3.      Less focus on initial lecture/briefing than current practices (thus giving more time for gaming and debriefing)

4.      Integrated training cycle that combines current training practices with theories from deliberate practice, experiential learning and game-based learning, i.e. a theoretically grounded framework

The value of this framework is related to the development of future serious games. Even though there are promising attempts to create more advanced AI agents in games we are still far from handling the complexity and flexibility that are prevalent in today’s training – at least to the point where the AI system can replace the instructors’ role during game-based training. However, studying what the instructors are actually doing, we can create requirements for such a system. In fact, this is an important part of our future work.

Reviewer’s comments

Authors’ response

Comment 1

“Reviewer A indicates that many of the recommendations have been implemented by militaries (but not perhaps public education). I have to agree with this statement. You paper is interesting, please don’t get me wrong, but I believe your paper would have more impact if you acknowledge prior work or even instances in military training in which instructors are being/have been brought into the gaming experience.  You do this a bit in the section called “the many roles of the instructor in game-based training” and I believe you could mention other work here, primarily work I have done in creating in-game roles for instructors, peers, and training cadre (this is a shameless plug for my own work which I hoped you would have found in your literature research so I hope you don’t mind if I have attached it directly for your perusal). I have other papers on the topic of multiple roles in-game for instructors and trainees to provide feedback in-game as well as to an automated AAR of logged game events. This approach has been instantiated in two U.S. military games. We have put instructors in these roles and the 2009 paper discusses data collected with trainees in the roles, not instructors, but you may be interested nonetheless as this approach seems to codify your coaching cycle notion.”

We have changed the paper to be more overtly targeted towards more general game-based training readers, for whom military practices are not widely known.

Your papers are good contributions to our argumentation – thank you!

Comment 2

“Given my point above, in the second paragraph, 1st page, you mention that “they (instructors) are reduced to mere observers, or, in the case of distance…” I would suggest that you soften this statement as my own experience with the US military in designing systems as well as observing training events would suggest that the instructor is always an active part of the process, in one way or another.”

The sentence has been rephrased.

Comment 3

“Paragraph 4 of the first section describes coaching and deliberate practice—when I read this, I asked myself, how is this scaleable to groups in the methods you describe?”

Our work does not get a clear answer to this question, but we agree that it is important. We do not wish to promote a framework that requires one-on-one tutoring, but we also acknowledge that coaching is difficult with too large groups. As we see it, larger groups have to rely on (1) automatic feedback from the AI system and (2) peer-to-peer feedback.

We have commented on this in the discussion.

Comment 4

“The third paragraph of your research approach suggests that you have collected and analyzed empirical data to develop your concepts. You really need to say more about this in detail (what did the data tell you?) or delete this paragraph from the paper as the reader expects to see a data analysis section in the paper that is never presented.”

We have added a paragraph on data analysis in the research approach section.

Reviewer’s comments

Authors’ response

N/A

N/A

Section of Paper

Comments

Response

Abstract

Introduction

Review of the literature

Although games and simulations offer some affordances that other mediated learning do not have, it is easy to get into the trap of viewing “simulation” or “game” as some kind of intervention, much like computer-assisted instruction was once conceived, resulting in media comparison studies of CAI vs. “traditional” instruction.  Richard Clark’s argument is that the media don’t influence learning, rather it is the effective use of instructional strategies and tactics.  For example, feedback to learners (correction of error) can be provided in a variety of contexts for learning, including simulations and games, and feedback is part of effective instruction  (e.g., see Merrill’s First Principles of Instruction—especially the ‘application’ principle).  The good news here is that the present study is not another “media comparison” study.   But the danger is nonetheless there to treat the idea of simulation or game as if it is a variable itself (or theoretical construct, which is not well-defined).  There can be ineffective simulations and games for a lot of reasons, because they do not employ what is known from instructional and learning theories (again, see Merrill, and also see Clark’s arguments). 

The authors concur that this study is not a media comparison study.

Research Questions

Methodology

Yes, but why would they not also apply what they had learned in the course?  And in fact, you later discover that they did.  So this basically renders the first research question meaningless.

-----------------------------------------------------

With unequal and relatively small n’s, any kind of statistical inference would be difficult to justify—especially if the within-group variances are not equal.

So why was the purple debriefing sheet NOT used that comes with the DSG?  It has a list of discussion items that reinforce many of the concepts and relationships from Rogers’ theory. 

Research questions were rewritten to clarify the questions the researchers investigated. 

In rewriting the paper, participants in the study were viewed as one group. 

Paper was modified to reflect that additional change models were used in the development of the Diffusion Simulation Game. 

The authors were not aware of a debriefing guide for the Diffusion Simulation Game until reviewers mentioned it. The paper now includes this information.

Results

Means should always be accompanied by the N and SD.

More than interesting, this makes the first research question meaningless.  The groups do not differ on the one dimension that presumably was manipulated (i.e., whether on not a theory was applied in playing the DSG).  Given this methodological flaw from which no recovery is possible, you might as well just talk about everyone as one big group.  The A vs. B distinction is irrelevant.

 N and SD were added to the results table.

Corrected labeling to Tables (not figures).

Revisions to the paper treat all participants as one group.

Discussion

It makes no sense to talk about A vs. B comparisons, since the groups are more alike than different.  The theory mismatch argument may be compelling initially, but if students learn from their experience, that should override whatever theory they might have in mind.  What a different theory could provide is dissonance that might interfere initially with learning from the experience.

Again, I have to ask, why were the debriefing materials provided with the DSG NOT utilized.  This debriefing was crucial for the board version of the DSG and is just as important for the online version.  Unfortunately, if the DSG is played outside a course context, there is no obvious way to control the debriefing.  People just play the game and draw whatever conclusions they may.  If they play the DSG enough, they may learn something that is consistent with theory about diffusion of innovations. 

 Thank you for the comment about the nature of the Diffusion Simulation Game being stochastic. The comment helped the researchers better understand and interpret survey results.

Limitations of the study

Conclusions

Section of Paper

Comments

Response

Abstract

None

Keywords

“I don’t think all of these words work as efficient keywords. Try narrowing down.”

Narrowed down to effective keywords

Introduction

1.     Suggested clarifying what students wrote. …“reflection what? Paper Journal?”
 

2.     Used a instead of one

…“was there more than one weakness? Maybe change to a”

1.     Changed to “wrote a refection paper about the experience”
 

2.     Changed to “a weakness of the study was in the area of debriefing”

Review of the literature

End of 2^nd paragraph.  …Thus, as players learn and experience the rules of a game by assuming a specific role, … “this reads like a run-on sentence. Try breaking up” 

Broke up to two sentences

Research Questions

None

Methodology

1.     Tense, 3^rd , 4^th, and 7^th paragraphs. “since this is your study and this is what you did, shouldn’t this be in past tense?”
 

2.     Name: The survey consisted of sixteen Likert scale. “technically it should be referred to as a Likert response format”

1.     Changed to past tense.
 

2.     Changed to “The survey consisted of sixteen questions”

Results

None

Discussion

“In this section it is not clear where the answers to the research questions are. While I think the author has answered the questions, the research questions should be specifically addressed here followed by the evidenced based answer (s)”

Research question one was addressed in paragraph 1-4 with discussions on the issue of game logs, the use of moves, and adopters gained in both groups. Research question two was addressed from paragraph 5- 10.

Limitations of the study

None

Conclusions

Although some negative comments about the game were indicated … “make this active voice”

Changed to “While there were some negative comments about the game…”

Appendix I

Likert scale

Changed to Likert response format

Figures

Figure 2 Survey Results “although the distribution of scores graphic is a terrific idea, technically for descriptive statistics the standard deviation should be presented along with the mean, even with a small N.”

Standard deviation was included in the table. (all “Figures” were changed to be “Tables” per APA guidelines)

Section of Paper

Comments

Response

Abstract

You referred to it previously as a simulation.  The reader may not realize that the “game” you are referring to hear is the same as the “simulation” you mentioned previously.  It would be helpful to the reader to make this connection explicit before using the terms interchangeably.

A statement was added to indicate the terms “simulation” and “game” are used interchangeably when used in reference to the Diffusion Simulation Game

Introduction

Is this speculation?  Do you have evidence to cite?

Again, be specific.  It was not any simulation they viewed positively, it was the Diffusion Simulation Game.

Literature review has been extensively rewritten to provide citations for statements.

Changes were made throughout the article to refer specifically to the  Diffusion Simulation Game

Review of the literature

This is strong language for a generalization about all simulation games.  Learning is always probabilistic due to the stochastic nature of humans.  I have never seen any simulation, or any other instructional method, that is always effective in meeting its learning objectives with all learners.

Though I have not read Peterson (2010) yet, I wonder if this single piece of “strong evidence” is based on a single game.  This may be something you want to expand on.

Without more detail, I would be inclined to discredit a statement like this in the same way that I would discredit a statement that states that “I-Pads are good for learning” while disregarding the instructional method being used with the device.   The effectiveness, or lack thereof, of games and simulations result from the effective (or ineffective) design of the game.  For example, I would not expect that a poorly designed game would be effective for learning just because it is a game.

Your lit review, though fairly logical and clear, does not lead to a particular gap in knowledge as far as I can tell.  I would like to see the review concluded with a more clear gap in knowledge.  Why is there a need for this study?

Suggestions for word changes were adopted in the revisions of the literature review.

For example, the word “convincingly” was deleted.

The gap in the literature was defined as the relative low use of simulations in higher education as well as limited information for educators on how to implement a simulation/game

Research Questions

The gap in literature (not sure what that is), the goal/purpose (does DSG help students learn content), and the research questions (will students perform better when using a change model to guide their gameplay than students who play the game with no guidance) are not aligned.

 These are significant issues that should be addressed. 

Research questions were rewritten to accurately reflect what the researchers wanted to investigate and learn about.

Fidelity of the Diffusion Simulation Game and the real life change process is addressed in the results and discussion areas of the paper

Methodology

Before or after participation in the study?

This addresses a topic I mentioned earlier as well.  However, debriefing is not the same as reflection.  Whereas reflection allows a player to consider their own gameplay experience, my understanding of debriefing is that others gameplay experiences are shared as well.  In any case, a brief definition of debriefing from the literature will clear up the misunderstanding.  You may find that you need to state that you used the reflective journals “in place of” debriefing for whatever reason. 

The paper describes that participants played the game after nine change models were studied and that the Diffusion of Innovation theory was one of the models studied in the course.

Information and discussion related to debriefing was increased in this second version of the paper.  The paper now clearly states that the reflective journal was used in place of a formal debriefing protocol.

The researchers were unaware that a debriefing guide for the Diffusion Simulation Game was/is available.

Results

The average you provide for your participants is based on their “last” gameplay session.  I am guessing that the average you are reporting for past DSG players includes all gameplays.  If so, this comparison is not valid.  At the very least, this fact should be made known.  If the average number of adoptees of past DSG games is based on the “last” gameplay, then state so.  If this is so, is the number of gameplays the same?  I would expect the more you play the game, the better you get.

 Also, I wonder if you are able to get another (better) measure of game performance.  Imagine a situation in which one player has 15 people in the trial phase and only 5 adopters.  Another player has 3 people in the trial phase and has 8 adopters.  Based on your indicator, the second person performed better.  However, if you consider “closeness to adoption”, the former outperformed the latter. 

If you have the data available, I would be interested in knowing the game performance based on a closeness to adoption in addition to the total number of adopters.

This would have been good to mention earlier so it is made clear to the reader that the treatment varied between participants and that the game performance data being analyzed was of their last gameplay, and I assume you have no way of knowing how many gameplays led up to that.  This makes comparing performance difficult as those who played more games would likely perform better.

The results section was rewritten to align results with the research questions.

Closeness to adoption was not investigated in this study.  I agree that this is an interesting aspect to investigate and is a good area for future research.

Reviewers were absolutely correct that this research project was an exploratory study.

In the revised paper, participants were treated as one group.

Discussion

. This makes sense because they did not have to play for the whole 2 year calendar (because they won).  Is there something else that is being implied here?

This was an attempt at debriefing, though I still think an expert would need to be involved in the discussion to ensure misconceptions are being addressed (blind not leading the blind). 

The discussion section was rewritten to align with the research questions and to better clarify the meaning of the findings.

No data is available on the time it took to play the Diffusion Simulation Game.

Weakness in the area of debriefing is discussed in the Discussion section.

Limitations of the study

As far as I can tell, no comparisons were made between group A and group B, and definitely no statistical comparisons.  The treatments were not necessarily different as well.  I am not sure why you need to mention group A and group B at all.  It may make more sense to note that X participants were explicitly asked to use a change model and the others were not. 

Limitations section was rewritten. Study participants comprised one group.

Conclusions

You have no evidence that your results are generalizable.  This statement would be more appropriate if it were to state “in this particular graduate-level online course”

This is a generalization that cannot be made based on your study.  Not only do you not have sufficient evidence to say this about simulations in general (as you were only looking at one particular simulation), you do not have evidence to say this about online courses (as you were looking at only one online course). 

Conclusions were rewritten to limit generalizability.