Fisheries and Marine Institute of Memorial University of Newfoundland

Application of the Keller Plan to Marine Education in the 21st Century

by

John Cross and John Tucker

Abstract  The Keller Plan (sometimes called The Personalized System of Instruction or PSI) was developed by Fred Keller and his associates in the mid 1960’s as, among other things, an alternative to conventional classroom teaching as had been practiced for thousands of years.  Since then, the Keller Plan has received very positive evaluations by students and in well-designed experiments, grades on both quizzes and final exams were “significantly and considerably” higher for Keller Plan participants as compared to students in lecture sections. 

At the Marine Institute, our experience with marine education has led us to develop an instructional method very similar to the Keller Plan.   This paper will outline the benefits of the Keller Plan and how these benefits are transferable to the current model at the Marine Institute.  It will also report on the implementation of the Keller Plan and its favorable reception by current students.  Based on the experience so far, it is anticipated that students following this model will enjoy above average success in the government set exams.

Key Words:  Keller Plan, Teaching Methods, Marine Education, Applied Mechanics, Distance Education

Improving the Effectiveness of Multiple Choice Questions in Examinations towards STCW Certificates of Competency.

By

Robert Mercer, Denis Drow, Gary Jeffery & Stephen Cross

 ABSTRACT

Over the last nine years the authors have done original research into the use and effectiveness of multiple choice questions (MCQ) towards examination for STCW Certificates of Competency, presenting the survey and studies at IMLA, IMEC and other conferences.

In many countries MCQ are part of examinations towards STCW Certificates of Competency, together with constructed responses and calculations, orals and practical assessments such as workshops and simulators. MCQ are more widely used in North America and Asia than in Europe. Examination methods, including MCQ, vary from country to country and even between devolved colleges in the same country, a situation of concern to maritime labour employers, and a circumstance acknowledged as a result of culture, history and the generalised wording of international convention.

The paper reviews the advantages and disadvantages of MCQ assessment as described in the literature, based on the relevant experience in other professions, since there are no related qualitative or quantitative studies in maritime education, apart from the authors’ research. The review concerns publications mainly from years 2012 to 2014: earlier reviews back to the 1930s are covered in the authors’ previous papers.

 As with other assessment methods MCQ have strengths and limitations. The paper examines methods that may improve MCQ effectiveness, including pre- and post-test analysis; software applications, and commercial solutions including question banks. The paper questions whether MCQ variants addressing different learning levels and teaching circumstances are appropriate, or would lead to further exacerbation of the present disparate international examination situation. The paper highlights the implications for instructor training requirements, specifically teacher training that includes assessment methodologies and MCQ construction. The paper examines the advantages and disadvantages of alternatives to MCQ to assess factual knowledge.

As with the authors’ previous papers the objective is to review the literature, gather the information and present the results of their survey and studies for the contemplation of maritime instructors. Teachers must make their own judgements regarding the effectiveness of any examination method, based on their own experience and understanding of their students, at the same time considering how well assessment meets the requirements of competency described in STCW.

Towards the Creation of a Formal, Auditable Standard for the Delivery of Distance Learning and E-Learning Programs for Mariners

John Tucker and John Cross

Abstract:  As a follow up to the paper presented in the IAMU AGA14 entitled ‘Setting the Standard’ – A Proposed Standard for Delivery of Online Courses”, this paper describes the efforts of the authors at the Marine Institute in collaboration with the maritime certifying authority, Transport Canada to prepare a formal standard for the delivery of online courses for the progression of mariners in their certification.  While developed in Canada, the technologies and delivery strategies involved are global. Thus the model for creating such a standard  has international application.

Several tasks were undertaken in an effort to further refine the conceptualized standard elements presented in the AGA 14 paper.  These include:

• Consultation with the maritime certifying authority. Transport Canada
• Interviews with relevant technical experts; and
• A review of several existing guidelines  for the delivery of online courses

The guidelines reviewed focused heavily on course delivery pedagogy and failed to emphasize the technical elements we were interested in addressing in an auditable standard.  An emphasis on technology is critical due to the nature of the course participants and the unique conditions under which they will participate in these courses.

This paper presents the findings of these efforts grouped into the following categories

• Technology Requirements
• Content Delivery Format
• Course Development
• Learning Management System
• Examinations

While this is still a work in progress, a draft standard under consideration by Transport Canada shall be discussed.

 Keywords: distance learning; e-learning; audit; standards; Manila amendment; maritime education and training; MET; maritime training institutions; maritime universities

DISTANCE TEACHING STANDARD

John Cross and John Tucker

ABSTRACT

The Manilla Amendment of STCW opens the door to distance delivery of courses to mariners that are required for advancement in certification level.  The Marine Institute of Memorial University of Newfoundland has been working on preparing distance courses for deck officers and marine engineers which would prepare them for writing the Transport Canada challenge examinations as part of their certificate advancement.

With an eye to developing these courses, the authors have developed a distance delivery standard which is applied to their online course development.  It is anticipated that the standard would allow courses to be developed in such a way that certifying authorities would find the courses and all evaluations delivered by the institution acceptable.

This paper describes the content of this new distance delivery standard to include the broad categories of:

• Delivery Infrastructure
• Receiving Technology
• Content Format
• Learner Management System (LMS)
• Instructor Standards

The standard will cover aspects of the delivery ranging from required bandwidth and minimum standards of computers and devices to be able to participate in a course to minimum expectations of students to be able to receive a response to a communication with the course facilitator while taking a course.

Much of the standard has been developed based on the authors experience in delivering online and blended delivery courses and is based on current course delivery methods employed at the Marine Institute.

Keywords:  Distance Education, Standards, Educational Technology

MULTIPLE CHOICE MYSTERIES: The Effectiveness of Multiple Choice Question Assessment in Maritime Education and Examination

Denis Drown, Robert Mercer, Gary Jeffery & Stephen Cross

Abstract         

Competency certificates are issued to ships’ officers and ratings following an education and examination process that includes multiple choice question (MCQ) assessment. Over the last seven years the authors have conducted original research into the use and effectiveness of MCQ, presenting the ongoing survey and studies at IMLA and IMEC Conferences.  

MCQ are used to assess both knowledge and competence to complete onboard tasks safely. The authors’ studies show that evaluating knowing/not knowing is only an element of MCQ, since there are other factors influencing assessment. There are concerns about MCQ effectiveness and reliability, especially where significant scores can be obtained by students with only slight knowledge or understanding of the subject.

The Paper overviews the authors’ previous survey and studies involving 1480 participants from 55 countries. The Paper follows on from work presented at IMLA 20 in Terschelling. New material available from academia since 2012 is reviewed, continuing the debate about MCQ effectiveness ongoing since the 1930s. Current reports from media and other sources are presented, indicating shifts in attitudes towards MCQ assessment. The Paper suggests that in maritime education little has changed since MCQ assessment first became popular in the 1970s. 

The Paper demonstrates the authors’ study methods, and presents a preliminary report on the latest study with participants from the IMEC community, attempting to identify the degree of major influences on MCQ effectiveness.

The Paper’s objective is to encourage discussion about the place of MCQ within maritime education and examination. The Paper suggest measures to improve the reliability of MCQ assessment.

Keywords: multiple choice question assessment: competence: STCW examination.

Testing in the Electronic Age

Tucker, J, M.Eng., P.Eng. & Cross, J, ONII, M.Eng., P.Eng.

Abstract With the Manila Amendments to SCTW that introduce distance learning and web-based learning as valid course delivery systems there have been several initiatives aimed at developing distance courses for mariners.  Certainly course content is rightly where the focus is right now, but the next logical step is the electronic examination of course material; i.e. electronic tests. 

However, this is a fairly formidable step due to two main factors:

•           First, there is a culture that must be addressed in both the institute and the learners.

•           Second, the technology must be considered and evaluated.

The cultural issue involved is that of moving from a paper based process which students have used for many years to an electronic one that is probably new to them.  This transition is helped to a certain extent because computers and electronic devices are very common now.  However an electronic exam may cause stress simply because it is unfamiliar.  Care must be taken not to introduce undue stress during what is a stressful time. 

The second factor is a technological one.  The computers and the software must be able to work together to provide a suitable environment.  Change is continuous when dealing with computers, but occasionally there are very significant changes that can cause problems.   And, there are the normal aspects to any exam delivery that must be dealt with.  Items such as the safe guarding of examination results, the integrity of examination material during a test (i.e. what is the consequence of a computer crash in the middle of a test) and the proctoring of an electronic exam are all things that must be considered. 

The authors have been using electronic/online evaluation tools in the delivery of face to face and distance courses and have faced many of the challenges described above.  To address them they have developed a set of procedures for administering evaluations using this technology by distance so as to integrate electronic testing while maintaining the highest level of integrity of the evaluation process.

This paper builds on discusses these experiences and discusses some of the problems and successes that the  Marine Institute has experienced  developing, implementing and examining course content through the use of computers and other media. It also highlights the benefits of administering course evaluations using this method by comparison to traditional methods and looks briefly at what could be possible in the future.

Keyword:  Electronic Testing, Examinations

'ENGAGMENT BY DISTANCE' A DISTANCE DELIVERY METHOD MAXIMIZING STUDENT ENGAGEMENT

John Tucker and John Cross

ABSTRACT

Distance delivery of courses is a concept that is well entrenched in history but has been driven by the technology of the era. The notes and books that were once sent by post have given way to the electronic and digital reality of communicating today. However with the switch to digital communications, new tools and techniques have arisen that empowers us as educators to deliver courses as never before. This results in a teaching and learning experience that is closer to being in an actual classroom with an instructor than ever before.

Taking advantage of these new tools, the authors have been developing a distance teaching and learning methodology whereby learners are actively engaged and challenged in an online setting while learning course content. This is achieved by delivering content in a secure virtual learning environment using Flash or HTML5 videos that are interactive, periodically pausing during the delivery of information to ask questions as the instructor would in a classroom.

Assessments are performed as would be done in a classroom setting as well, but distributed to the students electronically and created using software that randomly varies the parameters in the questions such that each student receives a unique version of the assignment. In addition the assignments are marked instantaneously which also seems to increase engagement.

The authors have learned through experience that for many students, even during an asynchronous course, it is necessary to have some synchronous interaction with the course facilitator. We have developed a novel method for 'distance tutoring' which involves audio and visual interaction as well as the use of a whiteboard style application for the facilitator to draw diagrams and demonstrate calculations or other course content as appropriate.

To this point the focus of the paper is on devices employed in distance delivery, but there are two different methods the authors use to employ these devices and deliver effective courses by distance.  These are the 'Bubble Method' and 'Master Method' and are described fully at this juncture.

The final piece to the puzzle is the method of evaluation in the form of tests and examinations. This paper will conclude by describing the network that is used by the Marine Institute as part of Memorial University of Newfoundland for delivering invigilated tests and examinations to students all over the world.

Keywords: Student Engagement, Distance Delivery, etc.

                                                                                                       Developing a culture of attentiveness

                                                                                                               John Cross and John Tucker

Abstract

In 1998 Mr. William O’Neil, the then Secretary-General of the International Maritime Organization (IMO), stated that human error is the biggest threat to people working at sea. Since then, the technology involved in operating ships has improved tremendously, but human error is still at the root of most safety issues that arise. The Manila amendments to the STCW Code set out certain safety concerns that contribute to human errors. Specifically it identifies lack of training, new technology and fatigue as areas that contribute human errors. However this paper focuses on another error– that of making inadvertent errors.

 It is the intention of the authors to use our teaching experience and tools that we have developed to adjust our teaching pedagogy. The goal of this effort is to encourage students to reorganize their approach to problem solving with the goal of reducing the commonly observed inadvertent error. Pedagogical content knowledge includes an understanding of what makes the learning of specific topics easy or difficult: the conceptions and preconceptions that students of different ages and backgrounds bring with them ... If those preconceptions are misconceptions, which they often are, teachers need knowledge of the strategies most likely to be fruitful in reorganizing the understanding of the learners. The purpose of this project then, is to develop a teaching strategy that will promote a reduction in the observance of these careless errors.

                                                                                       The Efficacy Of Simulator Exercises In The Delivery Of Bridge Resource Management Training

                                                                                                                                                      Robert M. Mercer

Abstract

Early Bridge Resource Management (BRM) or Bridge Team Management (BTM) courses were often delivered without the aid of simulators. These courses were generally effective in terms of providing participants with knowledge and attitudes that would enable them to integrate BRM concepts into their various operational situations however; there was little or no opportunity to develop the associated skills in the controlled environment of a simulator. The vast majority of BRM courses now include simulator exercises that are intended to provide realistic operational scenarios in which participants can apply good BRM practices. Navigational scenarios of varying complexity are often the basis of these simulator exercises for which participants develop a ‘passage plan’ prior to exercise execution where they work together as a ‘team’ to achieve the training objectives. It is difficult to design simulator exercises that will consistently elicit appropriate BRM related responses from participants to the extent required by the training objectives.

This paper looks at the efficacy of simulator exercises in terms of training objectives, suitability and relevancy to the overall objectives of BRM training in general. The paper will offer some suggestions for exercise development as well as some alternative methods of achieving BRM training objectives.

An Assessment of the Current Status of North American Industry's Commitment to Green Shipping

J. Tucker M.Eng., P.Eng.

 Abstract:  The Greenship movement has been underway for several years and is still gaining momentum in Scandinavia, Europe and Asia, yet it is just beginning to take hold in North America. There have been efforts to encourage industry participation in promoting greener shipping which have focused primarily on shipowners, seaways, terminals and ports. To this end, Green Marine Management Corp. was formed  to promote and facilitate environmentally friendly shipping activities and policies in Canada and the United States. The objective of this paper is to detail environmentally friendly shipping policies and programs instituted by the Green Marine participants in Canada and the United States.

This paper will first give a short history of Green-Marine, focusing first on its goals and objectives and then on the membership of the program and membership growth. Green Marine uses a series of performance indicators developed from the goals and objectives which are used by the participant companies in a self-evaluation.  These self-evaluations with respect to the Green Marine objectives are externally audited on a bi-annual basis. The results of these self-evaluations are compiled and analyzed and each of the 48 industry participants in North America are then interviewed. The interview questions are designed to discern the impact on the respective industries of following the Green Marine philosophy, and to discern what programs are currently in place or planned to be implemented to improve their self-evaluation scores. A synopsis of these interviews is presented.

Presence in Marine Simulator Training and Research

Robert M. Mercer and Scott N. MacKinnon

Abstract

Full mission ship simulators are typically employed to instruct mariners on standard operating procedures, bridge resource management, ship manoeuvring and/or familiarization with new technologies. However, human factors researchers are increasingly employing simulation technologies to better understand issues such as operator stress, mental workload, heuristic principles and situation awareness. Understanding the factors that influence how an operator makes and executes a decision will offer insight into situations where operator error may occur and provide guidance regarding system and bridge designs. This paper will explore the criteria for designing experimental methodologies appropriate for examining outcomes of operator performance during simulation exercises. Inherent in this approach is the need to design and implement realistic scenarios that engage the test subject. The concept of “presence” in virtual environments is explored in the context of preparing and controlling the simulation exercise used for data collection. Presence is defined as the subjective experience of being in one place or environment, even when one is physically situated in another.  The Situation Awareness Global Assessment Technique (SAGAT) can be employed to assess human performance and integrated navigation systems. SAGAT provides an objective explicit measure of situation awareness by directly comparing the operator’s SA to an operational “scenario”. Examples are drawn from human factors research completed at the Centre for Marine Simulation (CMS).

Towards STCW 2012: an analysis and discussion of selected topics

Fred Anstey, Catherine Dutton

Abstract

A diplomatic conference was convened in Manila in June of 2010 to adopt major revisions to the Standards of Training, Certification and Watchkeeping for Seafarers (STCW). The original 1978 Convention and Code were designed to establish basic requirements for training, certification and watchkeeping on an international level. In 1995 STCW was revised to give greater oversight, and to provide strict implementation obligations. The International Maritime Organization (IMO) has required the latest comprehensive review to reconcile inconsistencies and to address current global challenges. The call for submissions saw those with vested interests putting forward their views on needed changes. While all submissions had the interest of the global shipping community at heart, it is inherent, that with such diverse interests, the inevitable and resultant lobbying and negotiations would impact the final outcome. The revisions pertaining to marine security stipulate mandatory training for all levels of shipboard personnel as dictated by the International Ship and Port Facility Security (ISPS) Code. Mandatory training for the shore-based position of Company Security Officer (CSO) will continue to be ignored. Bridge and Engine-room Resource Management has been moved to the mandatory section of the Code but with an apparent reduction in requisite training elements. Advances in technology have created the need for and recognition by STCW of the electro-technical officer. With an overall increase in training as detailed by the revised STCW it will become increasingly difficult to monitor conformance, including adherence to the requirements for refresher and revalidation training. This paper will review these selected subject areas that have undergone changes in this latest iteration and due to enter into force in January 2012. A brief analysis of STCW training requirements for marine security, Bridge Resource Management (BRM), the electro-technical officer, and for refresher training and certificate revalidation has been undertaken and is presented as a catalyst for thought and discussion.

Keywords: Electro-technical, resource management, MET, refresher training, security, STCW.