Running Head: Needs Assessment 1

Needs Analysis of The Instructors of their knowledge of objectives

Brian Fitch

Western Governors University

Background
The Air Force Institute of Technology (AFIT) provides graduate education for the Department of Defense and NATO forces in the science and engineering disciplines. The school is a part of the Air University and the Air University is a part of the Air Education and Training Command. The Institute is divided into three smaller schools, the Civil Engineering School, The Systems Engineering School and the School of Systems and Logistics. The School of Systems and Logistics provides Professional Continuing Education to several career fields including science, engineering, program management, acquisition, and logistics. This analysis concentrates on the School of Systems and Logistics. The school further divided into three academic departments Logistics, Acquisitions, and Systems. The school has more than 100 instructors that are a mixture of military and civilian. The military instructors are typically assigned to the school for three years. Within the tree years they usually experience a one year deployment to support the war effort. The civilian instructors are employed with the school an average of ten years. Most of the civilian instructors are retired military and considered the best Subject Matter Experts (SME) available. When the school hires new instructors the potential new hire is required to complete a teaching interview. This interview involves the potential new hire teaching for one hour. Many of the potential new hires are very comfortable briefing to a group of people. Most of the teaching interviews are simply briefings. The purpose of the interview is to determine if the potential new hire is capable of teaching. After they are hired the faculty development department is required to train these new hires to be effective Air Force instructors.

Summary of Instructional Problem
New instructors at an Air Force graduate school hires Subject Matter Experts (SME) to teach professional continuing education. The SMEs do not have a background in education. All of the new SME have at least a Masters Degree in their field of expertise. Unfortunately many times these experts have no background in teaching or education. The lack of an educational background causes problems when the SME revise their courses material. The SME typically remove objectives they do not understand, change assessments and exercises resulting in courses that are not affective.

Current State: The SME have had no experience with the Air Force Instructional Systems Development (AF ISD) Process. They do not know understand why our courses have objectives and why their students need to be evaluated. The lack of understanding of objectives has resulted in the SME removing objectives from their courses and not being able to write appropriate objectives. The SME also do not understand the relationship between the objectives and the measurement devices used to assess the students.

Desired State: The SME will understand that there is a systematic process to develop courses that included the development of objectives. They will understand the purpose of the objectives and how to create objectives that are meaningful and correct. They also should be able to use the objectives to create effective course assessments.

Data Collection Processes
Three data collection instruments were used for the analysis. First, I used a questionnaire. I was able to easily tack completion of the questionnaires to ensure 100% return. A questionnaire was sent to both Deans of the school, the Department Heads for the three academic departments, and instructors of all three academic departments. The questionnaire has three types of questions, multiple choice, likert style and open ended questions. Second I used the questionnaire and performed face-to-face surveys with five of the instructors that were available but could not complete the questionnaire. Third I performed a review of the historical course control documents for 15 courses.

The questionnaires were sent to the School Deans and to the Department Heads to determine what impact the issue was having to the school as a whole.

A total of 20 questionnaires were sent. The questionnaires were sent to instructors of all three academic departments. I sent ten questionnaires to experienced instructors that have been teaching for more than four years each. Ten questionnaires were also sent to instructors with less than one year experience. The experienced instructors have had exposure to course objectives and know the current practices regarding objectives and assessing students. Less experienced instructors would not have the background instruction about objectives and assessments. The questionnaire responses from the experienced and non-experienced instructor should provide information regarding what is currently being done with objectives and what the experienced and non-experienced instructors know regarding objectives.

The questionnaire was used as a tool to perform five surveys. The surveys were necessary because the instructor’s availability and teaching schedule. All five of the instructors provided additional insight to the questions that I did not receive from the questionnaires. I also asked follow-on questions to clarify the instructor comments.

A review of the historical course control documents was also performed. The documents were specifically surveyed to discover how many of the courses had objectives or assessments that had been changed, removed since the original.

Data Analysis Results
100% of the questionnaires were completed and returned. Data analysis of the Lykert comments concluded that the instructors agree that course objectives are important. The results indicate that the instructors believe they know something about objectives and that many believe there is a connection between the objectives and the tests. From the surveys I conducted using the questionnaires I was able to determine that the more experienced instructors were taught during in-service briefings that the objectives and assessments were linked, however they were never provided any instruction on how they were connected. Some of the instructors have changed their objectives and tests without knowledge of

Instructors were provided multiple choice questions to determine what they knew regarding objectives and tests. I asked two questions regarding cognitive and affective domain objectives. If the instructor knew the difference between the two then I assume they know a little about objectives. Most instructors knew what type of objectives they had, the five that responded that they did not know were junior instructors. The junior instructors did not know what the domains were and responded with “Don’t know” (DK). Most instructors indicated that they did evaluate each student and they believed that in their course (somewhere) it is clearly indicated what the student must do to pass the course.
Table 1|||||
|||||
Knowledge of Objectives| | | | |
Multiple Choice Questions|Performance|LoL|LO|DK|
Which of the below are most typical in your course?|2|10|3|5|
|Yes |No|DK||
Do you know the difference between cognitive and affective objectives?|9|8|3||
Does your course have cognitive and affective domain objectives?|11|7|2||
Do you measure each student?|16|4|||
Does your course clearly indicate what a student must do to pass the course?|18|2|||
Before coming to AFIT/LS have you had any training on course objectives?|3|17|||
Do you know what the course objectives are in your course?|15|5|||
Have you ever changed the objectives in you course?|17|3|||

The instructors were provided likert scale comments to express their attitude toward objectives and tests. Most instructors agree that the course objectives tell the students what they must do to pass the course, however they do not agree that objectives and assessments are valuable to determine student mastery. I tried to clarify the comment regarding approval authority of the objectives and found during the surveys that many of the instructors believe they are the final authority for objectives and can change them.

Table 2|||||
|||||
Knowledge of Objectives| | | | |
Likert Scale Comments|Strongly Agree|Agree|Disagree|Strongly Disagree|
The course objectives tell the students what they must do to pass the course|3|15|2||
The assessment and the objectives are written at the same cognitive level.|5|10|5||
I know all I need to know regarding objectives.|1||18|1|
Objectives and measurement devises are valuable tools to determine student mastery of the objectives.|2|4|13|1|
Course sponsors are the approval authority for all course objectives.|5|3|12||
Formal affective domain objectives are necessary.||3|10|7|
It is important that the students understand the course objectives.|1|15|3|1|

The instructors were provided four open ended questions regarding their knowledge of the Air Force ISD (AF ISD) process and what they thought regarding objectives and tests. Only two of the junior instructors had knowledge of the AF ISD process. All of the senior instructors knew of the process. Many of them had not had any formal training but had learned of the process through in-service training. An overwhelming majority of the instructors indicated that the course and lesson objectives are in the syllabus and that they do not mention them or discuss them with the students. The instructors believe they would have benefited by attending one of the Air Force Basic Instructor courses before being allowed to teach their first class.

Table 3|
Knowledge of Objectives|
Open Ended Questions|
|
1. What do you currently know about the AF Instructional Systems Development (ISD) Process?|
The junior instructors know that there is a process for developing courses. The experienced instructors knew of the AFISD process but know little about it.|
2. Do you believe additional training on objectives and assessment will help you as an instructor?|
Overwhelmingly all of the respondents believed they would benefit from additional training on objectives and assessments|
3. What really happens with the objectives while you are teaching?|
The questionnaires and surveys indicated the objectives are provided to the students in the course syllabus at the beginning of the course and not mentioned. |
4. In your position as an AF instructor, what training do you wish you had before you began to teach?|
Most believe that they would have benefited in attending a basic instructors course|

The school leadership including the two Deans, and the three department heads were asked questions regarding their opinions of objectives and how they affect the school as a whole. They leadership believes that instructors should have a knowledge of objectives and what they are used for in their course. Leadership also believe it is important for all instructors to have information on objectives and the AF ISD process as a whole.

Knowledge of Objectives|
Leadership Questions|
1. What do you want instructors to know about course and lesson objectives?|
Leadership wants the instructors to know the basic information regarding objectives. The school has a curriculum development office that is expected to be experts in curriculum design|
2. Do you believe the instructors need to have additional information regarding course objectives or assessments?|
Instructors need to know what the purpose of objectives and how they are to be used in the classroom |
3. Who should receive information regarding the AF/ISD Process?|
All instructors need information about the AF ISD process|

A review of the historical course control documents was conducted. From the more than eighty course I reviewed twenty random course files. The review indicated that the objectives in only six of the course reviewed had formal approved changes to the objectives. This indicates that since the instructors indicated they typically change their course objectives do not change them formally.

Table 4| | |
Historical Course Control Document Review|
Course |Original Objectives |Changed Objectives|
Fam 103|10|6|
FIN 103|3|1|
IPM 301|5|1|
LAB 202|3|0|
LOG 040|8|0|
LOG 041|6|0|
LOG 042|1|0|
LOG 043|1|1|
LOG 044|1|0|
LOG 045|1|0|
LOG 046|1|0|
LOG 047|1|0|
LOG 048|1|0|
LOG 049|1|0|
QMT 110|2|0|
MRC101|5|3|
REQ 111|7|2|
LOG 499|6|0|
LOG 262|2|0|
SYS 109|7|0|

Need for Instruction
Based on the data collected, there is sufficient evidence that there is a need to improve and expand upon the knowledge instructors at this school have of course and lesson objectives. (Air Force Instruction 36-2235 vol 10) states “Some of the purposes of an objective are to: Serve as the foundation for instructional design.” If our instructors do not have a good foundational knowledge of objectives then they are doomed to fail as an instructor and the student will suffer.

Instructional Goal Statement
Using Air Force Instruction 36-2235 Volume 10 (Information for Designers of AF Instruction- Application to Education) and Air Force Manual 36-2234 (Air Force Instructional System Development) students will understand the function and use of objectives in their courses. Students will also understand the connection between their objectives and the assessments they use to measure students.

References
Air Force Handbook 36-2236 Volume 10 Information for Designers of Instructional Systems- Application to Education 1 November 2002, 53

Appendix A

UNOFFICIAL

MASTER Questionnaire for Brian Fitch school project

Department:________________________________________

Instructors (MC) 1. Which of the below are most typical in your course? a. Performance objectives b. Level of Learning objectives c. Learning Outcomes d. Don't know

2. Do you know the difference between cognitive and affective objectives? a. Yes b. No c. Don't know

3. Does your course have cognitive and affective domain objectives? a. Yes b. No c. Don’t know

4. Do you measure each student? a. Yes b. No c. Don't know

5. Does your course clearly indicate what a student must do to pass the course? a. Yes b. No c. Don't know

6. Before coming to AFIT/LS have you had any training on course objectives? a. Yes b. No c. Don’t know

7. Do you know what the course objectives are in your course? a. Yes b. No c. Don’t know

8. Have you ever changed the objectives in you course? a. Yes b. No c. Don’t know

Instructors (Likert)
1. The course objectives tell the students what they must do to pass the course Strongly Agree____Agree____Disagree____Strongly Disagree____

2. The assessment and the objectives are written at the same cognitive level. Strongly Agree____Agree____Disagree____Strongly Disagree____

3. I know all I need to know regarding objectives. Strongly Agree____Agree____Neutral____Disagree____Strongly Disagree____

4. Objectives and measurement devises are valuable tools to determine student mastery of the objectives. Strongly Agree____Agree____Disagree____Strongly Disagree____

5. Course sponsors are the approval authority for all course objectives. Strongly Agree____Agree____Disagree____Strongly Disagree____

6. Formal affective domain objectives are necessary. Strongly Agree____Agree____Disagree____Strongly Disagree____

7 It is important that the students understand the course objectives. Strongly Agree____Agree____Disagree____Strongly Disagree____

Instructors (Open) 1. What do you currently know about the AF Instructional Systems Development (ISD) Process?

2. Do you believe additional training on objectives and assessment will help you as an instructor?

3. What really happens with the objectives while you are teaching?

4. In your position as an AF instructor, what training do you wish you had before you began to teach?

Department Heads 1. What do you want instructors to know about course and lesson objectives? 2. Do you believe the instructors need to have additional information regarding course objectives or assessments? 3. Who should receive information regarding the AF/ISD Process?