1. We offer several initial licensure programs at both the baccalaureate and post-baccalaureate (alternate route) levels. Do we need to prepare separate program reports for each track?

The following procedures apply to programs in the same discipline (English, Elementary, etc.) that are at different levels (for example, undergraduate and post-baccalaureate) or different in other ways—but that have identical assessments. If the assessments are not identical then individual program reports must be submitted for each program.

In order to ensure that each program receives an individual decision, it is necessary for a shell (or template) to be created for each program in PRS. However, the following procedure will allow you to enter all of the information and text for these programs just one time. The system will then automatically copy the information into the other program report forms.

This is how it works:

1. First, you must notify NCATE which programs have identical assessments, rubrics and/or scoring guides. We will then ‘link’ these reports in PRS. For example, you notify NCATE that you have two Elementary programs: one is an undergraduate program and one is an MAT program, and they have identical assessments. This link must be made before any text is entered into the program report form.
   
2. NCATE will link these reports in PRS. You can tell that they are linked in PRS because the first boxes in the left column in PRS for those two programs will be shaded the same color.
   
   
3. The compiler of the report completes all the fields in the program report, adds attachments, etc. All the data charts should include data, disaggregated for each program being submitted. In the example above, that means that each data chart for each assessment would have two columns, one column for the data from the undergraduate program, and one column with data for the MAT program.
   
   
4. The compiler submits the report. The compiler sees a “thank you for submitting” note from NCATE when the report is submitted. In addition, the NCATE Coordinator at that campus also receives notification that the program report has been submitted.
   
   
5. After the first report is submitted, the compiler clicks on the program name in the second shell. The compiler is asked if they want to copy the information from the first report into the second shell. After the compiler clicks “ok,” PRS automatically copies all text, information and attachments into each report linked to the first one.
   
6. The compiler should go into the second report and change any necessary information (for example, in the example give above, the compiler would change the “Degree Level” from Undergraduate to Masters).
   
   
7. The compiler then submits the second report.
   
   
8. The above process can be repeated if there are three reports that have been linked. After the compiler submits the first report, they can then click on the program name in the third shell and gives the “ok,” all the information and attachments in the submitted program will be filled into all the program reports shells that are linked to it.

If you are not sure whether to prepare a single report or separate reports, call the Program Review staff at NCATE for advice.

2. For many of our programs, we’ve had fewer than ten completers over the past three year period. Are we required to submit program reports for those programs?

In Spring 2010 and Fall 2010, NCATE will defer review of low-enrollment programs, defined as programs with =5 completers in the last three years (in total). During this year, NCATE staff will work with states, institutions, and SPAs to develop a new strategy for review of these programs that will provide quality assurance but may not lead to SPA recognition. If you have further questions, please contact Margie Crutchfield (margie@ncate.org) or Robin Marion (robin@ncate.org).

3. We have several programs that are designed for candidates who enter the program with undergraduate training in their content area. Do we still have to submit SPA reports for these kinds of programs?

Many secondary post-baccalaureate programs have been developed for candidates who enter the program already having been prepared in the content area; typically candidates enter the program with an undergraduate major in the field. Many of these are MAT programs although in a few institutions these are called MEd programs. Most post-baccalaureate and alternate pathways programs are also designed to prepare candidates who come into the program with appropriate content area preparation. For the purpose of this discussion, these will all be called “Initial Licensure/Post Baccalaureate (IL/PB)” programs. NCATE recently approved a new process for approving these kinds of program, but only those in the secondary content areas: English, math, science, social studies and foreign language. This new process does not include programs in special education, elementary education or other areas. IL/PB programs will be able to combine programs into one secondary education program and submit one report. This report will address a generic set of standards but will include two discipline specific components: an evaluation of discipline-specific content knowledge components (through a transcript analysis) and an evaluation of discipline specific content pedagogy components (in the pedagogical assessments). For full information about this new process, see the document on the NCATE web site on the Program Review Resources page, or contact Margie Crutchfield (margie@ncate.org) or Robin Marion (robin@ncate.org).

4. We have a program to prepare middle level teachers. Each candidate receives a credential to teach in two content areas (e.g. middle level math and science). Do we have to submit programs reports to each of the content-specific SPAs and to the National Middle School Association?

Middle level programs that prepare candidates in two or more content areas and that meet the NMSA criteria for middle-level programs will submit program reports to NCATE/NMSA. NMSA reviewers will evaluate the submission and make a decision on whether or not the program will be nationally recognized by NMSA. In addition, NMSA reviewers will ensure that the 80% of completers pass the appropriate content test(s). National recognition of this program by NMSA will also be dependent upon the unit having nationally recognized programs in each of the appropriate content areas at the secondary level. NCATE staff will verify the status of the secondary content program areas. Middle level programs could be recognized with conditions by NMSA if secondary content area programs are still in process. For those few middle level programs that do not have secondary preparation programs, 80% of the candidates will still be required to pass the state test in the content area and the state must ensure the adequacy of the content preparation.

5. When are program reports due?

All program reports should be submitted at least one year prior to the institution’s scheduled on-site accreditation visit. Reports are due on September 15 for fall visits and March 15 for spring visits.

6. What is the process for submitting reports?

Program reports are submitted online at an NCATE website that is accessible with an institution ID and password. NCATE will prepare the submission site up to one year prior to the time you plan to submit. One to two months prior to the submission deadline, institutions submit to NCATE a list of programs to be submitted so that the URL, institution log in ID and password can be issued to them. If you would like a copy of the chart, please go to: http://www.ncate.org/institutions/ChartPRS09.asp?ch=90.

7. Who will review and make a decision on my program report?

Each program report is assigned to a team of two to three reviewers who have been trained in the standards by their professional associations and do not have a conflict of interest with the institution being reviewed. A lead reviewer will compile a team report, including a recognition decision representing the consensus of the team. The reports are reviewed by an audit committee of the professional association, which determines national recognition.

8. Upon what criteria will the recognition decision on our program be based?

Program reports are evaluated on how well assessments and data derived from assessments provide evidence that candidates meet the program standards of specialty professional associations (SPAs). National recognition decisions, therefore, are based on the success of program candidates as measured on credible assessments.

9. For programs with very small enrollments, how can a SPA make judgments based on data?

For very small programs, data derived from assessments are not, by themselves, a reliable indicator of program quality. However, if candidates (all two of them!) perform poorly on assessments, it will be important for the report preparer to reflect on why that poor performance occurred and how or whether the assessment should be changed as a consequence. No matter the size of the program, candidates are still expected to meet program standards. NCATE is currently exploring a new policy for programs with extremely low enrollments (5 or less completers over a 3 year period). For programs with extremely low enrollment, a deferment may be requested while NCATE staff develop a policy that applies. If you have further questions, please contact Margie Crutchfield (margie@ncate.org) or Robin Marion (robin@ncate.org).

10. What if our latest Title II data does not reflect an 80 percent pass rate in all program areas. What are our options?

NCATE policy requires a program to have an 80 percent pass rate on the state licensure exam in the content area in order to qualify for program recognition. The data must be derived from the most recent annual reporting period, as reflected by a state or testing agency report, or the institution’s own records (which would provide the opportunity to present a more current set of data). This requirement is waived for programs that (1) do not have a required state licensure test, (2) have not been in existence long enough to have produced an annual cohort of completers, and/or (3) have not produced a total of ten completers in the last three years.

A program report that does not reflect an 80 percent pass rate under Assessment #1 on licensure tests cannot receive or retain full national recognition; however, the program could be nationally recognized with conditions and would then be required to submit new test data within 18 months to show that the program is achieving an 80 percent pass rate.

11. How much assessment data are required in our program report?

For full recognition, programs will be required to submit data that represent two applications of the assessment. That is, the assessment must be given and data collected at least two times. If an assessment is in a class that is offered every semester, then the two applications could be satisfied in one academic year. If the assessment is in a class that is offered once per year, then the two applications would take two academic years. For revised and response to conditions reports, data from one application of the assessment would be required for full recognition.

NOTE: For units undergoing accreditation for the first time, programs are expected to be able to include in their program reports at least one year of data on all assessments in order to be eligible for full national recognition.

12. What can we include as attachments to the report?

Only the items specified in Sections I and IV of the report form may be attached to the report. Institutions will not be able to attach or submit any extraneous documentation with the report (e.g. handbooks, syllabi). Student work samples or artifacts cannot be submitted as part of the program review document. Documentation external to the institution (e.g. a report from the testing agency) will need to be scanned by the institution in order to be submitted as an attachment.

13. Can charts and graphs be uploaded into the program report template?

Charts and graphs cannot be input into the narrative fields or text boxes in the program report template. However, charts and graphs can be uploaded as attachments in two different places: within Section IV as part of your 6-8 key assessments or in Section I, in Question 4 where charts or graphs relate or enhance the responses Questions 1 or 2.

14. What happens if our narrative responses or our attachments exceed the stated page limits?

In PRS, character limits are given instead of page limits for each of the narrative parts of the report. Section I has a limit of 4000 characters per question, except for Section I, Question 2 which has an 8000 character limit. Section V has a 12000 character limit and Section VI has a 24000 character limit. PRS automatically limits the number of characters that can be entered for each narrative response in the report. For attachments, we ask that programs limit the length of each part of the submission to no more than 5 pages with each page approximately equivalent to one text page of single-spaced, 12-point type. This means that the entire file for all documents related to a specific assessment should not be more than 15 to 20 pages. Please note that no single attachment can be larger than 2 MB. There is also a limit of no more than 20 attachments per each program report.

If your narrative responses exceed the character limit, your response will be cut off once you reach “0.” This means that your response can be cut off in mid sentence or thought, so please be extremely cognizant of the length of your response. A good way to make sure that your response does not exceed the limit is to write it in a word document using the word count feature as a guide.

15. When you refer to the “6-8 required assessments,” how are you defining an assessment?

An assessment is an evaluated activity or requirement by which a program determines that specific outcomes or standards have been mastered by a candidate. However, for the purposes of the program report, one or more of the 6-8 assessment “slots” may actually be addressed with an amalgamation of assessments and data. The elementary education program report, for example, requires assessments in the areas of mathematics, English, science and social studies for its assessments on content and on lesson planning. (See question #14 below for more discussion of using more than one source of assessment data for one of the 6-8 assessments.)

16. Our institution has created assessments for performance-based reviews over the past five years, closely following guidelines from SPAs. But there are more than the 6-8 you now limit us to—more like 20 to 25. How should we report results for the pilot? Do we need to leave out important aspects of evidence?

Evidence that has been gathered about candidate knowledge and performance can be clustered into “composite” assessments and fitted into the five defined assessment areas, or the remaining three assessments. There is no reason to omit important pieces of evidence, but NCATE expects that over time the assessments that have been constructed in response to a performance-based system can be adapted to the new program report.

17. Can we use the same assessment for more than one of the 6-8 required assessments?

Yes, although you need to think carefully how you allot your assessments. For example, a comprehensive evaluation of a candidate portfolio or the student teaching semester may include in-depth assessments of both content knowledge and lesson planning. In this case, one section of the portfolio might be cited as Assessment #2 (content knowledge) and another section of the portfolio would be cited as Assessment #3 (ability to plan instruction).

18. What is the difference between the report of findings on each assessment required in Section IV of the report, and the use of assessments for improving the program we are asked to describe in Section V of the report?

For each assessment in Section IV, you must briefly summarize the results of the data from the assessment, as well as interpret the data findings about candidate knowledge and skills in relation to the standards addressed by the assessment. This provides reviewers with specific information about how each assessment addresses the specific SPA standards.

Section V requires an interpretation of what the overall results described in Section IV say about program quality and implications for improvement of the program and candidate performance. This use of data and the resulting changes or improvements can also be used for the third element of NCATE Unit Standard 2.

19. Our department of teacher education uses the same student teacher evaluation form for all secondary areas. Since the form is not aligned to the individual SPA standards, will the data be of any use?

For some of the required assessments in the program report, the data produced by the program’s assessments may have limited relevance to some or all of the SPA standards. For example, some state licensure tests are more closely aligned to the professional standards than others. For another, generic student teaching evaluations will not provide direct evidence of meeting specific SPA standards. In this case, program faculty could add a program-specific section to the generic assessment, they could develop discipline-specific rubrics, or they provide a very clear rationale and context information that demonstrate how a generic assessment, in the context in which it is implemented, does bear relevance to specific SPA standards.

20. What’s the best strategy for choosing other assessments to enhance our report (i.e. assessments 6-8 for most program reports)?

The strategy for choosing which additional assessments to submit should be based on several factors. For example, it could be that your content-based assessments are relatively weak, and your report might benefit from another assessment that demonstrates candidate mastery of content. Or it might be that your required assessments overall pay short shrift to one or more of the SPA standards. In that case, it might be prudent to submit an assessment and data that are more closely aligned to those standards. Another possibility is that your candidate data across assessments do not demonstrate the strengths of your program the way you would like, but you have another key assessment in which your candidates consistently perform well.

21. In the faculty information chart in Section I of the online template, should we include faculty from Arts and Sciences who teach the content courses that our candidates are required to take?

No; the only faculty listed here should be faculty responsible for professional coursework, clinical supervision and those who teach methods courses. Adjunct and part-time faculty should be listed only if they have taught or provided other services for the program during the most recent academic year. Please see instructions in AIMS about how to upload faculty information into the faculty information chart.

22. Within that same form - where we list three contributions for each faculty member in the areas of scholarship, leadership, and service - should we list a contribution in each of these three areas for each faculty member?

No; for example, a faculty member might choose to list three scholarly publications as her major contributions. However, if all faculty list scholarship as their contribution, the program report preparer may want to ask some faculty to highlight leadership or service experiences instead, in order to demonstrate that overall program faculty are contributing in all three areas.