Skip Navigation
E-portfolio Systems for Performance Assessment and Program Evaluation

Dr. Sarah McPherson, Ed.D.
Chair of the Instructional Technology graduate program at New York Institute of Technology
 

Ultimately it is about the manifestation of a new dimension of one's identity: the digital identity 
(Retrieved October 11, 2010 from http://www.eportfolio.eu).


Background and Theoretical Framework

A professional portfolio is a collection of artifacts that demonstrates a candidate’s ongoing effort and developmental growth in one or more areas (Paulson, Paulson, & Meyer, 1991).  It can also be used as a marketing tool for job interviews to display evidence of knowledge and skills (Kilbane & McNergney, 2001). The purpose of a portfolio system is to systematically organize evidence of meeting standards at three levels: curriculum of the program, faculty instruction and assessment, and candidates’ reflection on learning. Barrett & Carney, (2005) state that “a critical component of an educational portfolio is the learner's reflection on the individual pieces of work (often called "artifacts") as well as an overall reflection on the story that the portfolio tells”. Some colleges of education look to an electronic system for meeting NCATE Unit standard 2, although not required, as an assessment system to collect data about candidates' performance for purposes of program evaluation and improvement planning (Barrett &Wilkerson, 2004).

In that there are many types and purposes for portfolios, for this discussion it is important to distinguish the type of portfolio subject of this paper. According to Paulson and Paulson (1994) a portfolio for demonstrating learning and reflection is a constructivist-type portfolio which shows growth and development over time, in contrast to a positivist-type portfolio which contains candidate's ‘work’ considered representative of knowledge and understanding for assessing learning outcomes, using data to report learning outcomes across users, contexts, and purposes.  The e-portfolio described in this paper is more similar to a positivist approach as a performance reporting system to determine the candidates’ knowledge and skills listed in standards. 

An e-portfolio assessment system meets NCATE Unit standard 2 as a way to collect data about candidate performance for program evaluation and planning for improvement. For this reason, many institutions of higher education adopt electronic portfolios which allow candidates to systematically collect digital artifacts as evidence of meeting performance standards. In contrast to a traditional paper-based portfolio, the electronic portfolio allows candidates to collect and store evidence of learning in an electronic format.  E-portfolios also serve as an alternative to standardized tests or comprehensive exams for assessing candidates’ knowledge and skills. Barrett and Wilkerson (2004) describe paper-based portfolios as involvement in collecting, selecting, reflecting, projecting and celebrating learning processes. Furthermore, they describe the added dimensions of technology for e-portfolios as an ability to archive objects of learning, to link and think, to engage in storytelling, planning, and publishing.

Higher education institutions look to electronic portfolios as an assessment system that supports alignment of program goals and objectives with national and state standards. The e-portfolios is used to collect candidates' artifacts that demonstrate evidence of learning, a rationale for how their artifacts meets the standards and reflections on how the standards relate to their teaching and learning.The role of  faculty is to assess the artifact and to provide feedback  on the candidates rationale and reasoning, and reflections. At New York Institute of Technology (NYIT) faculty of the Master of Science in Instructional Technology (MSIT) program lead the adoption and implementation for the adopting and institutionalizing an assessment system, tool and process.  In fall 2005, a portfolio assessment sytem was established, a commercial web-based electronic portfolio called TaskStream™ adopted, and the implementation plan put into place which continues to support the assessment system today.  

The selection process included reviewing several products, and analyzing their functionalities, compatibility, ease of use, maintenance and support, and pricing structure. The NYIT committee reviewed features of several products to determine the potential to meet these criteria. Products were tested, technical support systems reviewed, and administrative functions considered. The criteria established include the following:

  • Ability for web-based access to program requirements for demonstrating mastery of AECT standards,  New York State Educational Technology Framework and the MSIT programs objectives.
  • Ability to add artifacts to the portfolio for assessment of candidates knowledge and skills.
  • Ability for candidates to archive and publish their individual web-based e-portfolio.
  • Ability for faculty to evaluate candidates’ artifacts by applying standards and rubrics.
  • Ability of the system to help faculty manage and report student learning outcomes.
  • Provision of user-friendly and accessible training and support for faculty and candidates.

Spring semester 2005 the faculty of the MSIT program launched the pilot implementation of TaskStream™ which  met the programmatic needs for evaluating student artifacts and tracking assessments. The TaskStream™ e-portfolio system has extensive features for lesson planning, building rubrics, designing standards-based assessments, as well as communication tools for faculty and candidates to support feedback and evaluation. The TaskStream™ system is sufficiently modular for initial implementation to use only a few features in meaningful applications while allowing faculty and candidates to learn the intricacies of the system at a manageable pace.

The process began by aligning program objectives, technology objectives, and field experience requirements with national AECT standards and New York State Education objectives for Educational Technology Specialist certification. The alignment process helped faculty understand the purpose and position of each course in the scope and sequence of the entire master’s program.  Keystone assignments functioned as the culminating assessment for candidates to generate artifacts to demonstrate the knowledge and understanding of key concepts learned in each course.  Sample keystone assignments include a personal educational philosophy, instructional unit and lesson plans, technology evaluation, multimedia module of instruction, school technology plan, and research projects.  TaskStream™ Rubric Wizard tool was easy to use to develop the assessment criteria application for determining the extent to which candidates' artifacts meet each standard across courses in the program.  See Figure 1 for Master AECT rubric.  The rubric for each standard occurred in each course aligned with the standard(s) designated as aligned with learning outcomes of the course.  

Context

The NYIT MSIT program is designed to develop expertise in uses of media and technology for teaching and learning. Candidates in the program are generally full-time teachers, seeking their first master’s degree, and professional certification in the area of their initial certification. Graduates of the program work as master teachers in P-12 classrooms, or as school or district instructional technology leaders, staff developers, and curriculum designers. Initial courses enhance teachers’ expertise with a wide range of technology media and applications to set the stage for understanding how instructional technology can fulfill its potential to enhance learning and meet the unique needs of diverse learners. Building on theories of learning, motivation, and curriculum design, candidates apply newly developed technology skills to core content curricular areas of math, science, social studies and language arts. The 36-credit program leads to eligibility for New York State certification as an Educational Technology Specialist.

The Instructional Technology program is offered in partnership with five NY State Teacher Centers in the lower Hudson Valley, on Long Island and the New York City United Federation of Teachers, as well as on the NYIT campuses on Long Island and Manhattan. There are 5 full-time faculty members, approximately 40 adjuncts and more that 400 part-time graduate candidates. The e-portfolio system pilot was undertaken to prepare for NCATE accreditation and national program approval by Association of Educational Communication Technology. A basic MSIT student e-portfolio may include  a resume and keystone assignments for courses, i.e. instructional design curriculum units and lesson plans, multimedia learning modules, web-based instructional materials, school-based technology plans, and research papers. Artifacts demonstrate exemplary technology skills, i.e. web pages, Macromedia Flash projects, Inspiration diagrams, classroom blogs and wikis, and other web 2.0 instructional applications.  Documentation and reflection papers for 50 hours field experience in selected courses are also archived in the system to meet New York certification requirements. 

Implementation

There is no single formula for planning to adopt and implement an electronic portfolio system that works with every institution or program. Many institutions have had paper-based portfolio systems, which are converted to an electronic format. Others, similar to NYIT, have used a less systematic process for collecting assessment data which is very difficult to organize and compile meaningful information for program evaluation. However assessment is critical for program evaluation; reflective practice is essential for determining student learning.

As we know from the change process (Fullan, 2001) it is important to keep in mind that no matter how much front-end planning you do, needs change, people change, and thus plans outlined often times needs to change. In fact, it is important to plan for change. The system should allow for flexibility in how things are set up, and make it easy to implement changes over time. The Folio Assessment System in TaskStream™ is comprised of a Folio Assessment Program and a Directed Response Folio (DRF). The DRF is easy for candidates to upload their work and MSIT faculty to access for evaluation.  After faculty evaluate the artifacts candidates submitted, the assessment data are collected through the report-management features of  TaskStream™. The report management process related to NCATE accreditation requires consideration of an array of very specific questions, such as:

  • What data do we need to collect and report?
  • Which standard sets should be included?
  • What artifacts do we need to collect to generate the assessment data needed?
  • How can rubrics be used in the evaluation process?
  • How can each course in a program be integrated into the system?
  • How will the faculty integrate the program standards and learning outcomes into their courses?
  • How will candidates submit artifacts to demonstrate their knowledge and skills?
  • What training do faculty need for facilitating candidates developing appropriate artifacts, and for evaluating candidates work?
  • How well will the data reports meet the NCATE accreditation reports?

The primary aspect of the pilot implementation phase was intellectual in nature; the purpose was to gain consensus among program faculty to determine how to assess program goals. Typically, at NYIT the faculty makes decisions on the assessment information to report for each program. The process and lessons learned were shared with teacher education and school counseling faculty for expanding the application to other programs in the School of Education.  Once the pilot proved successful and was used for NCATE reporting, the e-portfolio system became 'institutionalized'. However the process requires continually updating as TaskStream™ is updated, monitoring to ensure that faculty are complying with the requirements for candidates to submit artifacts and with their responsibility to evaluate candidate's submission.  The fidelity of reports is reliant on consistent implementation.  Reports generated by the system allow faculty to look at individual and group mastery of program objectives as well as state and national standards. However it can be perceived as an additional requirement not integral to the actual teaching of the course.  As a result, in some cases, faculty other than the course instructor, are called upon to evaluate the candidate's artifacts. This is possible, although not preferred, with TaskStream™ because it is a database of rubrics standard for each section of all courses in the program. The data was relatively easy to collect and evaluate for the recent report for the AECT accreditation with all the data stored in TaskStream™, although for some sections, candidates' artifacts evaluations were incomplete.   

Training and Support

The adoption of new technology by faculty is never an easy task. In addition to the usual problems of implementation, the MSIT program is offered on multiple campuses and courses are taught by no less than 40 individual full-time and adjunct faculty. These factors create a major training challenge. The faculty member assigned the training responsibilities applied a model-based formula Marcus (1995) which states that Adoption = Perceived Value, Resources, Communication among adopters. As with most new technologies being introduced, the initial question of the faculty is "how it will help me?". The initial overview of the e-portfolio system demonstrated how it would assist faculty in the collection, organization and evaluation of the graduate candidates’ work. 

The design of the TaskStream™ system facilitates access to technology resources. It is a total web-based system allowing faculty and candidates to access it from anywhere an Internet connection available. Documentation is readily available online and in printable format. The vendor provides detailed online quick-start training guides on using the portfolio and webcasts for changes and updates. In addition, NYIT faculty developed training guides in both printed and electronic formats, specifically for candidates on how to upload digital files to the DRF, and for faculty related to how to evaluate candidates' work.  

Face-to-face hands-on training facilitated communication among the adopters allowing faculty to assist each other and share ideas for implementation. Initially training sessions were held as a small focus group of full-time faculty members facilitated by the NYIT training coordinator. Next, the system was introduced to the remaining full-time faculty at a regular faculty meeting using compressed video connection between the three NYIT campuses. Adjunct faculty members close to campus were invited to attend. In addition, several sessions for adjunct faculty were held in-person at a number of sites using video-conferencing equipment at off-site locations.  Training for new faculty continues through live webinars on Elluminate™ periodically scheduled each semester.  Also a Faculty Forum is setup in BlackBoard as a repository of all training materials, videos, print materials, and archived webinars that faculty can access at anytime.  

The TaskStream™ system allows faculty to continue their conversations through the discussion board feature. A general discussion board is set up for all program faculty with a separate forum for faculty for all sections of each course. These discussion boards facilitate full-time and adjunct faculty collaboration with increased opportunity to share ideas and ask questions. A faculty member monitors the discussions boards to answer any questions posted and to provide information about updates and upcoming training events. An added bonus and opportunity for collaboration is that all faculty participants have access to online discussions with faculty from all other colleges and universities using this e-portfolio system.

Evaluation

The NYIT committee collected data from a variety of sources and analyzed the result in the end of Spring 2005 semester. The Spring 2005 semester involved approximately 250 MSIT candidates use of the e-portfolio system in all sections of the 12 courses in the MSIT program. The candidates uploaded performance assessment artifacts deemed to demonstrate knowledge and skills that meet the designated program standards. At the end of Spring 2005 semester surveys were distributed to faculty to investigate their attitudes toward using the e-portfolio. Also faculty participated in focus groups to probe the issues, concerns, and suggestions that arose during the pilot implementation. Several candidates’ e-portfolios were analyzed to measure how their performances align with AECT standards and program goals. Individual candidates were interviewed to investigate their attitudes, benefits and motivation for using e-portfolios, and ideas for future uses of their e-portfolio.

Faculty focus group

The e-portfolio project helps faculty reflect on alignment between the standards and their implementation in the learning environment. The e-portfolio system allows collection of exemplars and a mechanism for candidates to monitor their learning progress through the entire master’s program. Faculty feel that the e-portfolio system provides a framework which serves as an anchor for their teaching. It increases opportunity for communication between candidates, helps adjunct faculty at off-site teaching centers to be more aware of the program objectives. It collects data more thoroughly, and generates data reports more efficiently.

The challenge of using a standardized assessment program is obtaining faculty consensus for the appropriate keystone assignment, criteria and assessment rubrics. To meet the program goals, assessment criteria are very high level so that faculty cannot rely on the rubric to assess the precise discreet learning objectives of individual assignments.  However, by having criteria for each standard set, gross evaluation reflected in a single numeric score is easily attained.  Faculty, including adjunct faculty, for other programs in NYIT School of Education provided input into developing keystones and rubrics course by course.  The MSIT program began the e-portfolio process with keystones and rubrics already developed which facilitated a quicker adoption rate.  

Faculty survey

The results of the initial faculty survey concur with the faculty interviews. Most faculty members believe that using TaskStream™ could help them and the candidates to become more aware of the program objectives and AECT standards (82% agree or strongly agree, N=10), collect assignments in various formats (73% agree or strongly agree), facilitate submission of assignments (92% agree or strongly agree), encourage candidates to be more self-regulated and organized (43% agree or strongly agree), acquaint candidates with curriculum standards (73% agree or strongly agree) and facilitate communication (73% agree or strongly agree). They also identified (64% agree or strongly agree) that the training materials provided are helpful. 

Candidates’ perspectives

From the candidates’ perspectives, the keystone assignments in the e-portfolio system helps them demonstrate their professional and technology skills. The upload to the web-based system and the associated web-based tools for a personal webpage, lesson plan, rubrics, standards sets are all helpful in designing and developing the artifacts. Candidates can decide whether they intend to display their artifacts publicly. The list the keystone assignments for each course in their program helps candidates monitor their own progress of learning and understand the scope of what they will learn throughout the program. Faculty frequently encourage candidates to review their artifacts and reflect on the evidence of their knowledge and skills, and evaluate their growth throughout the program. If candidates are comfortable with their work and see the value of the e-portfolio, they will find various functions and applications for its use. For instance, the e-portfolio can be useful as part of a resume in finding a desirable teaching position or when applying for a transfer, new position or tenure.

Discussion

Implementation, like design, varies from institution to institution. Training  and support are key for participating faculty to using the e-portfolio system with candidates, as well as providing support for all users. Often candidates figure out how to use the system fairly quickly and easily, but faculty adoption sometimes requires more human support. Faculty buy-in and ownership are crucial to the successful adoption of the e-portfolio process. Administrative leadership is also vital to support the adoption and ensure that faculty and candidates have what they need for implementation. 

Having used TaskStream™ for an e-portfolio assessment system for the past five years through two NCATE cycles has demonstrated the value for program evaluation.  The accountability to provide evidence of student learning that meets standards is easily accomplished through consistent and continual implementation of the process in every course.  However, it is not an automatic process. Ongoing training and support is needed to keep faculty up-to-date and bring new faculty on board.  Continual monitoring of implementation is also required for maintaing a reliable and consistent systematic process.  

A down-side of an institutionalized assessment process is the difficulty in making changes. Keystone assignments may be written in fairly general, global terminology to be resilient to time, but, with technology, there is always the compelling urge to be current and have an agility for change.  The assessment process can hinder staying current and agile in some aspects for advancement.  Faculty training and mindset may exacerbate the inertia.   However if assessment and evaluation are viewed as an ongoing change process for continuous improvement, basic tenets of change continue to apply (Fullan, 2001). That is, acknowledge past experiences and assimilate them into system refinements or redesign; provide adequate professional development and support; plan for incremental changes; maintain open lines of communication; provide reinforcement and feedback loops; and celebrate successes.  

References

Barrett, H.  & Carney (2005).  Conflicting Paradigms and Competing Purposes in Electronic Portfolio Development.  Submitted to Educational Assessment. Retrieved October 11, 2010, from http://electronicportfolios.com/portfolios/LEAJournal-BarrettCarney.pdf.

Barrett, H. & Wilkerson, J. (2004). Conflicting paradigms in electronic portfolio approaches. Retrieved April 24, 2005 from http://electronicportfolios.org/systems/paradigms.html.

Eportfolio EU. Eportfolio components. Retrieved October 11, 2010 from http://www.eportfolio.eu/eportfolio/components.

Fullan, M. (2001). The new meaning of educational change. 3rd ed., New York: Teachers College Press.

Kilbane, C., & McNergeny, R. (2001). Focus on technology: Digital teaching portfolios build, display talent. Retrieved April 05, 2005 from http://www.aacte.org/Programs/Research/focus_tech_digital.htm

Marcus, J. (1995). In Gilbert, S. An online experience. Change, 27(2), 28-45.

Paulson, F. & Paulson, P. (1994).  Assessing portfolios using the constructivist paradigm. Paper presented at AERA, New Orleans, ED376209. 

Paulson, F., Paulson. P., & Meyer, C. (1991). What makes a portfolio? Educational Leadership, 48(5), p.60-63.


Figure 1.  AECT Master Rubric

Unacceptable 

Acceptable 

Target 

Score/ Level

AECT 1: DESIGN -Candidates demonstrate the knowledge, skills, and dispositions to design conditions for learning by applying principles of instructional systems design, message design, instructional strategies, and learner characteristics. 

The candidate is unable to design effective conditions for learning. They lack the knowledge in the principles, theories and research associated with instructional systems design, message design, instructional strategies, and learner characteristics. 

The candidate's knowledge, skills, and dispositions to design conditions for learning are well-grounded in principles, theories, and research associated with instructional systems design, message design, instructional strategies, and learner characteristics. 

The candidate's knowledge, skills, and dispositions to design conditions for learning are fully grounded in principles, theories and research associated with instructional systems design, message design, instructional strategies, and learner characteristics. 

 

AECT 2: DEVELOPMENT - Candidates demonstrate the knowledge, skills, and dispositions to develop instructional materials and experiences using print, audiovisual, computer-based, and integrated technologies. 

The candidate demonstrate simplistic knowledge, skills, and dispositions in the development of instructional materials and experiences. Their products do not reflect basic principles, theories, and research related to print, audiovisual, computer-based, and integrated technologies. 

The candidate demonstrates essential knowledge, skills, and dispositions in the development of instructional materials and experiences by applying basic principles, theories, and research related to print, audiovisual, computer-based, and integrated technologies. 

The candidate demonstrates complex, integrated knowledge, skills, and dispositions in the development of instructional materials and experiences by applying principles, theories, and research related to print, audiovisual, computer-based, and integrated technologies. 

 

AECT 3: UTILIZATION - Candidates demonstrate the knowledge, skills, and dispositions to use processes and resources for learning by applying principles and theories of media utilization, diffusion, implementation, and policy-making. 

The candidate lacks basic knowledge of principles, theories, and research related to media utilization, diffusion, implementation, and policy-making. 

The candidate shows evidence that he/she can use processes and resources for learning that are grounded in principles, theories, and research related to media utilization, diffusion, implementation, and policy-making. 

The candidate routinely uses processes and resources for learning that are grounded in principles, theories and research related to media utilization, diffusion, implementation, and policy-making. 

 

AECT 4: MANAGEMENT - Candidates demonstrate knowledge, skills, and dispositions to plan, organize, coordinate, and supervise instructional technology by applying principles of project, resource, delivery system, and information management. 

The candidate is unable to plan, organize, coordinate, and supervise instructional technology. They lack essential knowledge of principles, theories, and research related to project, resource, delivery system, and information management. 

The candidate demonstrates baseline ability to plan, organize, coordinate, and supervise instructional technology through application of principles, theories and research to project, resource, delivery system, and information management. 

The candidate demonstrates extensive abilities to plan, organize, coordinate, and supervise instructional technology through application of principles, theories, and research to project, resource, delivery system, and information management. 

 

AECT 5: EVALUATION Candidates demonstrate knowledge, skills, and dispositions to evaluate the adequacy of instruction and learning by applying principles of problem analysis, criterion-referenced measurement, formative and summative evaluation, and long-range planning. 

The candidate is unable to effectively evaluate the adequacy of instruction and learning. They lack the basic knowledge of principles, theories, and research related to problem analysis, criterion-referenced measurement, formative and summative evaluation, and long-range planning. 

The candidate demonstrates essential knowledge and skill in the evaluation of instruction and learning by applying principles, theories and research related to problem analysis, criterion-referenced measurement, formative and summative evaluation, and long-range planning. 

The candidate demonstrates extensive knowledge and skill in the evaluation of instruction and learning by applying principles, theories, and research related to problem analysis, criterion-referenced measurement, formative and summative evaluation, and long-range planning. 

 

Search New Horizons

 

New Horizons Links

New Horizons home

About Us (NHFL)

Current Journal

Submission Guidelines

Subscribe

Facebook Icon Twitter Icon

New Horizons Shop

Featured Item: Research-Based Strategies to Ignite Student Learning: Insights from a Neurologist and Classroom Teacher

By Judy Willis | Purchase

Visit the New Horizons store on Amazon.com for more selections

New Horizons store on Amazon.com