In 2018, the McGraw Center and the Office for Information Technology undertook a comprehensive evaluation of Princeton’s Learning Management System (LMS). The review was initiated in part because Blackboard Learn had not been evaluated since it was adopted in 1999. Over the past decades, the LMS market landscape has changed considerably and LMS providers have greatly expanded and improved their products. In addition, the McGraw Center and OIT felt that our LMS was significantly underutilized and that good stewardship demanded that we thoughtfully and comprehensively survey all options for our next iteration of a campus-wide LMS.
After obtaining approval from the Strategic Advisory Group on IT (SAGIT) to begin the analysis, the LMS Evaluation Team created a thorough and inclusive process for selecting our new LMS solution.
The evaluation team’s efforts followed four phases:
Phase 1
Assembly of LMS Evaluation Team and Initial Market Review.
Spring-Fall 2018. An evaluation team with members from OIT, the University Library, the McGraw Center, and the Office of the Dean of the College was assembled. Based on a review of the LMS market landscape, the evaluation team identified three top vendors.
Phase 2
Information Gathering and Definition of Evaluation Criteria.
January-March 2019. The evaluation team interviewed over 120 individual faculty members, visited 30 Academic Departments and Programs, and met with Library Staff, Undergraduate Student Government, the Graduate Student Government, and the Academic Managers Group. The evaluation team developed a weighted list of 200 functional and technical requirements to evaluate the LMS products.
Phase 3
Identification of Top Two Vendors.
March-May 2019. The three LMSs under consideration were scored on the extent to which they fulfilled the 200 functional and technical requirements. The scoring was based on vendor feedback and the evaluation team’s review of each LMS. The evaluation team also gathered qualitative feedback on the products under consideration by inviting colleagues in the McGraw Center and OIT to perform a set of tasks on test courses in each platform. The User Experience Office in OIT reviewed the vendors’ VPATs (Voluntary Product Accessibility Template). This process resulted in the selection of the two top vendors.
Phase 4
Identification of Recommended Vendor
May-June 2019. Once the LMS solutions under consideration had been narrowed down to two, the evaluation team conducted an in-depth and broad product analysis. The review included:
- Technical Architecture and Security Review
- Onsite Vendor Demonstrations
- Faculty Evaluation Workshop
- Students and Staff Evaluations and Survey
- Accessibility Audits
- Cost Comparisons
- Peer Reference Checks
Technical Architecture and Security Review. The two remaining vendors were asked to provide detailed information about their product’s architecture and security designs for review by OIT’s Architecture & Security Review (ASR) committee. The goal of the review was to ensure that each product met Princeton’s technical standards in this area.
Onsite Vendor Demonstrations. Each vendor was invited to campus during the third week of May 2019, to give face-to-face product demonstrations to faculty, staff, and the evaluation team, and to answer questions about solution capabilities, usage, etc.
Faculty Evaluation Workshop. The McGraw Center hosted a half-day Faculty Evaluation Workshop on May 29, 2019. The workshop was designed to obtain faculty feedback on both of the products under review.
Students and Staff Evaluations and Survey. Staff and students were recruited to complete a set of tasks in each of the LMS systems and then provide feedback about their experience. Their review included whether navigation on the LMS platforms was intuitive and straightforward, and which product they preferred.
Accessibility Audits. OIT’s User Experience Office reviewed the two finalist products for compatibility with common accessibility tools and the use of accessibility design practices.
Cost Comparisons. The evaluation team gathered product, support and implementation costs from each vendor and performed a comparative analysis of the data.
Peer Reference Checks. Each of the vendors was asked to provide five higher-ed peer references for their products. The evaluation team met virtually with each reference school to obtain their feedback on the vendor, product, implementation approach, project resources, etc.
Conclusion
Canvas emerged from this process as the clear choice, both among faculty and students, and the evaluation team. All of our Ivy League peers currently use Canvas and the feedback we received from reference schools was very positive. On July 10, 2019, the Dean of the College approved our recommendation to adopt Canvas as our next campus-wide learning management system.
