Oral Roberts University

When a university’s most popular class is taught by its president, well, that’s a time for capacity testing.

Oral Roberts University (ORU) in Oklahoma is one of the most technologically sophisticated universities in the country. Still, its student experience is only as good as the EdTech software it relies on.

“The president said, ‘I need to know my class won’t have any issues,’” says Mike Mathews, VP of innovation and technology for ORU.

That’s not an easy promise to make when over 900 students will be logging in, accessing, and simultaneously submitting a pop quiz on a software platform that had recently moved to the cloud.

Mike needed a third-party that could give the new, cloud-based setup a stress test for capacity management. After seeing Deft speak at a Smart Campus Summit, Mike wanted Deft for the job:

“It just made sense — Deft wasn’t a cloud-based company trying to blow smoke.”

He tasked the Deft team with writing a performance test that would make sure thousands of students could take pop quizzes at the same time without fail.

“The No. 1 benefit is peace of mind. From endpoint to endpoint, we’re able to test for capacity planning.”

Mike Mathews
VP of innovation and technology at ORU

Setting up capacity testing to hold EdTech vendors to their promises

At ORU, Mike is leading the charge in higher education to use new means of data collection and predictive analytics to promote student success. Third-party capacity testing verifies that the vendors those innovations depend on are meeting their SLAs and terms of service — and troubleshooting what’s gone wrong when they aren’t.

“We live in that day where people expect flawless, uninterrupted service. Our students do, and so does our university president. Without flawless service, we never get ahead on innovation.”

Dr. Kathaleen Reid-Martinez
Chief academic officer at ORU

To build a proper capacity test, Deft first had to understand what the real-world circumstances would look like. With up to a thousand people in a class (all logging in and answering questions at the same time, some internationally), our team had to write a test capable of performing multiple actions from around the world.

The result was a capacity testing tool that launches from all over the world. It simulates everything a student would do, running the whole test-taking process and using step-functions to orchestrate data collection and aggregation. We store the raw data and present the aggregated metrics, so anyone can see how the software is performing both from the raw data and an easy to interpret dashboard.

Using performance testing best practices to trace back the flaws

It’s natural to want to design tests to be holistic — to include every element, end to end, in performance testing and capacity planning. There’s a problem with that strategy, though: If it fails, how do you isolate the cause?

To make Mike and his team at ORU 100% confident that they could handle a sudden crush of student quiz-takers, we had to break the test into logical pieces. That meant testing the network separately from the application.

“If something failed, no one could identify what it was. The WiFi, the server, the router, the firewall — all these things would start getting blamed. Deft’s implementation saved us from chasing story after story about IT gaps.”

Mike Mathews
VP of innovation and technology at ORU

The ORU team was able to test the network themselves, with some recommendations from us on the best gear and capacity planning strategies. We took on the application testing, removing any other variables so the data we reported back reflected only the software capacity testing.

The capacity performance testing results

We ran the tests, adjusted the tolerance to match real-world circumstances, and ran it again. The full report made its way from the ORU team to the EdTech vendor. There, the vendor was able to look at the results, identify a previously unknown error, and marshal the resources to resolve it.

Instead of having to find out through an outage that affected faculty and students, Mike and his team were able to work with the vendor to address the shortfalls in advance of the system’s deployment. By the time the software had to be put to the real capacity test of a full classroom, Mike and his team knew it was ready to go.

“If I could test like this for every one of our services, I would.”

Mike Mathews
VP of innovation and technology at ORU

Building a system for future capacity planning as a service

ORU came to us with an immediate need, but we didn’t want to throw out long-term thinking just because it was a short-term project. As we wrote the test, we architected the environment so it could be repurposed to test other services across the university. Effectively, we delivered a test and a framework for a future capacity tester at the same time.

Now, Mike and his team have the confidence that this particular piece of quizzing software will work as promised, and a path to make sure all of ORU’s vendors deliver as well. Because it’s built completely in the cloud, Mike can spin up a test, spin one down, run it quarterly or turn it off. There’s no maintenance cost — just on-demand proof of performance.

“With Deft, I’ve found a true practitioner and consultant. That gives me hope.”

Mike Mathews
VP of innovation and technology at ORU