When the internet goes down, a common piece of advice is to keep restarting one’s router until it finally works.
That frustratingly familiar scenario played out on a massive scale Thursday, when a problem in Cornell servers sparked a cascade of disturbances in an internet infrastructure that thousands of students, faculty and staff have come to closely rely upon to perform basic tasks. This forced Cornell’s IT department to conduct “a series of router restarts” in the hopes of bringing the network back online.
The outage — which arose from a yet to be determined “issue” in Cornell’s two major server farms, located in Rhodes Hall and the Computing and Communications Center — was first reported at 10:30 a.m., prompting Cornell’s IT department to quickly scramble for a solution.
The campus Wi-Fi, eduroam, Cornell Dining, Net-Print, Tableau and Student Essentials came to a screeching halt, disrupting everything from library loans to prelims. In an email to The Sun, one first-year veterinary student described a scene of drama as downed servers caused a major examination to be delayed amidst the confusion.
“An hour and 15 minutes into our six-hour final the servers went down,” Natalie Katz wrote. “They’re still not back up and we were allowed to go home for two hours and had to come back at 4. We’re all just sitting here, exhausted and frustrated, still not able to take our final.”
According to performance reports, CIT spent three and a half hours investigating the mishap, which was deemed a “severity one issue,” before calling the vendor that supplied the routers at 2:30 p.m. in order to “identify the root cause of the problem.”
In response, CIT temporarily shut down server farm routers from 3:15 to 3:30 p.m, which restored some affected services, but still left others inoperable.
A series of additional restarts were performed into the evening, until by 7:30 p.m. the same day, CIT reported that all services had finally been restored and were investigating the initial source of the outage.
The outage, which lasted for most of the day, caused a slew of key Cornell services to seize up in disarray. Alec Faber ’20, who works in Olin library, said he came in to work at the printer support desk only to be met with scores of frustrated students griping about their inability to leverage on-campus printing resources.
“There were several students around the printers who were very frustrated because they were not even getting the net print login pop-up,” Faber told The Sun.
According to Faber, the Cornell’s IT Services informed him that the campus-wide service outage had affected “virtually all of Cornell’s online services,” precluding them from even “accessing their own IT help database.”
After learning of this recent development, Faber said he “spent most of [his] shift telling people that ‘yes, all printing services are down,’ [and] ‘no, I did not know when it would be fixed, and no, there is nowhere else on campus you’ll be able to print.’”
“Naturally several students were quite upset about this as they had homework and papers due, and I wish there was something I could’ve done for them because I know what it’s like to be in that situation,” Faber added. “But I was at the mercy of whoever was fixing the problem.”
A University spokesperson said that Cornell had no further information to share on the incident.
Shivani Sanghani ’20 contributed reporting