When two Cornell network administrators began a routine investigation into why a University website had rebooted, they had no idea they would be handing their passwords over to a hacking group sponsored by a foreign government possibly seeking “revenge” against the United States.
But the hackers had rebooted the website as a trap, and as the administrators entered their passwords to investigate the reboot in May of 2014, hackers recorded their keystrokes with malware surreptitiously installed on the website.
With the administrators’ credentials, state-sponsored hackers remained undetected for months in the School of Industrial and Labor Relations, accessing scores of administrator passwords, compromising dozens of computers, peeking into at least one staff calendar and leaving backdoors to maintain their presence in what was the largest ever state-sponsored cyber attack on the University, according to senior information technology administrators at Cornell.
Nicholas Bogel-Burroughs / Sun Staff Writer
State-sponsored hackers breached the security of Cornell’s Industrial and Labor Relations School in 2014, administrators have confirmed.
The hackers, who IT administrators believe were bankrolled by a foreign government, used custom malware to avoid detection and extended their reach throughout the college in what is known as an advanced persistent threat.
“Like a squirrel putting away a bunch of nuts for the winter, they were saving this to do something with it,” Rob Bandler, deputy director of IT security at Cornell, said of the hackers. “They want to camp out, they want to watch, they want a jumping off point where they can try to do things from inside the perimeter, which is generally easier.”
Hackers used SQL injection, a method where instructions are inserted into a website’s data field, to maliciously gain control of at least one website maintained by the Cornell Yang-Tan Institute — then the Employment Disability Institute — and reboot the server, drawing in network administrators.
After recording the administrators’ keystrokes, hackers used those credentials to access an application IT staffers use to store passwords, according to a senior IT official who, like others interviewed for this article, spoke on the condition of anonymity because he was not permitted by the University to discuss details of the hack.
IT staff use the application, Password Manager Pro, to store not personal passwords, but administrative passwords that allow them to access servers, workstations and digital signage like that shown in the ILR conference center.
Using passwords stored in the application, the hackers broke into more IT assets in the ILR school and monitored an IT administrator’s work calendar to determine when he was out of the office, so they could expand the breach without being discovered.
“Hackers used the IT admin’s credentials to look at his calendar, see when he has a dentist appointment or is out on vacation,” the senior IT official said. “By accessing the IT administrator’s calendar, they could identify times to continue their progress attacking things within the college when they felt that, because of reduced staff availability, IT would be less attentive and less likely to discover the activities of the attackers.”
Unbeknownst to Cornell, the hackers poked around the ILR school for about four months until late one afternoon in October 2014, when eight Internet Protocol addresses — numbers identifying hardware connected to a network — in the ILR school began acting strangely, tipping off IT security that something was wrong.
After an initial investigation determined that the hackers had infested a sizable portion of the ILR school’s network, IT staff contacted a third-party firm to contain the attack, expel the hackers and analyze the intrusion.
Forensics from the data-breach response company showed that although the hack was serious, it was limited to the ILR school; valuable targets like credit-card processing software and a directory service that stores important information about users were not penetrated. The company also concluded that no data was extracted by the hackers, according to Bandler and two other IT administrators.
By analyzing the hackers’ custom malware, looking at the language and syntax of their code and sharing information with other universities and victims of similar hacks, Cornell was able to determine which foreign government was behind the hack, Bandler said, although he and others declined to identify the country.
“We understand the goals of the hack and I’m not going to discuss them, but the nature of them was more of a revenge against the United States in particular, not against ILR or Cornell,” Bandler said.
The same group of hackers “also found web vulnerabilities in other places around the United States that we found out later,” he continued. “That’s what they’re paid to do — go find places where [they] can leverage something [their] government wants.”
Because of the formidable threat state-sponsored hacks pose, Bandler said Cornell and other universities spend a tremendous amount of effort guarding against and searching for that kind of breach.
Bandler explained that hacks sponsored by foreign governments are almost constantly being levied against Cornell and other research universities, and they are often carried out by better hackers backed with more money than financially-motivated hacks or “hacktivism.”
Eva Vincze, director of the Security Management and Digital Forensics programs at George Washington University, said hackers regularly attack educational institutions to sharpen their skills before moving on to more critical targets like government networks.
“Universities are great training grounds for young hackers, especially groups working for nation-states who are learning how to do this,” Vincze said. “We’re a teaching tool, unfortunately, in a bad way. But the other side of that is we have gotten so much better at protecting university systems because, unfortunately, a lot of time we’re playing defense.”
The attack on Cornell was never reported or disclosed by Cornell, and students, faculty members and non-IT staff are largely unaware of the breach that IT staff spent months tirelessly working to contain and expel.
“This wasn’t kept from the public, but it’s not widely known,” Bandler said of the 2014 hack.
When Cornell University Media Relations discovered The Sun was investigating, they asked the paper to not publish any story about the hack, warning that it “threatens to seriously undermine Cornell University’s ability to carry out our mission to protect the information security of our faculty, staff and students.”
Cornell Media Relations also told IT administrators to refer any inquiries about the hack to media relations. A media relations spokesman declined to comment on the 2014 breach and said publicizing a successful cyber attack could make contingent IT staff not want to work for the University and could result in a sharp increase in attempted hacks.
But those who worked to expel the hack, as well as independent experts who spoke to The Sun, said any potential embarrassment stemming from a successful hack is vastly outweighed by the educational benefit of making details about the attack public and reducing the stigma of being the victim of a cyber breach.
“Hiding [the hack] under the carpet is not going to make it go away,” Vincze said. “All of these are just learning experiences. I think that, in that sense, everybody who is hacked should share with the community responsibly.”
“We’re at the point now, in 2016, where I don’t think there is one single university or one single business that hasn’t been hacked,” she added. “It’s the general public that needs to be much more aware.”
Derek Ruths, associate professor at the McGill University School of Computer Science, said publicizing hacks can lead to an increase in hack attempts, but those are likely to be “script kiddie” attacks carried out by amateur hackers using downloadable programs.
“It would be really bad if Cornell was susceptible to script kiddie attacks,” Ruths said. “I’m more inclined to think that you’d have a bunch of idle computer scientists — maybe even at Cornell — who think this is a fun thing to try and do on a Saturday night.”
The massive amount of work in the months after the hack was rough for IT staff and disruptive for faculty members, who had to change their passwords multiple times, Bandler said, but was also a wake-up call that forced Cornell to update some of its technology practices.
Nicholas Bogel-Burroughs / Sun Staff Writer
The Cornell IT Security Office is located at 120 Maple Avenue.
Bandler said his mantra to staff members is to “never let a good crisis go to waste,” and the 2014 hack was a prime example.
“You suddenly have the attention of a college, of a whole group of people, who have experienced something very unfortunate,” he said.
The senior IT official said Cornell was more interested, after the hack, in pursuing security practices that had been languishing, such as requiring two-factor authentication for some programs and allowing IT to exert more control over the security of websites or programs previously not managed by IT staff.
“It’s embarrassing to have this incident in the public eye, but there is an awareness-raising benefit in discussing the incident,” the senior IT official said.
The biggest impact of the hack was the work it took to expel the hackers and clean up the compromised systems, administrators said. Technology staffers had to search for and delete the custom malware hackers left behind, which meant temporarily shutting down websites — some for close to a month — before cleaning them and uploading them onto a new server.
Up to 40 individual computers were penetrated by the hackers, but IT cleaned many more as a safety precaution, according to the senior IT official. And, because Cornell cannot pinpoint exactly when the hack began, the process required staff to install an entirely new operating system instead of reverting to a backup.
“It was crisis mode for at least a week or two, then it was just hardcore production mode for another two weeks,” said a technology staffer in the Yang-Tan Institute, which maintains about 50 websites, including the one hackers rebooted to gain entry. “There were three or four of us that probably put in 60-hour, 70-hour weeks for about a month.”
But the crisis was also an opportunity for IT staff to develop more channels of communication as ILR IT and the Cornell IT Security Office on Maple Avenue all battled to eradicate the hackers, the YTI technology staffer said.
The hack, while undetected for months, was discovered relatively quickly, according to a report by a leading cybersecurity company that said organizations, in 2014, detected hackers a median of seven months after intruders gained entry to a network. Cornell discovered the hack about four months after it began, and then spent three months fixing vulnerabilities in the network and instituting more security measures.
Cornell was the victim of a much less powerful state-sponsored attack in August 2015 that affected a few individual, isolated machines, the senior IT official said, adding that cyber attacks sponsored by foreign governments are a growing threat.
“Every research university is subject to these sort of state-sponsored attacks,” he said. “Unfortunately, they’re increasing in frequency and sophistication.”