February 5, 2016

Cornell Tech Students Make App for Filtering Sarcastic Reviews

Print More

A group of Cornell Tech students has created a mobile application named TruRatr, which automatically detects sarcasm in consumer app reviews, according to Christopher Hong, a Bloomberg LP software developer and the project’s mentor.

TruRatr weeds out misleading product reviews based on language sentiment, leaving viewers with only genuine assessments. Consumers will be able to use authentic information to make well-informed purchases without having to sift through hundreds of reviews, according to tech magazine ArsTechnica.

Currently, the app also removes sarcastic reviews from app pages and revises the product’s rating to account for the change.

The team of six students, who are studying topics from mechanical engineering to computer science to business, started designing the app last fall in a collaboration between Cornell Tech and Bloomberg, according to ArsTechnica.

Hong explained that TruRatr operates by looking for “sentiment shifts” — a positive sentiment followed by a negative sentiment or vice versa — in app reviews.

“I love getting yelled at” is an example of a sarcastic statement, because it has a positive-sounding verb paired with a negative object, according to Hong.

Another example of a review TrueRatr might filter out is “I REALLY FVCKING LOVE THIS APP BUT THE CAMERA OF THE LENS IS NOT WORKING ON MY SNAPCHAT!! PLEASE FIX IT!! THANKS!!” — a hypothetical review of the Snapchat app — according to ArsTechnica.

Detecting reviews like this would raise Snapchat’s review ratings from 3.0 to 3.82, using TrueRatr’s algorithm.

Ming Chen, one of TrueRatr’s back-end developers, added that sarcasm detection is complicated and cannot be easily optimized.

“Whether a review is sarcastic or not is not determined by a simple formula, but by a machine learning algorithm,” Chen said. “That means we extract the patterns from the training set, and there are many patterns instead of a single one.”

As part of their research, the team analyzed over 158 sarcastic reviews from Amazon Mechanical Turk — a crowdsourcing service — and used a “training set” of 1,188 reviews to search for patterns, according to ArsTechnica.

Chen said TrueRatr’s algorithm searches for cases of all known patterns of sarcasm in any given review, and then “combines the number of occurrences to determine whether a review is sarcastic or not.”

TruRatr was created in response to the shifting nature of consumer buying behavior, according to the project team’s presentation.

“The influence of user-generated reviews is stronger than ever but is primarily held back by lack of accountability and transparency,” the team said. “As number of reviews per product increases, reading through reviews is not feasible.”

The team tested TruRatr on a sample of 200 sarcastic and non-sarcastic Amazon reviews, and the app correctly identified 75 percent of them.

The algorithm behind TrueRatr is currently open source. The team behind the app hopes this will allow more people to be able to test the technology and improve on it, according to ArsTechnica.

  • I’ve written about earlier and also admirable work by Cornell in my book: “Finding Reliable Information Online: Adventures of an Information Sleuth.” (Rowman & Littlefield 2015) Unfortunately, research tells us that for every fake review detection software program developed, people being hired to write these reviews by companies through places like Fiverr are quickly able to find a work around and beat the system. I’m not sure we need to sift through hundreds of reviews, what we really may need is trustworthy human curators to collect the reviews and ensure their authenticity. A collection of research now estimates that between 10-30% of reviews are fake, that people typically can’t spot the fakes though they think they can, and that snowball effects occur so that an early positive or negative review can influence subsequent reviews and skew the results. Please see lesliestebbins.com — or look up my book on Google Books to see more.

  • DisappointedAlum

    Why would you do this? Sarcastic amazon reviews are some of the must pure and joyous things in this world?