Battle of the Virtual Assistants
Some people enjoy making their mobile devices' virtual assistants say funny things, but the humor fades quickly when the features fail to reply to simple commands. When it comes to virtual assistants, competition is fierce among Microsoft, Apple, and Google. We gave Cortana, Siri, and Google Now the same set of tasks and ranked each one's efficiency, accuracy, and more. While Cortana and Google Now performed well in some categories, Siri proved to have the most accurate and efficient responses coupled with the highest customer satisfaction. Our objective was to explore the various ways Siri outperformed the competition and to give consumers an expectation for their own virtual assistants' performances.
Watch Them in Action
We took Siri, Cortana, and Google Now on a test drive to discover firsthand how each virtual assistant's performance measures up. We asked them to perform several tasks - here's a preview of how well each assistant heeded our requests:
Schedule my dentist appointment for December 1st at 1 pm
When does Kung Fu Panda 3 come out?
Where can I buy tires for my car?
Responding to Basic Commands
With a sweeping victory across the accuracy and satisfaction categories, Siri was the clear winner in terms of responding to commands. Google Now was a mixed bag in its second-place victory, with Cortana receiving higher ratings in a few categories.
The greatest disparity among the three assistants was when they were tasked with sending a text message - vital for drivers who want to text hands-free while on the road. Siri wiped out the competition with a whopping 82% accuracy rating, Google Now landed at 68%, and Cortana achieved only 60% accuracy.
After testing basic commands, we decided to see whether users awarded Siri top marks for overall performance.
When it comes to user satisfaction, Siri once again came out on top, while Cortana trailed. With high marks virtually across the board, Siri is a pro at basic commands. Her strong personality and dry wit don't hurt her popularity either: Siri has made headlines for correcting users who call Caitlin Jenner 'Bruce' and coaxing an autistic child out of his shell.
Google Now scored points for talking sports stats and was deemed acceptable for most other basic tasks. While she may not have Siri's sass, Google Now takes her personal assistant gig to the next level by scanning communications to look for upcoming events, bill due dates, and more to add to the calendar.
While Cortana ranked lower than the others in most categories (do not ask her about calorie counts or shopping options), she does have some positives: Her 18 emotions infuse some personality, and she's great at spouting Super Bowl stats.
Siri remains a fan favorite, but we wanted to determine whether her popularity spans the entire country.
The Impact of Geography
Interestingly, each side of the country appears to have different taste in virtual assistants. Siri was the top choice for both sides of the country, with more than 78% of Easterners and more than 84% of Westerners reporting satisfaction. But the Western U.S. showed an even clearer preference, giving lower marks to the others. Less than 50% were satisfied with Cortana, and only about 65% were happy with Google Now. Only slightly fewer Easterners prefer Cortana and Google Now than Siri. Is it simply brand preference? Or do regional differences such as accents and sense of humor come into play?
Along with the questions on region, we wanted to tackle the subject of gender. We decided to explore whether virtual assistants respond differently to men and women. Do both men and women believe Siri to be a better performer and Cortana to be deficient, as we saw when we looked at the numbers in aggregate?
The Impact of Gender
Men and women tend to have distinct communication styles and voices of contrasting pitches. How do these differences affect virtual assistants' responses? Upon speaking to Cortana and Google Now, women receive follow-up questions two times more often than men (10% from Cortana and 8% from Google Now vs. 5% and 4% respectively). Siri, however, responds with follow-up questions to men (6% of the time) more than women (4%). Are women less clear and thorough with Cortana and Google Now, or does Siri simply listen to women better?
Next, we tackled the issue of phone cases. They lend character and add protection - but how do they affect virtual assistants? We wanted to know whether phone case use might be giving an unfair advantage or disadvantage to the virtual assistants. Could it be possible that Cortana is simply worse at deciphering muffled sounds?
The Impact of a Phone Case
We discovered that phone cases can positively impact a virtual assistants' overall performance, as forgoing a case can lead to more incorrect answers. Once again, Siri was the clear leader across most categories, with Google Now edging it out for fewer incorrect answers in the absence of a case.
So which assistant comes out on top overall?
The survey results are pretty much unanimous: Siri is the favorite virtual assistant. When Apple introduced Siri in 2011, her AI capabilities seemed like something from the future. However, her ability to compete a variety of commands with accuracy has led her to receive the highest rankings from our survey respondents.
Of course, since Android users were excluded from Siri's audience, the debut of Google Now in 2012 earned it a fanbase that saw its distinct advantages over Siri One - including the ability to activate Google Now hands-free with a voice command, "Okay, Google." According to our survey, though, it still has some catching up to do.
While Cortana came in third place, she's still relatively new to the virtual assistant landscape, launching in 2014, and her capabilities remain impressive.
Siri might have won this at this time, but with incredible advances in voice recognition and AI technology, a 81% user satisfaction rating might not be enough to hold onto that title for long.
We ran a survey asking participants to perform a list of commands on their smartphones. With these findings, the virtual assistants were ranked based on accuracy (whether or not the virtual assistant properly responded) and satisfaction (how happy users were with the way the virtual assistant performed the command) All participants were randomly selected, users were required to use Cortana, Siri, or Google Now.
Feel free to share the images found on this page freely. When doing so, please attribute the authors by providing a link back to this page, so your readers can learn more about this project and the related research.