THE SETUP: There’s no perfect way to evaluate a talking AI database, let alone four of them, but I tried to cover as many fundamental topics as I could.
In the end, I settled on 12 broad categories Iâd want my assistant to nail:
â˘Â Travel
â˘Â Email
â˘Â Messaging
â˘Â Sports
â˘Â Music
â˘Â Weather
â˘Â Calendaring
â˘Â Social
â˘Â Translation
â˘Â Basic Tasks
â˘Â General Knowledge
â˘Â Personality
Within those categories, I asked a handful of questions and commands that seemed like they would be common.
Itâs also important to note which devices I used, since that can have an effect on how well each assistant performs technically.
â˘Â For Siri, I used an iPhone SE.
â˘Â For Google Assistant, I used a Google Pixel XL.
â˘Â For Cortana, I used a Lenovo ThinkPad X1 Yoga.
â˘Â And for Alexa, I used an Amazon Echo Dot.
Itâs worth mentioning that Amazon is currently rolling out Alexa to Kindle Fire tablets, but I wasnât able to get that update before writing this piece.
With that out of the way, onto the tests.
The first category was TRAVEL. After supplying each assistant with my New Jersey address, I asked, “How do I get to the Port Authority Bus Terminal?”
Google Assistant made things the easiest. It knew exactly what I was talking about, and immediately opened up a route in Google Maps, as if I had already searched it in there. It put me on my way within seconds.
Siri got the idea, but in a roundabout manner. It showed a card with six different results, and weirdly said, âOne option is Port Authority Bus Terminal on New York in New York. Do you want that one?â After confirming, though, it did the same thing as Google, just with Apple Maps â which is better, but still not the best.
Being stuck in a Bluetooth speaker, there wasnât much Alexa could do. It did use Yelp data to identify the right place, tell me its location, and tell me its hours, though it didnât give any mapping info in its companion app.
Cortana pulled up a list of three potential places, then asked which one I wanted. When I said âthe first oneâ (which was correct), it did a Bing video search for the phrase âthe first one.â Not great! When I clicked on the right result, though, it opened up a route in Windows Maps, which is fine, but not on Googleâs level.
Winner: Google Assistant
“Is there any traffic on the way to the Empire State Building?”
Siri and Google Assistant did the best: Both verbally explained the delays and estimated how long itâd take to get there from my location. Iâll give Google Assistant the edge for opening up a route in Google Maps for good measure.
Cortana merely brought up a widget full of info about the Empire State Building; I had to click on a âdirectionsâ button to see what I wanted.
Alexa, meanwhile, only provided an estimate to one specific address at a time, which you have to configure within the Alexa app.
Winner: Google Assistant
next slide will load in 15 secondsSkip AdSkip Ad“Book me a flight to Boston.”
Nobody really nailed this, but again, Google Assistant was the best of a mediocre bunch. It brought up a Google Flights card with various airlines and starting ticket prices. You can click on it to schedule as needed, but it weirdly defaulted to flights happening two weeks from now. If you specify you want the ânext flight,â itâll show more recent results, but you also canât be too conversational with it: When I tried to say âIâll take the first one,â it didnât understand.
Thatâs still better than the others, though.
Siri just gave Bing results, Cortana opened up a Bing search in Microsoftâs Edge browser, and Alexa can only track and search for flights through a downloadable Kayak add-on, not book them.
Winner: Google Assistant
“Call me an Uber.”
Siri did well here. If you enable the Uber app, itâll ask what kind of ride you want (UberX, UberBlack, etc.), then bring up an Apple Maps widget showing the closest car, ETA, and an estimated cost. Confirm that itâs got the right details, and it sends the car on its way. Itâs more or less exactly what youâd want.
Cortana has it down, too. Connect your Uber account, say the type of car and destination you want, and it sets everything up.
Alexa is a bit more finicky, but if you download the Uber âskill,â it works in much the same way. Being confined to a speaker just makes it a little more uneasy for the time being.
Google Assistant is the outlier. It canât do any of this today, but that should change once Google opens up the platform to third-parties in December.
Winner: Siri (with Cortana and Alexa close behind)
Next, let’s see how easy it is to EMAIL.
“Send an email to Jeff Dunn.” (Yes, I have multiple addresses.)
Siri and Google Assistant were neck and neck. Each one recognized my name in my contacts, had me dictate the message, and sent it out quickly. Siri let me specify a subject line, Google Assistant had a richer interface. The major difference is that Siri only works with Apple Mail, while Google uses Gmail. I prefer the latter.
Cortana can do most of the same things, but I wasnât able to get it to work as smoothly. It repeatedly identified me as âJeff doneâ â it is my curse â and often asked if I wanted to text someone rather than email.
Alexa couldnât recognize the question. There are hacky workarounds you can create with IFTTT to get something close to the idea, but obviously thatâs not ideal.
Winner: Google Assistant
“Do I have any new emails?”
Siri took this one. It told me how many unopened emails I had received since yesterday, then displayed those â and only those â in a simple Mail widget.
Google Assistant didnât say anything, and brought up a card with two not-quite-recent emails from a few months ago. Click on one, and it brings up a weirdly isolated Web page for that specific email, not the Gmail app.
Alexa had the same problem as above, and Cortana suggested I make a Web search for the phrase âDo I have any new emails.â No thanks.
Winner: Siri
next slide will load in 15 secondsSkip AdSkip Ad“Read my most recent email.”
Same as above. Siri is still tied to Mail, but it immediately brought up my last email, then read me the sender, date, subject line, and message â all hands-free. It awkwardly cut itself off where the text did, however.
Google Assistant didnât actually read the message; it just displayed my last two emails in another info card. And again, it doesnât give a direct link to the Gmail app.
Alexa still couldnât do anything. And once again, Cortana opened a Bing search for the phrase âRead my most recent email.â
Winner: Siri
Next, let's look at MESSAGING.
“Send a text to Jeff Dunn.” (Yes, I have multiple numbers.)
Siri, Google Assistant, and Cortana all recognized the name, then made it easy to see and send a message.
Alexa has more third-party workarounds for this, but itâs not as straightforward as using SMS. Given itâs stuck in a Bluetooth speaker, though, I can give it a little more of a pass.
Winner: Siri, Google Assistant, and Cortana
“Do I have any new texts?”
Siri was the only one to work here. Not only did it read off the sender and message, it also asked if I wanted to reply, and walked me through a new text from there.
Google Assistant told me it couldnât read texts. This is bad â not because it seems fairly basic, but because the main Google app can do it already. It works just as well as Siri there, but the fact that itâs siloed off is strange.
Again, I didnât use Alexa and Cortana on a phone, so I wasnât able to make any headway here.
Winner: Siri
“Read me my Tweets.” (To test third-party integrations, and because I needed some darkness in my life.)
The phrasing threw Siri off. It is capable of reading tweets, but only if you ask something like âWhatâs trending on Twitter?,â or âSearch Twitter for #NewMacBook.â In that case, itâs okay, but poorly curated.
Alexa is the most useful here â provided you download the Twitter Reader skill. Then, it can read the last tweets you liked, trending tweets, the last tweets to mention you, and so on.
Cortana didnât recognize the phrase, either, but is at least capable of sending tweets if you use a phrase like âTwitter, new tweetâ beforehand.
Google Assistant gave me a Google search result, though it can share an article to Twitter, and again, might get better as Google opens up the platform.
Winner: Alexa
next slide will load in 15 secondsSkip AdSkip AdOkay, let’s move onto something more fun: SPORTS.
I started with an easy one, asked during the middle of Game 5 of the World Series: “Who’s winning the Cubs game?”
Siri, Google Assistant, and Alexa churned out the current score with aplomb.
Cortana, on the other hand, brought up links to articles about the Cubs.
Winner: Siri, Google Assistant, and Alexa
“Who do the Celtics play next?”
Everyone got it right. Hooray!
Winner: Everyone!
After asking the previous question, I asked, “Who is their top scorer?” to see if these assistants understood context.
Nobody could do it. Not hooray! Worse, everyone but Google Assistant couldnât recognize the word âscorerâ without multiple tries. Instead, I got a lot of âtop score.â
Winner (or "least loser"): Google Assistant
“Who won the Premier League?”
The correct answer is Leicester City. Unfortunately, none of the assistants appear to be footy fans. Even when I specified âWho won the Premier League last year?,â nobody got it right.
Siri brought up the current league table. Alexa had no idea. Cortana pulled up the latest scores. And Google Assistant thought itâd be a good time to read the Wikipedia entry for the Premier League in general. It wasnât.
Winner: Nobody!
next slide will load in 15 secondsSkip AdSkip Ad“Which quarterback started the last game for the Cleveland Browns?”
For the record, the answer here is Josh McCown.
Only Siri was able to outright say that, but it did so with the response above, where it said âJosh McCownâ five straight times, before reading off the rest of the Browns roster. This, as you can imagine, was horrifying.
Google Assistant served up a card with Wikipedia info on past Cleveland Browns QBs, while Cortana called on its old friend Bing to show QBs on the current Browns roster.
Thatâs still better than Alexa, though, which rattled off names like Brady Quinn, Jeff Garcia, and Tim Couch â all of whom played for the Browns years ago.
Winner (begrudgingly): Siri
I first asked each assistant to play a song: “Queen Jane Approximately” by Bob Dylan.
This one was tricky. No one assistant works with every music streaming service, which leaves you walled into specific gardens. It's a similar deal with video.
Apple is the worst about it. Siri works harmoniously with Apple Music, but thatâs the only music service it supports.
Google Assistant is a little better, if only because it has some support for Spotify, but it still really wants you to use Google Play Music or YouTube. (I use the former, so it quickly started a âRadioâ playlist based on my song).
Cortana is similar â just replace Googleâs stuff with Microsoftâs little-known Groove Music service.
Relatively speaking, Alexa made things the easiest. It defaults to Amazonâs new Music Unlimited service, but is also the most realistic about Spotifyâs dominance. It even lets you make Spotify your default player.
Winner: Alexa
“Play me some new music.”
Nobody impressed here, but Alexa came closest. Instead of playing entirely new albums, though, it started the last music I added to my Amazon Music library. Playing just a bit loose with the phrasing there, Amazon.
Still, that was better than the rest.
Siri merely shuffled my library of songs, Cortana searched Bing for music videos based around the phrase âplay me new music,â and Google Assistant hilariously opened up the album âNewâ by Paul McCartney in Google Play Music.
Winner: Alexa
Then I played a song off my laptop (Wilco’s “War on War”), and saw who could correctly identify it.
Cortana and Siri caught it â the former then gave a link to buy through Groove Music, while the latter, weirdly, gave a link to Shazam. (Since itâs using that tech to find the song.)
Alexa and Google Assistant were incapable â although, Alexa can almost always identify music if it's playing out of the speaker itself. This was a bit troubling for Google Assistant since, again, itâs something the main Google app can already do.
Winner: Siri and Cortana
next slide will load in 15 secondsSkip AdSkip Ad“What’s the weather going to be like tomorrow?”
Everyone got the point across, but Siri was the weakest since it didnât explicitly say what the forecast would be. Instead, it says something like, âlooks like sun tomorrow,â and forces you to look at your phone to see the little Weather Channel widget it brings up. The point of these things is to go hands- and screen-free.
Winner: Google Assistant, Alexa, and Cortana
A conversational follow-up: “What about Boston?”
Google Assistant and Cortana were able to keep up and provide similar forecast info. Siri and Alexa didn't understand the follow-up, and required me to outright ask âWhatâs the weather going to be like in Boston?â
Winner: Google Assistant and Cortana
“Will I need an umbrella this week?”
Everyone but Cortana got the gist. (It didnât say anything; it just pulled up a graph with weather info.)
Of the other three, Google Assistant was the clearest. It was the only one to take the next step and specify which day to expect rain, whereas Siri and Alexa only noted if rain was expected at all over the next seven days.
Winner: Google Assistant
Now it’s time for CALENDARING.
next slide will load in 15 secondsSkip AdSkip Ad“Do I have anything scheduled tomorrow?”
Everyone recognized just how many events I had on my calendar, then read them back to me. It's mostly a tie. Though itâs worth noting how Cortana and Siri try to push you toward their companion apps, while Alexa and Google Assistant want you to use Google Calendar.
Winner: Google Assistant and Alexa
Then, as a follow-up, “How about the rest of this week?”
Surprisingly, only Siri was able to connect the dots. It quickly expanded the calendar widget from the last question to include my whole week, then read it out.
Google Assistant and Cortana needed me to clarify âwhatâs on my calendar this week?,â while Alexa told me the query wasnât currently supported.
Winner: Siri
I asked each assistant to “Schedule dinner with Mom on my calendar Wednesday night,” then saw how easy it was to change and cancel that appointment after it was made.
Siri killed it here. It recognized âMomâ from my contacts list, asked when I wanted the appointment to start, then was entirely fast and straightforward when I asked to confirm, change, and eventually cancel it. Again, though, it defaults you to Appleâs stock app â without Calendar, thereâs little you can do.
The other three were able to schedule things easily enough, but struggled when it came time to modify the date.
Google Assistant was very smooth at creating an event, but wouldnât change it from the app no matter how many ways I tried to phrase it. Alexa had the same issue. Cortana required me to specify which time I wanted to change my appointment to, then wouldnât let me cancel it with my voice alone.
Winner: Siri
Now it’s time for SOCIAL stuff.
“What’s happening near me?”
Alexaâs openness benefitted it here â if you download the StubHub skill, you can get some semblance of a useful response and hear what shows are playing in your area.
Itâs far from perfect, but itâs better than Siri (which resulted in a Bing search) and Google Assistant (which just gives you a shortcut to Eventbrite).
Cortana wasnât terribly far off, though: It brought up a widget with upcoming comedy shows and sports events in New York City, with an option to see more events through Bing.
Winner: Alexa and Cortana
next slide will load in 15 secondsSkip AdSkip Ad"What are some Chinese restaurants near me?”
All of them were able to pull up a few good spots, but I preferred Siri for being the most proactive in actually getting me a seat.
Siri listed the closest place to me, along with its Yelp rating and the fact that itâs âinexpensive,â then asked if I wanted to call the venue or get directions. I said Iâd take the latter, so it opened up a route on Apple Maps. When I said I wasnât interested, it moved onto the next closest spot. This is all common sense and useful.
Winner: Siri
“I want to make a reservation at Mission Chinese.”
Again, only Siri could do this, thanks to its integration with OpenTable. In the example above, it found the closest restaurant near me, but was able to note that I couldnât book right now since it was closed. It even offered to call the place if I wanted. The other three were either incapable or brought back search results.
Winner: Siri
“How do I say ‘where is the library’ in Spanish?”
For the most generic of foreign language questions, only Google Assistant and Cortana were able to help out. They even pronounced it, too.
Siri brought back Bing results, while Alexa was stumped.
Winner: Google Assistant and Cortana
Then it was time to see how well each assistant went about BASIC TASKS: opening apps, setting and cancelling alarms, creating reminders, and changing the screen lock time (on a phone).
This is the area where these assistants are most reliable. There werenât any major problems with requests like the ones above, but there were a few quirks worth noting.
-- Siri was able to change the screen timeout time, while Google Assistant wasnât.
-- Siri was also able to open up a website in a browser (Safari, naturally), while Google Assistant would only open a site within the Assistant app itself, not Chrome.
-- Alexa was also the most thoughtful at setting alarms: It was the only one to ask if I meant a time for the afternoon or the evening.
Winner: Siri and Alexa
next slide will load in 15 secondsSkip AdSkip AdNext up was GENERAL KNOWLEDGE, or just how dumb each of these things are. I asked a bunch of random questions: “How far is the Sun from the Earth?,” Who is the current Speaker of the House?,” and “What is the square root of pi?”
Only Alexa got all three correct while speaking the answers aloud.
Cortana got them right, too, but resorted to a graphic to show the Sunâs distance from our dark blue marble.
Google Assistant was the only one to get questions outright wrong: On the Speaker of the House question, it pulled up a card that listed Paul Ryan on it, yet still cited a source that claimed it was John Boehner. On the pi equation, it pulled up a card with the correct answer, but only said, âThe square root of pi, like pi itself, is an irrational number.â
Things like that are more exceptions than the norm, but still, it's not bulletproof.Â
Winner: Alexa
“What’s the news today?”
Alexa and Google Assistant pulled up a âFlash Briefingâ from NPR with the major headlines of the day. Iâll take it. The latter used it as part of a wider audio playlist, with other clips from Fox News, Bloomberg, ESPN, and BBC, so it wins out.
Siri awkwardly said, âHereâs some news for âwhatâs the news today,ââ then brought up a Bing search widget with mostly relevant stories.
Cortana, meanwhile, returned a few news stories based on the actual word âtoday.â So the top result was something about the Today show. When I asked âwhatâs the news,â it gave me a dictionary definition of the word ânews.â Amazing.
Winner: Google Assistant
Let’s close with the goofy one: PERSONALITY.
I asked each assistant to tell me a joke, and braced myself for bad puns. They didn’t disappoint.
Siri: âThe past, present and future walk into a bar. It was tense.â
Google Assistant: âWhat do you call an empty jar of Cheez Whiz? Cheez Was!â
Cortana: âHow many politicians does it take to change a lightbulb? Two. One to change it, and another one to change it back again.â
Alexa: âWhatâs brown and sticky? A stick.â There are a lot of dads on these development teams, I presume.
Winner: People who enjoy bad jokes.
Cortana and Google Assistant were the most eager to comment on their rivals.
While Siri and Alexa keep things close to the vest, Google and Microsoft seem happy to be a little more snarky.
Hereâs Google Assistant on Siri: âI know sheâs from Cupertino. Thatâs pretty close to Mountain View.â
And Cortana on Google: âI like to imagine the âIâm Feeling Luckyâ button in Clint Eastwoodâs voice.â
Winner: Google Assistant and Cortana
next slide will load in 15 secondsSkip AdSkip AdFinally, I saw who was up for playing a game.
Google Assistant blows everyone away here, with quiz games, MadLibs, a digital Rubikâs Cube, and more. You can sink some time into it.
Siri stonewalled me, while Alexa played a very surreal game of Rock-Paper-Scissors. (You do not know the emptiness of artificial intelligence until you call out âPaper!â to a speaker that doesnât answer in return.)
Cortana appeared to only have one game, where it gives you clues about a movie and you try to guess what it is. When I said, âI want to play a different game,â it then Bing searched videos for that particular phrase.
Winner: Google Assistant
So what did we learn here?
If the hodgepodge of results above didnât make it clear, none of these things are at a place I could comfortably call âgood.â There is a ton of work to be done.
The problems here are large and sweeping:
⢠Each assistant still feels like a fragile, thinly-veiled web of loosely connected services, because thatâs what they are. Itâs almost impossible to tell when one of them wonât be able to do the thing you asked.
⢠You have to be okay giving up your location and loads of personal data to get the most out of them.
⢠There are numerous instances where using a Web browser is simply faster for doing fundamental tasks. There is a reason most people use their Echo for the simplest of functions â itâs not worth slowing down your workflow to do anything else.
⢠Each one is still wildly finicky when it comes to phrasing. They all think too much in black and white; one misplaced or forgotten word is often enough to discard an entire request.
⢠Itâs incredibly uncomfortable to speak to a inanimate thing in public.
⢠In Google Assistantâs case, normalizing the need to call on a brand (âokay Googleâ) whenever you need a hand is something out of Orwell.
All that said, if I had to take one, I’ll give Google Assistant the slight edge.
I know, I know â there are a lot of embarrassing moments for Google up there. If I were to simply count each âwinnerâ above, Siri would be right there with it. Things are closer than the recent Google hype machine would suggest.
Technically, though, Google Assistant was the smoothest to use. I didnât have to repeat myself as many times as I did with the others. It was also the most adept at understanding context. Even if Siri feels more in tune with the device itâs a part of, I felt more like a robot talking to it. The Assistant is also fun â it runs like something Googleâs people enjoyed being creative with. And in the areas where Google was behind, the gap wasn't huge.
The main thing, though, is potential.
Google is sitting on a treasure trove of data. It has a massive ecosystem of popular software services on which the Assistant can call. Itâs a leader in search, email, calendaring, music, streaming hardware, the works. Itâs everywhere, and the Assistant is the line that runs through everything. At least, that's the idea.
Siri, on the other hand, is patchwork: It needs Yahoo here, WolframAlpha there, Appleâs own middling apps everywhere else. It'll be hard for it to feel as cohesive.
As for the others: For what it is, Alexa is fine. It excels at simple tasks, itâs open, and itâs the best at some of the nichier things I didnât get to, like online shopping (of course) and smart home control. It just canât handle as much as Google.
Cortana is a Bing shortcut.
Ultimately, youâre looking to the future with these things. Going forward, Google has the most cushion to lean on.