Overview of Intelligence Pillar in the Cortana Intelligence Suite

By: Dattatrey Sindol | Comments | Related: > Cortana

Problem

I have been following the tip series on Microsoft Cortana Intelligence Suite and have been exploring the areas from previous tips. Now I would like to know more about the offerings that enable Intelligence and some pointers on getting started.

Solution

We have been using the traditional business intelligence systems for past few decades and big data systems for past decade to solve various business problems and to enable businesses to stay competitive in the market. However, as there is more and more data getting generated / collected, new devices and technologies being adopted at a rapid pace, there is also a need to enhance / revamp the analytics systems by making the systems more intelligent. How can we predict what could potentially happen in future to enable businesses to stay competitive in the market? In this tip, we will look at the fourth pillar of Cortana Intelligence Suite which contains offerings to do exactly this.

Overview of Intelligence in the Pillar in the Cortana Suite

Cortana Intelligence Suite comprises of the following offerings as part of the Intelligence pillar, that can be used to make existing systems intelligent and to build more advanced intelligent applications:

Cognitive Services
Bot Framework
Cortana

Cognitive Services in the Microsoft Cortana Intelligence Suite

Cognitive Services are a set of APIs that enable Cognitive Functionality via natural and contextual interaction with tools / applications that augment the user experience by leveraging the power of artificial machine-based intelligence.

Cognitive Services APIs are currently available across the following major categories:

Vision: Vision APIs enable processing and analysis of faces, images, emotions, and videos by leveraging image processing algorithms and enable building smart and personalized applications. Some of the APIs include: Computer Vision API, Face API, etc.
Speech: Speech APIs enable processing of spoken language within the applications by leveraging APIs that allow conversion between speech and text, speech recognition, etc. Some of the APIs include: Bing Speech API, Speaker Recognition API, etc.
Language: Language APIs enable applications to process natural language / user language to evaluate the sentiment and topics and understand what the user is trying to convey. Some of the APIs include: Bing Spell Check API, Linguistic Analysis API, etc.
Knowledge: Knowledge APIs allow us to map complex data and information to achieve some of the complex tasks including intelligent recommendations and semantic search. Some of the APIs include: Academic Knowledge API, Recommendations API, etc.
Search: Search APIs enhance the search experience across applications and web pages and make the search smarter and more engaging. Some of the APIs include: Bing Web Search API, Bing Autosuggest API, etc.

Here are a few examples of typical use of Cognitive Services APIs:

Capture an image of a person, say who is shopping at a supermarket, and pass it to the face detection API which can analyze the image and provide more information about the image like gender, age, emotion, reaction, etc. to understand how customers like products and how easy they are finding them at the supermarket.

For instance, take a look at How-Old.net and pass a photograph as an input and see how it predicts the age of the person pretty accurately in most cases. Following screenshot shows the age predicted by How-Old.net for one of the photos of Warren Buffet pulled up using Bing Search. As we can see, it predicted the age with more than 90% accuracy.
Recordings of customer care executives talking to the customers can be analyzed using Speech and Language APIs to understand the sentiment of both customer as well as the customer care executive, analyze the patterns which can enable businesses to fine tune their internal training and quality practices to provide better customer service / support.
And many more similar scenarios can make use of Cognitive Services.

Refer to the following resources to learn more about Cognitive Services:

Bot Framework in the Microsoft Cortana Intelligence Suite

A Bot is a web service that interacts with users in a conversational format. Depending upon how/where the bot is configured, users can interact with the bots via different channels. Some of the common channels include SMS, Skype, etc.

The Microsoft Bot Framework is a comprehensive offering for building and deploying high quality bots, to engage users across common channels of conversation.

Here are few highlights of the Bot Framework:

Bots can be integrated into the same channels commonly used by the users without requiring them to leave their favorite communication channel and at the same time providing an enhanced digital user experience.
Bots can be deployed across various popular channels including but not limited to SMS/Text, Skype, and Facebook Messenger among others.
Bots can be used in conjunction with Cognitive Services to make the Bots more intelligent by adding senses that enable them to see, hear, and interpret similar to humans.
Language Understanding Intelligent Service (LUIS) can be used to make the Bots understand the language thereby making the user experience more seamless and close to a human to human interaction.
The Bot Framework Emulator allows Bot Developers to test and debug their Bots locally.

Here is a typical example of an intelligent Bot. The Skyscanner Bot on Skype allows you to search for flights for your upcoming trip.

You can view an example of a Search Query which will give you an idea of how to ask a question.

An Example Query on Skyscanner Bot within Skype

Now let us say I search for a flight from Bengaluru, India to Chicago, USA and see what this bot comes back with.

Response to a Flight Booking Query on Skyscanner Bot within Skype

Now you can go on with further searches or click on Book which will re-direct you to the Skyscanner website where you can book your tickets. There are various other Bots available on Skype and other popular channels.

Refer to the following resources to learn more about Microsoft Bot Framework:

Cortana in the Microsoft Cortana Intelligence Suite

Cortana is a Digital Assistant that helps users stay informed and be productive by getting things done via easy commands across different platforms and devices.

Here are few highlights of Cortana:

Offers an interactive search interface.
Recognizes both text based as well as voice based commands.
Performs various common tasks based on the input commands.
The application can be either activated in the background or launched in the foreground via appropriate commands based on the task to be performed.
Works across different platforms and devices.
Offers a robust and comprehensive extensibility framework to incorporate the user application functionality into the Cortana experience.

Here is a typical example of Cortana Digital Assistant on Windows 10. Let us say I search for current weather condition in Bengaluru, India by issuing a voice command to Cortana.

Request for Weather Condition in Cortana on Windows 10

Now Cortana comes back with detailed Weather Update about Bengaluru, India as shown below.

Response to Weather Condition Request in Cortana on Windows 10

As we can see, it reduces so many clicks required to get the same information by opening a browser and searching for this information. Cortana brings back this information so quickly and easily making it pretty much effortless.

Next, let us say I want to block my calendar for tomorrow 8:30 AM.

Request to Setting up Appointment in Cortana on Windows 10

Cortana comes up with a dialog with all the necessary fields required to setup the appointment as shown below.

Response to Appointment Request in Cortana on Windows 10

We can just fill in the required details and add this appointment to the calendar. Like this, we can go on interacting with Cortana to get information / updates and to get the things done.

Refer to the following resources to learn more about Cortana: