The Hindu Business Line : Tuning in to <I>your</I> moods...

One recent Saturday afternoon, while answering e-mails, I was interrupted by a call from the credit card company. It was the third `gentle' reminder to pay my bills.

I had paid them and the bank too had acknowledged debiting my account. I was furious and gave them "a piece of my mind".

Slamming the phone down, I returned to continue typing the e-mail. The computer did not allow me. All the keys were locked and nothing worked. A few seconds later, I heard, "You are not in a good mood. This is not the time to type these emails. Relax, Calm down and come back".

What! Was that a voice from heaven or my inner voice?

Well!........ Neither!.

Enter the world of Affective Computing.

Much like my own parental advice and admonition, "Never reply or respond when angry", the computer sensed my mood and barred me from sending an ill-phrased e-mail. That is Affective Computing.

It refers to computers that can sense, interpret people's emotions and act either prophylactically or reactively. It uses the principles of artificial intelligence but the stimulus or input is the user's emotion. Digital Pets or Virtual Pets are early devices that "emoted". They became happy, sad, hungry or even died. Tamagotchi is a popular digital pet created in 1996.

Recognising emotions

How can computers recognise emotions?

In general, for a computer (machine) to recognise any quantity, the following are needed:

Perceivable effects produced by changes in the quantity

A sensor or transducer to record the changes and

Software to interpret the measurements.

Based on the interpretation, the computer can act.

In humans, each emotional state is defined by features or characteristics. By identifying or recording them, the affective state can be inferred. Emotions can cause changes in physiology (temperature, blood pressure, perspiration, pulse rate, galvanic skin response, etc), appearance (raising of eyebrows, twitching of lips, etc), behaviour (pace of walking, slamming the door, etc) or others (choice of words, etc.). Of these, physiological changes can be measured and appearance can be recorded with a camera. But, it is difficult to measure others.

How accurate are the measurements?

Today's technology allows precise measurements, but difficulties arise in interpreting them. If humans have difficulty in understanding others' emotions, can computers do so correctly?

Here is where neurofuzzy techniques (a combination of neural networks and fuzzy logic) comes into the picture. Fuzzy logic states that one can associate a feature with an emotion only with a certain level of probability. Hence, there will be probabilities for the two or more different states that a single sentic signal may represent.

With each interaction, the knowledge base (of sentic signals and the emotions they denote) increases and over time, the probability that a sentic signal denotes one emotion increases (is it not the same with humans also?). The neural networks assist the computer to learn by itself (self-learning).

Yet, there will be challenges in uniquely identifying the emotional state. How can happiness be distinguished from ecstasy? Or disgust from contempt? Also, how do we distinguish a passing facial expression indicative of an emotive state as an expression of true feeling? For example, twitching of the face momentarily does not indicate an emotion that warrants an action. It is a short transitory phase only.

The emotions that can be recognised have to be broadbased only and depend on the learning capability of the computer.

By collecting a lot of data, eventually a model can be developed that identifies meta-patterns of moods, describes the different affective states and the possible transition from one state to another.

Additionally, as the user experiences different emotions, the model can be "educated" to recognise them and possibly predict the next emotional state. For example, anger can be followed by verbal abuse or censure.

Each affective state should be identified by type of emotion, degree of intensity (angry or very angry), duration of that state, and frequency of occurrence of that state. A person's affect should be viewed in the context of his/her environment. For instance, if the person has not slept for a few days or is tense due to the looming deadline, then if the model can include this along with sentic signals, the recognition of the affect state will be realistic and meaningful.

The Media Laboratory in MIT is conducting pioneering work in Affective Computing. They have several projects involving machine learning, vocal and visible recognition, development of sensors, etc.

Convo is a non-commercial project in the UK. A software application recognises people's utterances and classifies them into nice, nasty or neutral. Accordingly, Ditto the donkey portrays an emotion. The objective is to train the model to recognise and respond emotionally to a human being's replies.

Components of affective computers

In a simplistic way, affective computers are a combination of feedback systems, central processors, fuzzy logic and neural networks. They can be thought of as several logical components. A sensing component senses the sentic signal. An identifying module identifies the signal and relates it to the emotional state. The cognition module processes these emotional states (identified periodically). The fuzzy logic assigns probabilities to the identified states.

With neural networks, the model "learns" continuously and updates itself. The action module acts according to the information received from the cognition module. The cognition model can not only receive data but also possibly send information to the identifier, to "watch out" for certain types of signals or to be alert about certain behavioural changes. From these data, a model of the emotional life of the user can be built.

How frequently should the sensor collect data? Should all sensors collect at the same frequency? Should instantaneous changes in emotional state be considered? What should be the minimum duration of an emotional state for it to influence behaviour? These are questions that are being researched.

Affective state impacts learning

It is proven beyond doubt that emotions and moods influence learning. Good teachers recognise and respond (or sense and react) appropriately to students' affective states which can be inferred from facial expressions, posture, conversational pace, and choice of words. In fact, this is the strength and value of small classrooms, gurukulas, and individual tutors. Since affective state influences motivation, a student who is motivated and affectively engaged will learn fast and well and perform other tasks also well. Cognition, motivation and emotion play an important role in learning.

Today, in e-learning and other self-learning environments, the information is presented in a sequential and structured manner. What if the "good human teacher" is emulated or simulated in the self-learning environ? That is the goal of the Auto-tutor project at University of Memphis, US. Autotutor is an "animated conversational agent with synthesised speech, gestures and facial expressions that display emotions". It questions, poses problems and engages the student in a "mixed-initiative dialogue" while he/she constructs an answer.

Confusion, frustration, anger, fear, joy or even Eureka type of experience are possible during learning.

For example, if a student seems puzzled or has paused without typing, it is an indication of confusion. If student raises eyebrows or shows facial features of being taken aback, then he/she is surprised. If he/she types fast and has eyes wide open, then he/she has understood. If after a pause, he/she suddenly types fast then he/she has moved from a state of confusion to one of intense interest (Eureka). If there are no emotions or distinctive facial features then he/she is neutral.

By using a video camera, the eye movements, facial expressions, raising of eyebrows, twitching of lips, etc, can be recorded. The supine posture can be measured using sensors on the seat and back parts of chair. Conversational cues can be discerned by analysing the recordings of student's responses. By discerning the "mood" of the learner, the pace and content can be dynamically changed to make the learning process interesting and effective.

Other areas of application

What else can affective computers/systems do?

Affective computers can play a role in a variety of spheres, handling from simple tasks to complicated ones and these can be either in response to the mood or in preventing the user from doing certain things.

For instance, if a person is depressed, the computer may "engage him/her in conversation" or "play lively music to boost the morale". If a person is driving a vehicle rashly, (due to anger or being drunk) it can shut down the engine.

Imagine a video camera capturing the face of a criminal and a computer identifying the intent and accordingly triggering off the alarm or locking the safe? Corporations can use affective computers to sense employees' emotional states and accordingly assign tasks or meet their needs to make them more productive. The marketing department identifies the emotions of people watching an advertisement (through facial expression recognition software built into the television) and accordingly changes the message.

One can sense the state of the partner and accordingly propose?

Of course, all these raise a lot of legal issues such as invasion into one's privacy, and government or organisation's right to monitor people's emotions. Affective computers can also play a critical role in helping the user communicate with oneself (Reflexive Communication). This will help people see themselves better, i.e. serve as an emotional mirror.

Some of these may sound far-fetched. But, then with the rate at which technology is progressing, far-fetched may not really be that far.

(The author is Consultant, Satyam Computer Services Ltd, and can be reached at Chellaiah_S@satyam.com)