An Improved Three-Dimensional Model for Emotion Based on Fuzzy Theory

Show more

1. Introduction

The Emotion Model uses mathematical ideas to formalize changes in emotions, carries out quantitative analysis of emotions, and then reflects the changing trends of people’s psychology and emotions. At present, the existing emotion models have one-dimensional, two-dimensional, three-dimensional, four-dimensional representations, including OCC models (emotion model constructed by Ortony, Clore and Colins), Russel’s PAD model (Pleasure-Displeasure, Arousal-Nonarousal, Dominance-Submissiveness) emotional model, Plutchik’s cone-shaped model etc. Based on these classic emotional models, a glittering array of scholars have analyzed, improved and studied emotional models from various aspects. M. Hima Bindu, Priya Gupta, and USTiwary proposed 22 basic emotions in the corresponding OCC model in the 3D PAD model. Finally, the expression recognition was classified by changes in the face, mouth, etc., from which we can find out the relation between them [1] ; Yang Liu, Chunting Yang, Jing Yu proposed a model of personality and emotion, considering the internal factors of the individual and the external factors of the environment [2] ; Li Zhenlong, Wang Xiaoxia applied fuzzy logic to the tension, anger, happiness, confidence of PAD model so as to get more detailed emotion information [3] ; Guey-Shya Chen, Min-Feng Lee proposed the emotional model applied to E-Learning, and came up with corresponding emotion classification process [4] ; Sun Fengfang, Liu Yuanning etc. proposed a new Personality_OCC emotional model, considering the influence of different personalities on emotional changes under the classical OCC structure model [5] ; T. Kishi, T. Kojima, N. Endo, M. Destephe et al. proposed a dynamic emotion model applied to robots, and analyzed the emotions collected by sensors to realize real-time recognition of emotions [6] ; in IEE At the E-meeting session, Hasan Murat Akinci and Engin Yesil proposed an emotional model based on fuzzy cognition, which makes the classification of emotions more detailed through fuzzy rules [7] .

By referring to the models mentioned above, this paper will mainly be focusing on the application background of the model, the influence of personal personality and external environment on emotions, and how to make emotions more detailed, aiming to make the model more in line with the actual situation as well as emotion and be more specific. However, the models proposed in the above literature are not intuitive and can not be quantitatively expressed. This paper proposes the three-dimension emotion model based on fuzzy theory can express the position relation of emotion intuitively and quantificationally. At the same time, the fuzzy theory is used to realize the division of each emotion in different degree.

2. Introduction to Basic Model of Emotion

2.1. Six Emotions Proposed by Ekman

Human beings express their emotions through facial expressions mainly by using special movement changes in various parts of the human face. Therefore, the premise of obtaining emotional information is that the emotional types need to correspond to facial expressions. By studying the different expressions of the faces of people from different countries, Ekman defines six types of emotions corresponding to the associated facial expressions: anger, disgust, fear, happiness, sadness, and surprise [8] . Table 1 shows the correspondence between the Ekman’s basic emotion type and the facial expression. Through these correspondences, the emotional information to be expressed can be obtained from the facial expression.

2.2. Three-Dimensional State-Space Emotion Model

Emotion has multiple dimensions, which can be deemed as a feature of emotion. Different emotions of people are in different positions in the dimensional space

Table 1. Emotions to facial expression proposed by Ekman.

and are gradual. What’s more, the relation of different emotions can be demonstrated based on their position and distance and the application of dimension theory to emotion model establishment conforms more to the emotional changes and relationships of human beings in actual situation.

On the basis of a great deal of different dimensional models constructed by different scholars, as shown in Figure 1, the three dimensions of the three-dimensional state sentiment model are fear, anger and joy, and each dimension of the emotion model only takes values of 0, 0.5, 1 indicating the degree of emotional strength in this dimension, so that the space contains 27 position coordinates, representing 27 emotions [9] .

As one’s emotions change, the corresponding emotional position in the three-dimensional state-space emotion model will move to another position. Similar to the above process, the process of emotional change can be referred to as the Markov state-transition matrix at various locations in this confined region [10] .

There are m kinds of emotion types, and each type of emotion can be divided into n levels, and the emotion space contains ${n}^{m}$ state. Let $l={n}^{m}$ the Markov probability transfer matrix of the l dimension will be the formula (1).

$P=\left[\begin{array}{cc}\begin{array}{ccc}{p}_{1,1}& {p}_{1,2}& \cdots \end{array}& {p}_{1,j}\\ \begin{array}{ccc}\begin{array}{c}{p}_{2,1}\\ \vdots \\ {p}_{i,1}\end{array}& \begin{array}{c}{p}_{2,2}\\ \vdots \\ {p}_{i,2}\end{array}& \begin{array}{c}\cdots \\ \ddots \\ \cdots \end{array}\end{array}& \begin{array}{c}{p}_{2,j}\\ \vdots \\ {p}_{i,j}\end{array}\end{array}\right]$ (1)

Figure 1. Three-dimensional state-space emotion model.

In formula (1), ${p}_{i,j}$ is probability, indicating the possibility of the i change to j emotion. ${p}_{i,j}$ will meet the following relationship:

${\sum}_{j=1}^{l}{p}_{i,j}}=1,\text{\hspace{0.17em}}\text{\hspace{0.17em}}\left(i\in \left[1,2,3,\cdots ,l\right]\right)$ (2)

The above process is a description of the probability that emotions change from one type to another. In this way, each emotion in the emotional model space can be divided into various degrees, and the possibility of emotional change is affected by the position and distance of coordinates corresponding to different emotions in three-dimensional space.

In order to evaluate and measure the expression of emotions in the emotional space, the mathematical tool of entropy can be used. For the sentiment model space containing l points and m dimensions expressed in formula (2), if the emotion is the i category at a certain time, the probability of the transition to other categories constitutes the probability vector of the emotional change:

${\stackrel{\dot{}}{e}}_{1}=\left({p}_{i,1},{p}_{i,2},\cdots ,{p}_{i,i},\cdots ,{p}_{i,l}\right)$ (3)

Emotional entropy is defined as:

${A}_{i}=-C{\displaystyle {\sum}_{i=1}^{l}{p}_{i}\mathrm{log}{p}_{i}}$ (4)

In formula (4), the emotional entropy of i state is ${A}_{i}$ so the probability that i emotional state changes to j emotional state is ${p}_{i,i}$ and C is constant.

The established emotional entropy shows and illustrates the possibility of emotional change. If the entropy value is large, it indicates that the emotional state transitions faster, which is reflected in the change of personality; if its value is too small, it indicates that the emotional state transition is slow. Based on the constructed sentiment model, the probability function of different types of emotions can be obtained, and then the corresponding entropy value can be calculated. The obtained entropy can be used to measure the different types of sentiment probability functions. In some respects, emotional entropy establishes a bond between macro and microcosmic emotional changes [3] [4] . Combined with real life, the Markov matrix has the following constraints:

1) The possibility that an emotional category in an emotional space is transferred to another category (including itself) decreases as the distance between the category and other categories increases.

2) If there is no interference from the outside world, the emotion gradually approaches the vicinity of the origin (calm point), that is, the vicinity of the origin is a stable point, and the emotion gradually becomes stable.

3) If there is a contradiction between emotions, the probability of simultaneous performance is extremely small. For example, the two emotions of happiness and fear have a very low probability of simultaneous performance.

3. Fuzzy Theory

In actual daily life, lots of things, laws, and phenomena are described as uncertain, transitional, and non-absolute. For example, the temperature, the degree of beauty, the degree of youthfulness, etc., cannot be accurately quantified by traditional mathematics description. In order to solve these uncertainties, non-absoluteness, and transitional problems, fuzzy theory is adopted [11] . Accurate models and methods can be used to describe, process, and study certain unclear boundaries, incomplete information, and inaccurate data.

Set a domain as U, for any $u\in U$ define a mapping to the closed interval [0, 1] as shown in equation (5):

${u}_{A}\left(u\right):U\to \left[0,1\right]$ (5)

In formula (5), A is a fuzzy set on the domain U, also called fuzzy subset, as the membership function A of the mapping shown in formula (5),
${u}_{A}\left(u\right)$ is the membership degree A of the pair

As can be seen from the above definition, the membership function ${u}_{A}\left(u\right)$ is a fuzzy measurement method, the size of u value indicates the extent of the degree of belonging, and its range is [0, 1]. If ${u}_{A}\left(u\right)$ tends to 1, it means that the degree of u belonging A is high, and if ${u}_{A}\left(u\right)$ is close to 0, it means that the degree of u belonging A is low. There are three ways to represent the membership function:

1) In general, it can be expressed as:

$A=\left\{\left(u,{u}_{A}\left(u\right)\right)|u\in U\right\}$ (6)

2) In the case that U is finite set, the fuzzy set

$A={\displaystyle {\sum}_{i}\frac{{u}_{A}\left({u}_{i}\right)}{{u}_{i}}}$ (7)

In the formula (7), ∑ on the right side represents a symbol, and does not represent the addition of numbers. Or it can be expressed as a vector form of Equation (8), in which case the order of the elements in each set has been determined:

$A=\left({u}_{A}\left({u}_{1}\right),{u}_{A}\left({u}_{2}\right),\cdots ,{u}_{A}\left({u}_{n}\right)\right)$ (8)

3) If the domain U is an infinite set, you can represent the fuzzy set A as:

$A={\displaystyle {\int}_{U}\frac{{u}_{A}\left(u\right)}{u}}$ (9)

The premise of establishing a fuzzy set is to construct the membership degree. So far, the methods of constructing the membership function include a variety of methods. The fuzzy statistics method and fuzzy distribution are introduced below.

The fuzzy statistical method to obtain the mapping function from the domain U to the closed interval [0, 1] mainly includes four elements: the domain U; one element ${u}_{0}$ in the domain; a variable set ${A}^{*}$ of boundaries, ${A}^{*}$ is corresponding to a fuzzy concept a and fuzzy set A; condition s. In the idea of fuzzy statistical method, the appearance of ambiguity is caused by the uncertainty of the division s of corresponding concepts a in statistics.

In the process of realizing the fuzzy statistical method, it is necessary to judge whether ${u}_{0}$ belongs to by multiple tests ${A}^{*}$ , the number of ${u}_{0}\in A$ in the subtest n is calculated, and the membership frequency of ${u}_{0}$ can be obtained.

$\text{Frequencyof}\text{\hspace{0.17em}}{u}_{0}\in A=\frac{\text{Numberof}\text{\hspace{0.17em}}{u}_{0}\in A}{n}$ (10)

When the number of times increases gradually or even infinity, the frequency gradually remains unchanged, and the membership degree ${u}_{0}$ belongs to A is:

${u}_{A}\left({u}_{0}\right)={\mathrm{lim}}_{n\to \infty}\frac{\text{Numberof}\text{\hspace{0.17em}}{u}_{0}\in A}{n}$ (11)

The fuzzy distribution is a mapping function from the domain U to the closed interval [0, 1]. The method uses experimental statistics to obtain the approximate distribution of membership degrees, and compares with some functions to select similar function distributions. Finally, the distribution function related parameters are obtained on the basis of experiments. Commonly used fuzzy distributions include parabolic distribution, normal distribution, rectangular distribution, ridge distribution, Cauchy distribution, and trapezoidal distribution. The commonly used constructor domain U to the closed interval [0, 1] mapping function also includes expert binary comparison sorting method, illustration method, empirical method and so on. The construction methods of the above various mapping functions are based on experiments and experience. The construction of the membership function is realized by the above method, which paves the way for the application of the fuzzy theory.

According to the mapping function determined by the above process, the “If...Then...” rule can be defined. In the rule, the “If...” part is a precondition for the rule, and the “Then...” part is a condition based conclusion. The rules of “If...Then...” consist of three processes:

・ Determine the membership value of each proposition in the “If...” section.

・ If the “If...” part consists of multiple parts, use the fuzzy operator to calculate the total membership value of “If...”.

・ Determine the degree to which the conclusion is true according to the preconditions of the rule.

4. Improved Three-Dimensional Emotion Model

In the application of human-computer interaction systems, it is very important to obtain emotional information through recognizing facial expressions. It is very important to construct a suitable emotional model first. The model not only makes different facial expressions correspond to different emotions, but also reflects the positional relationship of different emotions. In the basic emotion model, Ekman’s six basic emotions correspond to facial expressions and emotions, but they do not indicate the relationship between different emotions. However, the three-dimensional state space emotion model takes anger, joy and fear as the axis, which can not only intuitively and quantitatively describe the type of emotion, and also analyze the probability of emotional changes. Thus combined with the expression recognition system applied in the human-computer interaction system, it has the following shortcomings:

・ The types of emotions considered in some models cannot be obtained only by expression classification recognition.

・ The model does not consider how to extract detailed main emotional information from facial expressions.

Combined with the application background of human-computer interaction system, in order to further improve the emotion model, this paper improves the original three-dimensional state space emotion model in the following aspects:

・ In order to match emotions with facial expressions, six types of emotions are selected as emotion categories. Ekman maps basic emotions to related basic facial expressions and defines six basic emotions including anger, fear, sadness, happiness, surprise, and disgust.

・ In order to visually express the position and relationship of different emotions, the three-dimensional state space method emotion model is selected as the basic structure model, considering three basic emotions, namely fear, anger, joy, and certain emotions at a certain time. These three basic emotions are added by different weights.

・ Projecting the six types of emotions in the selected Ekman into the state space emotion model to intuitively and quickly obtaining the positional relationship between the recognized emotion and the expected emotion from the three-dimensional map based on the expression recognition.

・ According to the actual situation, we can know that each type of emotion has a certain range of variation. Therefore, in order to distinguish the intensity of each type of emotion, combined with the fuzzy theory, each type of emotion is defined to have a certain spherical range, and different areas in the range are expressed. The degree of strong, medium, or weak emotions of a certain kind of emotion is defined by fuzzy rules. The emotional model obtained by combining Ekman’s basic emotion, state space method emotion model and fuzzy theory is shown in Figure 2. The schematic diagram of each type of emotion degree is shown in Figure 3. In Figure 3, ${r}_{1},{r}_{2},{r}_{3}$ indicate the sphere radius of different degrees of emotion.

Let D show the Euclidean distance from the measured emotion to the standard emotion [12] , that is, the distance between the projection point corresponding to the tested emotion and the basic standard emotional center, as shown in formula (12):

$D=\Vert X-Y\Vert =\sqrt{{\displaystyle {\sum}_{i=1}^{n}{\left({x}_{i}-{y}_{i}\right)}^{2}}}$ (12)

In the formula, $X={\left({x}_{1},{x}_{2},\cdots ,{x}_{n}\right)}^{\text{T}}$ and $Y={\left({y}_{1},{y}_{2},\cdots ,{y}_{n}\right)}^{\text{T}}$ show emotion

Figure 2. Six emotion model of Ekman, Three-dimensional state-space emotion model, improved model based on fuzzy theory.

Figure 3. Three different divisions of a certain emotion.

vectors of the standard emotion of n dimension and tested emotion of n dimension. Define the membership function:

${u}_{A}\left(X\right)=\{\begin{array}{l}\left[0.00,0.35\right)\text{\hspace{0.17em}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}\left(0\le D\left(X\right)\le {r}_{1}\right)\\ \left[0.35,0.70\right)\text{\hspace{0.17em}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}\left({r}_{1}\le D\left(X\right)\le {r}_{2}\right)\\ \left[0.70,1.00\right)\text{\hspace{0.17em}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}\text{\hspace{0.17em}}\left(D\left(X\right)>{r}_{2}\right)\end{array}$ (13)

According to the definition of the membership function and the actual situation, when the membership degree of each emotion is set [0, 0.35), the degree of emotion is strong. When the membership degree is [0.35, 0.7), the degree of emotion is medium, and the degree of membership is [0.7, 1.0), the level of emotion is weak. In conjunction with Figure 3, define the “If...Then...” rule based on membership. For a certain kind of emotion:

・ If $0\le \left|D\right|\le {r}_{1}$ , then the degree of emotion is strong.

・ If ${r}_{1}<\left|D\right|\le {r}_{2}$ , then the degree of emotion is medium.

・ If ${r}_{2}<\left|D\right|$ , then the degree of emotion is weak.

The fuzzy rules defined above are organized into the form shown in Table 2.

Among them, each type of emotion is divided into different types, that is, each type of emotional degree is divided into six types of emotions ${r}_{i1},{r}_{i2}$ , $i=\left\{1,2,\cdots ,6\right\}$ , which need to be obtained through experiments. Through the definition of the above fuzzy rules, the quantitative division of the emotional level is realized, that is, the fine classification of the expression is quantitatively realized on the basis of the rough classification of the expression, which is more in line with the actual situation. In the improved emotion model, the one-to-one correspondence between emotion and facial expression is realized, and the advantages of intuitively quantitative analysis of emotional positional relationship and analysis of emotional change probability in the three-dimensional state space emotion model are fully utilized, and the basis of the emotional category is obtained. The definition of fuzzy rules on the above can make the emotional information more detailed. While applying the model to the human-computer interaction system we can help the computer to obtain the user’s current detailed emotional state and improve the human-computer interaction capability.

Through the establishment of the above model, the basic process of using the

Table 2. Definition of fuzzy rules.

model to perform facial expression recognition and obtain more detailed emotional information is also determined:

・ Obtaining a useful facial area by detecting a face.

・ Correspond different types of facial expression characteristics to emotion types, and perform rough classification recognition on facial expressions to obtain the main emotion types of facial expressions.

・ Understand the emotion division based on the fuzzy rules so as to get detailed information.

It can be seen from the above process that the model is not only used for the rough classification of emotions and facial expressions in the expression recognition process, but also the fuzzy rules in the model are used to implement the emotional classification of emotions.

5. Conclusion

This paper introduces a three-dimensional emotion model based on the improvement of fuzzy theory. Firstly, the two basic emotion models of Ekman’s six basic emotions and three-dimensional state space emotion model are introduced. Then the basic knowledge of fuzzy theory is introduced, including the definition and determination of membership function, the definition of fuzzy rules, etc. Finally, the merits as well as demerits of Ekman’s six basic emotion models and three-dimensional state space emotion models in the background of human-computer interaction system are analyzed. Moreover, an improved three dimensional model for emotion based on fuzzy theory is generated, which can map Ekman’s six emotions to three dimensional space-state model and realize a win-win situation. Furthermore, by defining the rules of fuzzy theory, we can make the emotion division more detailed and get more consistent with the actual situation. At the same time, the improved model can determine the basic process to get more detailed emotion information by using expression recognition.

Acknowledgements

This work was supported in part by a grant from the Characteristics innovation project of colleges and universities of Guangdong Province (Natural Science, No. 2016KTSCX182, 2016), a grant from the Youth Innovation Talent Project of colleges and universities of Guangdong Province (No. 2016KQNCX230, 2016).

References

[1] Zhang, L., Song, Y.Y., Liu, L. and Liu, J. (2016) Dissociable Roles of Internal Feelings and Face Recognition Ability in Facial Expression Decoding. NeuroImage, 132, 283-292.

https://doi.org/10.1016/j.neuroimage.2016.02.049

[2] Liu, Y., Yang, C. and Yu, J. (2009) Research on a Model of Emotion for Virtualagent. International Conference on Test and Measurement, ICTM’09, Hong Kong, 5-6 December 2009, 96-99.

[3] Li, Z.L. and Wang, X.X. (2009) Emotion Modeling of the Driver Based on Fuzzy Logic. IEEE Intelligent Transportation Systems Magazine, ITSC’09, St. Louis, MO, 4-7 October 2009, 1-5.

[4] Chen, G.S. and Lee, M.F. (2012) Detecting Emotion Model in e-Learning System. 2012 International Conference on Machine Learning and Cybernetics (ICMLC), Xian, 15-17 July 2012, 1686-1691.

https://doi.org/10.1109/ICMLC.2012.6359628

[5] Sun, F.F., Zhu, X.D., Liu, Y.L., Zhang, X.X., Zhang, W., Li, W.T. and Li, X. (2012) A New Method of Personality_OCC Affective Model. Journal of Jilin University (Science Edition), 50, 106-110.

[6] Kishi, T., Kojima, T. and Endo, N. (2013) Impression Survey of the Emotion Expression Humanoid Robot with Mental Model Based Dynamic Emotions. 2013 IEEE International Conference on Robotics and Automation, ICRA’13, Karlsruhe, 6-10 May 2013, 1663-1668.

https://doi.org/10.1109/ICRA.2013.6630793

[7] Akinci, H.M. and Yesil, E. (2013) Emotion Modeling Using Fuzzy Cognitive Maps. 2013 IEEE 14th International Symposium on Computational Intelligence and Informatics, Budapest, 19-21 November 2013, 49-55.

https://doi.org/10.1109/CINTI.2013.6705252

[8] Luo, B.H., Zeng, J.P., Duan, J.J. and Wu, C.G. (2015) Text Opinion Classification Method Based on Emotion Model. Computer Engineering, 41, 175-179.

[9] Zhang, X.Y., Zhang, T., Sun, Y., Zhang, W. and Chang, J. (2017) Emotional Speech Database Optimization and Quantitative Annotation Based on PAD Emotion Model. Journal of Taiyuan University of Technology, No. 5, 469-474.

[10] Zhang, H.P. and Chen, Q.H. (2018) Research on the Prediction of Network Public Opinion Based on Grey Markov Model. Information Science, No. 1, 75-79.

[11] Galina, I. (2018) Group Decision Analysis Algorithms with EDAS for Interval Fuzzy Sets. Cybernetics and Information Technologies, 18, 51-64.

https://doi.org/10.2478/cait-2018-0027

[12] Liu, W.F. and Li, B.L. (2016) Projectively Flat Finsler Metrics Defined by the Euclidean Metric and Related 1-Forms. Differential Geometry and Its Applications, No. 6, 14-24.

https://doi.org/10.1016/j.difgeo.2016.01.007