Using mobile crowdsourcing and geotagged social media data to study people’s affective responses to environments

When travelling in space, humans perceive the environment and evaluate it affectively. This chapter illustrates how mobile crowdsourcing and social media data can be used to study people’s affective responses to different environments. It also showcases how these affective responses can be used to provide a better understanding of human−environment interaction, as well as to enable smart geospatial applications (particularly navigation systems). This chapter also discusses some essential challenges that need further investigations when crowdsourcing people’s affective responses. Some of these challenges are participation motivating, data quality and privacy.


Introduction
When travelling in space, people perceive the environment and interpret it affectively. A lot of our daily behaviour and decision making is influenced by this kind of interpretation and affective evaluation of space (Kaplan & Kaplan 1989;Borst et al. 2009). For example, during wayfinding, which route to choose and which one to avoid may not only depend on distance, but also be influenced by people's affective evaluation of the environment, such as safety and attractiveness. Studying people's affective responses to environment contributes to a better understanding of people's spatial experiences and behaviour, as well as enables many applications, such as navigation systems and urban planning.
Conventionally, affect has been a central research topic within modern psychology. There were also many geographers, especially human geographers, approaching affective responses to environments when studying place. These conventional studies often employed empirical experiments in labs or in fields, which are often very expensive and time consuming, and hard to apply for largescale studies. In recent years, with the increasing availability of smartphones, researchers start to apply the principle of 'citizens as sensors' (Goodchild 2007), and explore the potential of crowdsourcing affective experiences from a large group of people. Furthermore, with the impetus of social networking services (e.g. Facebook, Foursquare, Flickr, and Twitter), large volumes of social media data have been continually created. These data, especially geotagged ones, contain lots of information about people's experiences and activities in various environments, which is a new and significant source for studying people's spatial experiences at different environments.
This chapter presents our on-going research efforts towards these trends. We illustrate how mobile crowdsourcing and social media data analysis can be used to collect people's affective responses to space, as well as the potential applications of these affective data.

Affective responses to environments
According to American Psychological Association (2006), affect is the experience of feeling or emotion. It is a key part of the process of an organism's interaction with stimuli (e.g. the environment). Russell (2003) argues that some affective response is always present within a person, as neutral, moderate or extreme.
There are many studies in literature focusing on structuring or modelling people's affective responses (Ekman and Friesen 1971, Russell 2003, Barrett 2006). The first group of studies tries to define basic distinct affective responses such as happiness, anger, fear, disgust or sadness (e.g., Ekman & Friesen 1971). This approach was widely applied in the field of ' Affect Computing' , which uses these basic affective responses to label facial expressions, and adapts services according to these labelled affective responses. Others adopt a dimensional approach to study people's affective responses, and argue that affective responses can be represented using some principal dimensions. For example, Russell and Barrett (1999) and Russell (2003) vary affective responses on the dimensions of valence (i.e. 'the subjective positive-to-negative evaluation ' , e.g. pleasant-unpleasant, comfortable-uncomfortable), and arousal (i.e. 'a sense of mobilization or energy' , e.g. activating-deactivating). There are other studies proposing to include one more dimension -motivational intensity (e.g. the strength of an urge to move toward or away from a stimulus) (Harmon-Jones et al. 2013). Barrett (2006) stated that valence is a basic component of affective responses (all people seem to be able to differentiate between pleasant and unpleasant affective states), while the degree of arousal may not be always experienced.
In this paper, we are particularly interested in affective responses evoked by environments. According to Russell (2003) and Barrett et al. (2007), environments are perceived not only according to their physical features, but also affectively. Some places are experienced as pleasing, while some others as disgusting and unsafe. These affective responses to environments are experienced as attributes or qualities of environments, which are commonly described with affect-related adjectives such as boring, safe, and beautiful.

Acquisition of affective responses
Affective responses to environments can be collected through various approaches, among which self reports and physiological recordings are the most prominent conventional methods for collecting such data. However, these approaches were traditionally used in laboratories with highly controlled conditions. Recently, with the rapid development of new technologies, novel methods and sensors have become available for collecting affective responses to environments (Huang et al. 2014;Resch et al. 2015). In the following, we highlight two of these current approaches.

Mobile crowdsourcing
Self-reports are the most direct way to gather information about people's affective responses (Barrett et al. 2007). Recently, the ubiquitous availability and use of smartphones, and the rapid spread of Web 2.0 potentially enable researchers to collect self-reported affective responses from a large group of people. In the EmoMap project (Klettner et al. 2013), we applied this mobile crowdsourcing approach to acquire people's affective responses via GPS-enabled smartphones.
To help people report their affective responses via smartphones, we developed an Affect-Space-Model. Based on the dimensional approach described above, the model adopts a two-level structure. Firstly, users are asked to rate their 'level of comfort' (i.e. the valence dimension) in their current environment on a 7-point Likert scale, from uncomfortable ('1') to comfortable ('7'). Secondly, users can optionally provide further details about their affective responses, particularly on the aspects of safety, attractiveness, diversity, and relaxation. These four aspects were obtained from an empirical study (Huang et al. 2014). In summary, each affective response to environments is collected as ratings on these five aspects (i.e. level of comfort, safety, attractiveness, diversity, and relaxation). Each affective response is then annotated with the GPS location obtained from user's smartphone.
The Affect-Space-Model was implemented in an Android mobile application ( Figure 1) to enable people to report their affective responses to environments anytime and anywhere. To help users locate themselves, this application shows the current location (obtained from GPS on smartphones) as a marker in an OpenStreetMap. Following the suggestions by Russell (2003), it particularly asks the question 'Here it is …' to explicitly make users focus on the environment. To better understand users' affective responses, some contextual information was also collected, i.e. company, familiarity with the place, time and weather (obtained from the Web). To encourage people's active contributions, this application was promoted to students, as well as to an urban walking community. 1 Users were asked to contribute their ratings anywhere and anytime they want in their daily life. Experiences within the EmoMap project show that with this mobile crowdsourcing approach, we are able to acquire affective data, which are evoked by realistic scenarios, directly from a large number of people. Until December 2013, more than 3,500 contributions were collected from more than 200 people. The contributions were distributed at different places of the world, with 98% of them for the city of Vienna (Austria). As an example, we visualize part of these contributions for the city of Vienna in Figure 2.

Extracting affective responses from social media data
Recently, the increasing availability of social media (e.g. Foursquare and Flickr) has led to the accumulation of large volumes of open social media data. Recent research has started to use social media data for studying people's affective responses (Mislove et al. 2010;Hauthal & Burghardt 2013). However, these approaches extract people's affective responses in various environments, which might not necessary be caused by or towards these environments. In the following, we extend existing research, and illustrate how social media data can be harnessed to extract people's affective responses to environments. Particularly, we focus on geotagged photos in Flickr.
For extracting affective responses from social media data, we apply sentiment analysis technique. Sentiment analysis (or opinion mining) is a natural language processing (NLP) technique, and aims to determine an author's attitudes, opinions or sentiments with respect to the topic written about. Different methods have been proposed for sentiment analysis, among which lexicon-based method is one of the most popular ones. Lexicon-based sentiment analysis employs NLP techniques to tokenize, split, and lemmatize text into a list words, and use word lexicons to determine the affective valence of these words as well as the valence of the text. Several affective word lexicons exist for this purpose, such as Affective Norms for English Words (ANEW) (Bradley & Lang 1999) and Finn Arup Nielsen's word list (AFINN) (Nielsen 2011). These word lexicons contain lists of English words with a valence score attached to each. For example, in AFINN, on a scale of -5 (negative, unpleasant) and +5 (positive, pleasant), "nice" is rated as +3, and terrible as -3. In this research, we use AFINN for the sentiment analysis, this is mainly because AFINN is particularly designed for microblogs and social media, and has been used by many other researchers (See http://neuro.imm. dtu.dk/wiki/AFINN for a list of studies using AFINN).
Due to the different natures of different languages, we only focus on Flickr photos with title/description in English language. To achieve this aim, we used an open language detection library provided by Cybozu Labs, Inc., which has a precision of 99% for 53 languages. 2 According to the observation of Kisilevich et al. (2010), the language pattern adjective−noun is the simplest and most popular example in English language to describe the characteristics of an object. For example, 'a beautiful place' and 'a dirty street' . Therefore, to extract affective responses to environments from geotagged photos in Flickr, we propose a lexicon-based sentiment analysis method, as depicted in Figure 3.
1) Firstly, for each geotagged photo, we use Stanford CoreNLP 1.3.4 library (http://stanfordnlp.github.io/CoreNLP/) to tokenize, split and lemmatise its title and description, and clean the sentences by removing English stop words (e.g. 'a' , 'by' and 'since'). A part-of-speech (POS) tagging process is also applied. Results of these steps are a list of words and their lexical category (e.g. noun, verb and adjective). 2) Secondly, we extract adjective−noun sets, e.g. 'interesting building' .
Again, Stanford CoreNLP library is applied. Results of this step is a list of adjective−noun sets. 3) Thirdly, we filter out adjective−noun sets that are not place-related. To achieve this aim, a list of place nouns is created, which consists of English place nouns (e.g. 'building' , 'street' , 'restaurant' , 'park' , 'museum' , 'opera' …) and study-area specific place-names from GeoNames (http://www. geonames.org/) (e.g. 'Stephansdom' , 'Karlskirche'). If the noun part of an adjective−noun set is not in the list, we consider the adjective−noun set as non-place-related. 4) Finally, for each adjective within the remaining adjective−noun sets, we check whether it is in the AFFIN affective lexicons. If yes, we assign the valence value to the photo. Otherwise, we use Java WordNet Library to get synonyms of the adjective, and check whether one of the synonyms is in the AFFIN affective lexicons. If yes, the valence value is also assigned to the photo. For each photo, we then average all the valence values of its title and description, and assign the result as the valence value of this photo.
This lexicon-based sentiment analysis method was applied to the Flickr photos uploaded for the city of Vienna (Austria) in the period of January 2007 and October 2011, which contain 107,353 data rows. Figure 4 shows the results. As can be seen from the results, different places are connected with different affective responses. However, to interpret these results, one should consider the bias brought by Flick, especially the contributing inequality. Several limitations of the proposed sentiment analysis method should be also pointed out. Firstly, it ignores slang and incomplete words, which often exist   in Flickr photo title or description. Secondly, this method uses a very simple language pattern (i.e. adjective−noun set) to identify affective responses caused by the environment. It should be improved to consider more language patterns. Furthermore, the method should be improved to check whether the geolocation of the photo actually matches the place the user is referring to in the title/ description of the phone. This will help to address the problem of location accuracy, as well as discover photos that are taken for remote environment objects ('pictures of the Eiffel tower might not be taken directly at the Eiffel tower').

Selected applications of affective responses
In general, collecting affective responses from a large number of people enables many innovative applications. They can not only be used to understand people's behaviour at different environments, as studied in environmental psychology and GIScience, but also bring benefits to other disciplines and application fields, such as Information and Communication Technology, urban planning, architecture and policy making. In the following, we illustrate two different case studies. The first case study used the collected affective data (see Section 3.1) to study the impact of environmental characteristics on people's affective responses. We focused on the area surrounding the Vienna University of Technology (Austria), mainly due to the diverse environmental settings within this area. We subdivided the area into three distinctive urban scenes according to their level of traffic and vegetation: A) green urban area (urban-green), B) urban area with light or no traffic (pedestrian lanes and one-lane street, urban-light traffic) and C) urban area with heavy traffic (roads ranging from two to three-lanes, urbanheavy traffic). These three urban scenes are compared according to the level of comfort ratings reported by the participants (Figure 5). The results suggest that the level of comfort ratings significantly differ between the three environmental settings (H(2)= 103.4, P < 0.001). For more details on the data analysis, refer to Klettner et al. (2013).
As can be seen from Figure 5, urban green areas show the most positive ratings among the three urban scenes, followed by areas of urban-light traffic. Urban areas with heavy traffic, on the other hand, show highly negative ratings. However, we argue that in order to draw a clearer conclusion, more research should be done on this aspect, e.g. a further classification of the study area, consideration of other contextual factors (such as time), validation of the quality of the affective data collected, and investigation of the bias caused by the 'participation inequality' (see Section 5 for more discussions).
Our second case study focused on how the collected affective data can be used to design smart human-centred geospatial applications, particularly navigation systems. Navigation systems aim to provide users with navigation guidance when visiting unfamiliar environments (Gartner et al. 2011). Route planning is a key component in navigation systems, and aims to compute an optimal route from an origin to a destination. Current route planning algorithms often fail to provide other routes aside from shortest routes or fastest routes. However, research in geography and environmental psychology has shown that humans may prefer many other different route qualities when choosing 'which way to take' , such as safety and attractiveness (Golledge 1995;Zacharias 2001).
We proposed that incorporating people's affective responses into route planning can help to provide routes with other qualities and characteristics. The basic idea is to aggregate affective ratings of similar users to model/approximate the current user's perception of different street segments. With this, a street network, in which each segment is encoded with a collective rating, can be generated. Based on this kind of street networks, we can compute routes with different qualities and characteristics, such as the most comfortable route (i.e. route avoiding uncomfortable areas) and the safest route. For more details on the algorithm, refer to Huang et al. (2014). Figure 6 shows an example of the route computed by considering the 'level of comfort' ratings, comparing to its shortest counterpart. An online empirical study with 64 human subjects was designed to evaluate the proposed routing Figure 6: The most comfortable route (green one, 478 meters) computed by considering people's affective responses (solely the 'level of comfort' ratings), comparing to its shortest counterpart (grey one, 442 meters). algorithm using a video task (participants were asked to choose the walk they prefer to take after watching the videos, which were taped along the Affec-tRoute and the shortest route) and two map tasks (given a start and an end, participants were asked to draw their preferred route). In the map task, a chisquare goodness-of-fit test showed that the routes computed by considering affective responses were significantly preferred over the shortest routes [P = 0.041]. Similar significant results were found in the map tasks. In conclusion, considering people's affective responses to environments leads to more satisfying routing results.
In the current research, we aggregated affective ratings from a large number of other users to provide route recommendations for the current user. However, it is still unclear how the routing results are influenced by the size/density of the affective ratings. More studies should be done on this aspect. On the other hand, the current experiment was implemented via an online questionnaire. Further research should be done to explore more realistic evaluation in more diverse environments.

Challenges and Lesson Learned
While mobile crowdsourcing and social media data are promising for studying people's affective responses to environments, and these affective responses enable many innovative applications, many challenges also exist. In the following, we highlight some of the essential challenges, and describe the lesson we learned from our current research, and discuss some hints for addressing these challenges.
Motivating participation. Motivating people to participate is key to crowdsourcing projects. This is even more important for crowdsourcing people's affective responses, which are subjective. Without enough contributions, it is impossible to get a comprehensive understanding about how environments are perceived by different groups of people. From our research, we learned that in order to motivate people's participation, it is essential to provide real or perceived 'benefits' for the contributors, and as well as to provide simple and intuitive ('easy-to-use') interface/tool to facilitate people's participation and contribution.
Data quality. Data quality is another big challenge when using mobile crowdsourcing and social media data to study people's affective responses. It is influenced by the 'participation inequality' of the contributors. Some of the examples of the aspect are: contributors might not be representative samples of the population; contributors might prefer to contribute at some places or some occasions/time, and avoid some other places or occasions/time. All these factors might strongly influence the quality of the data collected. Furthermore, compared to other data (such as those crowdsourced in OpenStreetMap), people's affective responses are subjective in nature, and no reference data are available for cross-checking. How to deal with the data quality of the crowdsourced affective data is still an open research question. A promising solution is to combine self-reports, social media data analysis and technical sensors (such as Galvanic skin response GSR, and Electrocardiogram ECG) when studying people's affective responses. With this, we can cross-check the data we collect.
Privacy. People's affective responses might contain lots of sensitive personal information. Therefore, when studying people's affective responses, especially at an individual level, it is important to develop methods to address the privacy concern. Anonymization technique, which was often employed in traditional empirical studies, might not work well for crowdsourcing projects and social media data analysis. For example, if we know an anonymous user often contributes at a particular place in the early morning (e.g. at 6:00) and at another place in the afternoon (e.g. at 14:00), we are potentially able to re-identify who the user is, as these two places might correspond to his/her home and office ). More research should be done to develop privacy-preserving techniques for mobile crowdsourcing and social media data analysis.

Summary and Outlook
The literature has shown that humans perceive and evaluate environments affectively, and these affective responses to environments influence our daily behaviour and decision-making in space. This paper presented our recent efforts towards these aspects. We illustrated two approaches to collect and acquire affective responses from a large number of people. Particularly, mobile crowdsourcing and social media data analysis were introduced. This paper also presented two different case studies to illustrate the potential applications of affective data.
Currently, we are exploring hybrid methods to study people's affective responses to environments, combining mobile crowdsourcing (self-reports), social media data analysis, and mobile sensing with sensors like Galvanic Skin Response. The privacy concerns of these affective contributions will also be addressed. We are also applying the affective data to other disciplines, such as urban planning and transportation.