I’m studying for my Computer Science class and don’t understand how to answer this. Can you help me study?
the concept of 3-F Method is introduced. Discuss the purpose of this concept and how it is calculated. Also perform your own research/analysis using these factors and provide your assessment on whether the United States need to introduce top talents in the field of big data and cloud computing by using bibliometrics.BIG IOT AND SOCIAL NETWORKING DATA FOR SMART
CITIES:
Algorithmic improvements on Big Data Analysis in the context of RADICAL city
applications
Evangelos Psomakelis12,Fotis Aisopos1, Antonios Litke1, Konstantinos Tserpes21, Magdalini
Kardara1 and Pablo Martínez Campo3
1Distributed
Knowledge and Media Systems Group, National Technical University of Athens, Zografou Campus, Athens,
Greece
2Informatics and Telematics Dept, Harokopio University of Athens, Greece
3Communications Engineering department, University of Cantabria, Santander, Spain
{fotais, litke, nkardara, tserpes, vpsomak}@mail.ntua.gr,pmartinez@tlmat.unican.es
Keywords:
Internet of Things, Social Networking, Big Data Aggregation and Analysis, Smart City applications,
Sentiment Analysis, Machine Learning
Abstract:
In this paper we present a SOA (Service Oriented Architecture)-based platform, enabling the retrieval and
analysis of big datasets stemming from social networking (SN) sites and Internet of Things (IoT) devices,
collected by smart city applications and socially-aware data aggregation services. A large set of city
applications in the areas of Participating Urbanism, Augmented Reality and Sound-Mapping throughout
participating cities is being applied, resulting into produced sets of millions of user-generated events and
online SN reports fed into the RADICAL platform. Moreover, we study the application of data analytics such
as sentiment analysis to the combined IoT and SN data saved into an SQL database, further investigating
algorithmic and configurations to minimize delays in dataset processing and results retrieval.
1
INTRODUCTION
Modern cities are increasingly turning towards
ICT technology for confronting pressures associated
with demographic changes, urbanization, climate
change (Romero Lankao, 2008) and globalization.
Therefore, most cities have undertaken significant
investments during the last decade in ICT
infrastructure including computers, broadband
connectivity and recently sensing infrastructures.
These infrastructures have empowered a number of
innovative services in areas such as participatory
sensing, urban logistics and ambient assisted living.
Such services have been extensively deployed in
several cities, thereby demonstrating the potential
benefits of ICT infrastructures for businesses and the
citizens themselves. During the last few years we
have also witnessed an explosion of sensor
deployments and social networking services, along
with the emergence of social networking (Conti et al.,
2011) and internet‐of‐things technologies (Perera et
al., 2013; Sundmaeker et al., 2010) Social and sensor
networks can be combined in order to offer a variety
of added‐value services for smart cities, as has
already been demonstrated by various early internet‐
of‐things applications (such as WikiCity(Calabrese et
al., 2007), CitySense(Murty et al., 2007),
GoogleLatitude(Page and Kobsa, 2010)), as well as
applications combining social and sensor networks
(as for example provided by (Breslin and Decker,
2007; Breslin et al., 2009) and (Miluzzo et al., 2007).
Recently, the benefits of social networking and
internet‐of‐things deployments for smart cities have
also been demonstrated in the context of a range of
EC co‐funded projects (Hernández-Muñoz et al.,
2011; Sanchez, 2010).
Current Smart City Data Analysis implies a wide
set of activities aiming to turn into actionable data the
outcome of complex analytics processes. This
analysis comprises among others: i) analysis of
thousands of traffic, pollution, weather, waste, energy
and event sensory data to provide better services to
the citizens, ii) event and incident analysis using near
real-time data collected by citizens and devices
sensors, iii) turning social media data related to city
issues into event and sentiment analysis , and many
others.
Combining
data
from
physical
(sensors/devices) and social sources (social
networks) can give more complete, complementary
data and contributes to better analysis and insights. In
overall, smart cities are complex social systems and
large scale data analytics can contribute into their
sustainability, efficient operation and welfare of the
citizens.
Motivated by the modern challenges in smart
cities, the RADICAL approach (RADICAL, 2016)
opens new horizons in the development, deployment
and operation of interoperable social networking and
Internet of Things services in smart cities, notably
services that could be flexibly and successfully
customized and replicated across multiple cities. Its
main goal is to provide the means for cities and SMEs
to rapidly develop, deploy, replicate, and evaluate a
diverse set of sustainable ICT services that leverage
established IoT and SN infrastructures. Application
services deployed and pilotedinvolve: i) Cycling
Safety Improvement, ii) Products Carbon Footprint
Management, iii) Object‐driven Data Journalism, iv)
Participatory Urbanism, v) Augmented Reality, vi)
Eco‐ consciousness, vii) Sound map of a city and viii)
City-R-Us: a crowdsourcing app for collecting
movement information using citizens smartphones.
The RADICAL platform is an open platform
having as added value the capability to easily
replicate the services in other smart cities, the ability
to co-design services with the involvement of cities’
Living Labs, and the use of added value services that
deal with the application development, the
sustainability analysis and the governance of the
services.
The RADICAL approach emphasizes on the
sustainability of the services deployed, targeting both
environmental sustainability and business viability.
Relevant indicators (e.g., CO2 emissions, Citizens
Satisfaction) are established and monitored as part of
the platform evaluation.End users (citizens) in
modern smart cities are increasingly looking for
media‐rich services offered under different space‐,
context‐, and situational conditions. The active
participation and interaction of citizens can be a key
enabler for successful and sustainable service
deployments in future cities. Social networks hold the
promise to boost such participation and interaction,
thereby boosting participatory connected governance
within the cities. However, in order to enable smart
cities get insight information on how citizens think,
act and talk about their city it is important to
understand their opinion and sentiment polarity on
issues related to their city context. This is where
sentiment analysis can play a significant role. As
social media data bring in significant Big Data
challenges (especially for unstructured data streams)
it will be important to find effective ways to analyse
sentimentally those data for extracting value
information and within specific time windows.
This paper has the following contributions:
Innovative smart city infrastructure for
uniform social and IoT big data aggregation
and combination.
Comperative study over Sentiment Analysis
techniques efficiency, to reduce record,
retrieval, update and processing time.
A novel technique for n-grams storage and
frequency representation in the context of big
data Sentiment Analysis.
The rest of the paper is structured as follows:
Section 2 gives an overview of related and similar
works that can be found in the international literature
and in projects funded by the European Commission.
Section 3 presents the RADICAL architecture and
approach. Section 4 presents details about the
Sentiment Analysis problem and related experiments,
while in section 5 we provide the future work to be
planned in the context of RADICAL and the
conclusions we have come into.
2
RELATED WORK
Recently, various analytical services such as
sentiment analysis found their way into Internet of
Things (IoT) applications. With the devices that are
able to convey human messages over the internet
meeting an exponential growth, the challenge now
revolves around big data issues. Traditional
approaches do not cope with the requirements posed
from applications for analytics in e.g. high velocity
rates or data volumes. As a result, the integration of
IoT with social sensor data put common tasks like
feature extraction, algorithm training or model
updating to the test.
Most of the algorithms are memory-resident and
assume a small data size (He et al., 2010) and once
this threshold is exceeded, the algorithms’ accuracy
and performance degrades to the point they are
useless. Therefore even if we focused solely on
volume challenges, it is intuitively expected that the
accuracy of the supervised algorithms will be
affected. An attempt from (Liu et al., 2013) to use
Naïve Bayes in an increasingly large data volume,
showed that a rapid fall of the algorithms accuracy is
followed by a continuous, smooth increase
asymptotically tending from the lower end to the
baseline (best accuracy under normal data load).
Rather than testing the algorithm’s limitations,
most of the other approaches are focusing on
implementing parallel and distributed versions of the
algorithms such as (He et al., 2010; Read et al., 2015).
In fact most of them rely on the Map-Reduce
framework so as to achieve high throughput
classification (Amati et al., 2014; Sakaki et al., 2013;
Wang et al., 2012; Zhao et al., 2012) whereas a
number of toolkits have been presented with
implementations of distributed or parallel versions of
machine learning algorithms such as (“Apache
Mahout: Scalable machine learning and data mining,”
n.d., “MEKA: A Multi-label Extension to WEKA,”
n.d.; Bifet et al., 2010). While these solutions put
most of the emphasis in the model and the
optimization of the classification task in terms of
accuracy and throughput, there is a rather small body
of research dealing with the problem of feature
extraction in high pace streams. The standard solution
that is considered is the use of a sliding window and
the application of standard feature extraction
techniques in this small set. In cases where the
stream’s distribution is variable, a sliding window
kappa-based measure has been proposed (Bifet and
Frank, 2010).
As reported in (Strohbach et al., 2015), another
domain of intense research in the area of scalable
analytics is for an architecture that combines both
batch and stream processing over social and IoT data
while at the same time considering a single model for
different types of documents (e.g. tweets Vs
blogposts). Sentiment analysis is a typical task that
requires batch modeling in order to generate the
golden standards for each of the classes. This process
is also the most computationally intense, as the
classification task itself is usually a CPU bound task
(i.e. run the classification function). In a data
streaming scenario the golden standards must be
updated in a batch mode, whereas the feature
extraction and classification must take place in real
time.
Perhaps the most prominent example of such an
architecture is the Lambda Architecture (Marz and
Warren, 2015) pattern which solves the problem of
computing arbitrary functions on arbitrary data in
realtime by combining a batch layer for processing
large scale historical data and a streaming layer for
processing items being retrieved in real time from an
input queue or analytics in e.g. high velocity rates or
data volumes. As a result, the integration of IoT with
social sensor data put common tasks like feature
extraction, algorithm training or model updating to
the test.Most of the algorithms are memory-resident
and assume a small data size (He et al., 2010) and
once this threshold is exceeded, the algorithms’
accuracy and performance degrades to the point they
are useless. Therefore even if we focused solely on
volume challenges, it is intuitively expected that the
accuracy of the supervised algorithms will be
affected. An attempt from Liu et al (Liu et al., 2013)
to use Naïve Bayes in an increasingly large data
volume, showed that a rapid fall of the algorithms
accuracy is followed by a continuous, smooth
increase asymptotically tending from the lower end to
the baseline (best accuracy under normal data load).
3. THE RADICAL APPROACH
The RADICAL platform integrates components
and tools from (SocIoS, 2013) and (SmartSantander,
2013) projects, in order to support innovative smart
city services, leveraging information stemming from
Social Networks (SN) and Internet of Things devices.
Using the aforementioned tools, it can collect,
combine, analyze, process, visualize and provide
uniform access to big datasets of Social Network
content (e.g. tweets) and Internet of Things
information (e.g. sensor measurements or citizen
smartphone reports).
The architecture of the RADICAL platform is
depicted in Figure 1. As can be observed, all IoT data
are pushed into the platform through the respective
Application Programming Interfaces (IoT API and
Repository API) and are forwarded to the RADICAL
Repository, comprised by a MySQL database, formed
based on the RADICAL Object Model. The devicerelated data, as dictated by this object model, are
saved in the form of Observations and Measurements.
Observations correspond to general IoT events
reported (e.g. a sensor report or bicycle “check-in”
event), while Measurements to more specific metrics
included in an Observation (e.g. Ozone
measurements (mpcc) or bicycle current speed
(km/h)). On the other hand, SN data are accessed in
real time from the underlying SN adaptors, by
communicating with the respective Networks’ APIs.
In cases of Social Networks like Foursquare that
provide plain venues and statistics, the adaptor-like
data structures do not make sense, thus relevant
Social Enablers are used to retrieve venue-related
information data.
On top of the main platform, RADICAL delivers
a set of tools (Application Management layer) that
allow end users to make better use of the RADICAL
platform, such as configuring the registered IoT
devices or extracting general activity statistics,
through the RADICAL Configuration API. Lastly,
the RADICAL Data API allows smart city services to
access the different sources of information (social
networks, IoT infrastructures, city applications),
combine data and perform data analysis by using the
appropriate platform tools.
Service Application Layer
Object Driven
Journalism
Participatory
Urbanism
Cycling
Safety
Augmented
Reality
Carbon
Footprint
EcoConciousness
Sound MAPP
Municipality
Cockpit
City R-US
Application Management Tools
Sustainability
services
Governance
Toolkit
Application
Development
Toolkit
Radical Configuration API
Radical Data API
Platform Tools
Configuration Tools
Resource
Directory
IoT Manager
Data Rep.
Configurator
Event Broker
RADICAL
Data
Repository
Sentiment
Analysis
Service
Storer
Event
Detection
Repository
Gateway
Register
Manager
Node
Manager
Service
aggregator
IOT API
GWFPSens
GW4IoT Devices
SocIoS Core Service
Repository API
…
SN Adaptors
GW4Serv
…
City
Applications
IoT devices
SN Enablers
Social Media
Social Networks Venues Services
Figure 1: RADICAL Platform Architecture
As can be seen in the Service Application Layer,
in the context of RADICAL a wide range of Smart
City services of various scopes has been developed:
Citizen journalism and Participatory
Urbanism: Those two interrelated services
allow citizens reporting events ofinterest in the
city, by posting images, text and metadata
through their smartphones.
Cycling Safety:Cyclists, acting as human
sensors can report the situation in the city
streets through their smartphones.
Monitoring the carbon footprint of
products, people and services: By using a
range of sensors, the CO2 emissions in specific
places in a city may be monitored.
Augmented Reality in Points of Interest
(POI): Tourists use their smartphones to
identify and receive information about points
of interest in a city.
Propagation of eco-consciousness:Leverages
on the viral effect in the propagation of
information in the social networks as well as
the recycling policy of a city, through
monitoring and reporting relevant actions on
citizens’ smartphones.
Social-Orientated Urban Noise Decibel
Measurement Application: Noise sensors are
employed throughout the city and citizens are
able to report and comment noise-related
information through SNs under a hashtag.
City Reporting application for the use of
Urban Services: This service gathers sensory
data along with SN check-ins in city venues, to
construct a traffic map throughout the city,
leveraging the process load of anycentralized
decision making process.
The aforementioned services are piloted in six
European participating cities: Aarhus, Athens,
Genoa, Issy les Moulineaux, Santander and the region
of Cantabria. Figure 2illustrates a screenshot example
of the RADICAL Cities’ Dashboard, where general
statistics on device registration and activity for a
service throughout different cities in a specific time
period is provided. In overall, during the last pilot
iteration, RADICAL Repository had captured a total
of 5.636 active IoT devices sending 728.253
Observations and 5.461.776 Measurements.
Most of the services above depend on the
aggregation of those IoTdata with social data
stemming from online Social Network sites. E.g. in
the Participatory Urbanism service, citizens’ reports
sent through smartphones and saved in RADICAL
Repository are combined with relevant tweets (under
a city service hashtag), as well as POI information
that can be collected from similar SNs.
Figure 2: RADICAL Cities Dashboard presents smartphone registrations and measurements for the AR service in the cities
of Santander and Cantabria over a period
Thus, given the size of the datasets acquired by
smart city services, along with the rich social media
content that can be retrieved through the RADICAL
platform adaptors, big data aggregation and analysis
challenges arise. Data Analysis tools are the ones that
further process the data in order to provide
meaningful results to the end user, i.e. Event
Detection or Sentiment Analysis.
When it comes to Big Data, as in the RADICAL
case, where millions of user-reported events are
aggregated along with millions of SN posts and an
extraction of general results is required, the challenge
accrued is two-fold: First, the tool must ensure the
accuracy of the analysis, in the sense that data
classification is correct to a certain and satisfactory
extent, and second, processing time must be kept
under certain limits, so that results retrieval process
delay is tolerable by end-users. Moreover, it is
apparent in such analysis that a trade-off between
effectiveness and efficiency exists. The latter is a
most crucial issue in Big Data analysis and apart from
the policy followed in data querying (e.g. for queries
preformed in an SQL database), it is also related to
the algorithmic techniques employed foranalyzing
those datasets.
In the context of this work, we focus on the
Sentiment Analysis on the big IoT and SN related
datasets of RADICAL, as this was the most popular
functionality among participating cities and almost all
of the RADICAL Smart City services presented
above make use of it. The goal of the Sentiment
Analysis service is to extract sentiment expressive
patterns from user-generated content in social
networks or IoT-originated text posts. The service
comes to the aid of the RADICAL city administrators,
helping them to categorize polarized posts, meaning
sentimentally charged text, e.g. analyse citizens’
posts to separate subjective from objective opinions
or count the overall positive and negative feedback,
concerning a specific topic or event in the city.
4. SENTIMENT ANALYSIS
EXPERIMENTS AND
PERFORMANCE IMPROVEMENT
4.1
Introduction
The term Sentiment Analysis refers to an
automatic classification problem. Its techniques are
trying to distinguish between sentences of natural
language conveying positive (e.g. happiness, pride,
joy), negative (e.g. anger, sadness, jealously) or even
neutral (no sentiment texts like statements, news,
reports)emotion (called sentiment for our purposes)
(Pang et al., 2002).
A human being is capable of understanding a
great variety of emotions from textual data. This
process of understanding is based on complicated
learning procedures that we all go through while
using our language as a means of communication, be
it actively or passively. It requires imagination and
subjectivity in order to fully understand the meaning
and hidden connections of each word in a sentence,
two things that machines lack.
The most common practice is to extract numerical
features out of the natural language (Godbole et al.,
2007). This process translates this complex means of
communication into something the machine can
process.
4.2
Natural Language Processing
In order to process the natural language data, the
computer has to take some pre-processing actions.
These actions include the cleansing of irrelevant,
erroneous or redundant data and the transformation of
the remaining data in a form more easily processed.
Cleansing the data has become a subjective task,
depending on the purposes of each researcher and the
chosen machine learning algorithms. The
transformation of the sentences in another form now
is clearly studied and each approach has some
advantages and disadvantages. This paper will detail
three approaches, two widely used and one that had
some success in improving the accuracy of the
algorithms: the bag of word, N-Gramsand N-Gram
Graphs(Aisopos et al., 2012; Fan and Khademi, 2014;
Giannakopoulos et al., 2008; Pang and Lee, 2008).
The bag of words approach is perhaps the most
simple and common one. It regards each sentence as
a set of words, disregarding their grammatical
connections and neighbouring relations. It splits each
sentence based on the space character (in most
languages) and then forms a set of unrelated words (a
bag of words as it is commonly called). Then each
word in this bag can be disregarded or rated by a
numerical value, in order to create a set of numbers
instead of words.
The N-Grams are a bit more complex. They also
form a bag of words but now each sentence is split
into pseudo-words of equal length. A sliding window
of N characters is rolling on the sentence creating this
bag of pseudo-words. For example if N=3 the
sentence “This is a nice weather we have today!” will
be split in the bag {‘Thi’, ‘his’, ‘is ’, ‘s i’, ‘ is’, ‘is ’,
‘s a’, ‘ a ’, ‘a n’, ‘ ni’, ‘nic’, ‘ice’, ‘ce ’, ‘e w’, ‘ we’,
‘wea’, ‘eat’, ‘ath’, ‘the’, ‘her’, ‘er ’, ‘r w’, ‘ we’, ‘we
’, ‘e h’, ‘ ha’, ‘hav’, ‘ave’, ‘ve ’, ‘e t’, ‘ to’, ‘tod’,
‘oda’, ‘day’, ‘ay!’}.
This technique takes into regard the direct
neighbouring relations by creating a continuous
stream of words, it still ignores the indirect relations
between words and even the relations between the
produced N-Grams. Of course it is impossible to have
a predefined set of numerical ratings for each one of
these pseudo-words because each sentence and each
N number (which is defined arbitrarily by the
researcher) produces a different set of pseudowords(Psomakelis et al., 2014). So machine learning
is commonly used to replace these words with
numerical values and create sets of numbers which
can be aggregated to sentence level.
An improvement on that approach aims to take
into consideration the neighbouring relations between
the produced N-Grams. This approach is called NGram Graphs and its main concept is to create a graph
connecting each N-Gram with its neighbours in the
original sentence. So each node in this graph is an NGram and each edge is a neighbouring
relation(Giannakopoulos et al., 2008). This approach
gives a variety of new informationto the researchers
and to the machine learning algorithms, including
information about the context of words, making it a
clear improvement of the simple N-Grams(Aisopos et
al., 2012). The only drawbacks are the complexity it
adds to the process and the difficulties of storing,
accessing and updating a graph of textual data.
4.3
Dataset Improvements
At the core of sentiment analysis is its dataset. We
are gathering and employing bigger and bigger
datasets in order to better train the algorithms to
distinguish what is positive and what is negative.
Classic storage techniques are proving more and more
cumbersome for large datasets. ArrayLists and most
Collections are adding a big overhead to the data so
they are not only enlarging the space requirements for
its storage but they are also delaying the analysis
process. So new techniques for data storage and
retrieval are needed, techniques that will enable us to
store even bigger datasets and access them with even
smaller delays.
The most commonly used such technique is the
Hash List(Fan and Khademi, 2014), which first
hashes the data in a certain, predefined amount of
buckets and then creates a List in each bucket to
resolve any collisions. This method’s performance is
heavily dependent on the quality of the hash function
and its ability to equally split the data into the buckets.
The target is to have as small lists as possible. That is
the case because finding the right bucket for a certain
piece of data is done in O(1) time but looking through
the List in that bucket for the correct spot to store the
piece of data is done in O(n) time where n is the
number of data pieces in the List.
Moreover, in Java which is the programming
language that we are using, each List is an object
containing one object for each data piece. All these
objects create an overhead that is not to be ignored. In
detail the estimated size that a hash list will occupy is
calculated as:
12 + ((B − E) ∗ 12) + (E ∗ 4)
+ (U ∗ (N ∗ 2 + 72))
Equation 1: Size estimation of Hash List where N=NGram
Length, U=Unique NGrams, B=Bucket Size, E=Empty
Buckets.
The worst case for storage but best for access time
is when almost each data piece has its own bucket. In
this case, for N=5, S=11881376, U=S, B=(26^N)*2,
E=11914220, we have a storage size of 1110 MB. The
best case for storage but worse for access time is when
all data pieces are in a small number of buckets, in big
lists. In this case for N=5, S=11881376, U=1,
B=(26^N)/2, E=200610 we have a storage size of 23
MB. In an average case of N=5, S=11881376,
U=7510766, B=26^N, E=2679046 we have 682 MB
of storage space needed. The sample for the above
examples was the complete range of 5-Grams for the
26 lowercase English characters which are 26^5 =
11881376.
Our proposed technique now, the one that we call
Dimensional Mapping, has a standard storage space,
depending only on the length of the N-Grams. The
idea is to store only the weight of each N-Gram with
the N-Gram itself being the pointer to where it is
stored. That is achieved by creating an N-dimensional
array of integers where each character of the N-Gram
is used as an index. So, in order to access the weight
of the 5-Gram ‘fdsgh’ in the table DM we would just
read the value in cell DM[‘f’][‘d’][‘s’][‘g’][‘h’]. A
very simple mapping is used between the characters
and an integers: after a very strict cleansing process
where we convert all characters in lowercase and
discard all characters but the 26 in the English
alphabet, we are just subtracting the ASCII value of
‘a’. Due to the serial nature of the characters that
gives us an integer between 0 and 26 that we can use
as an index. A more complex mapping can be used in
order to include more characters or even punctuation
that we now ignore.
The Dimensional Mapping has a standard storage
size requirement, dependent only on the length of the
N-Grams as we mentioned before. The size it
occupies can be estimated by the following formula:
=
(26 )
∗ 4 + ∑ ((26 − ) ∗ 12)
=1
Equation 2: Dimensional Mapping size estimation with N
being the length of N-Grams.
This may seem large but for the 5-Grams the
estimated size is just 51 MB. Compared to the worst
case of Hash Lists (1110 MB) or even the average
case (682 MB) it seems like a huge improvement.
This is caused due to the fact that the
multidimensional array stores primitive values and
not objects, which reduces the overhead greatly.
Moreover, we can now say that accessing and
updating a certain data piece can be done in O(1) time
with absolute certainty, with no dependency on the
data itself or a hash function. This had significant
results in speeding up the execution times of the
analysis, enabling us to look into streaming data and
semi-supervised machine learning algorithms.
4.4
Results
We measured three main KPIs for the result
comparison. Two of them (success ratio, kappa
variable) were measuring the success ratio of
classification and one (execution time) the
algorithmic improvement. We present them bellow.
We run experiments on 5-Grams stored in classic
ArrayList format, in Hash Lists and in Dimensional
Mapping. After storing the N-Grams in these formats
we applied a 10-fold cross validation on each one of
the seven machine learning algorithms we chose:
Naïve Bayesian Networks, C4.5, Support Vector
Machines,
Logistic
Regression,
Multilayer
Perceptrons, Best-First Trees and Functional Trees.
Then we recorded the three KPIs for each one of these
21 experiments. The results for the first two KPIs are
shown in the bar chart that follows. In the same chart
we have included the KPIs for a threshold based
classification, using an arbitrarily set threshold.
5. CONCLUSIONS
RADICAL platform, as presented in the current
work, successfully combines citizens’ posts retrieved
through smartphone applications and Social
Networks in the context of smart city applications, to
produce a testbed for applying multiple analysis
functionalities and techniques. The exploitation of
resulting big aggregated datasets pose multiple
challenges, with timely-efficient analysis being the
most important. Focusing on data storage and
representation, multiple techniques were examined in
the experiments performed, in order to come up with
the optimal algorithmic approach of Dimensional
Mapping. In the future the authors plan to use even
larger and more complex datasets, further leveraging
on the effectiveness of these social networking
services.
Figure 3: A comparison of the three KPIs as shown in the
sentiment analysis experiments
ACKNOWNLEDGEMENTS
As of the execution times the following table
contains a summary of the results:
This work has been supported by RADICAL and
Consensus projects and has been funded by the
European Commission’s Competitiveness and
Innovation Framework Programme under grant
agreements no 325138 and 611688 respectively.
Table 1: Execution time in seconds summary – comparing
for the various algorithms and techniques
ArrayLists
Hash
List
Dimensional
Mapping
Thresholds
1691
5
4
Naïve
Bayes
12302
7
7
C4.5
21535
9
8
SVM
20662
147
177
22251
9
11
MLP
21224
41
48
BFTree
23319
25
19
FTree
22539
16
16
Logistic
Regression
REFERENCES
Aisopos, F., 0001, G.P., Tserpes, K., Varvarigou,
T.A., 2012. Content vs. context for
sentiment analysis: a comparative analysis
over microblogs., in: Munson, E.V.,
Strohmaier, M. (Eds.), HT. ACM, pp. 187–
196.
Amati, G., Angelini, S., Bianchi, M., Costantini, L.,
Marcone, G., 2014. A scalable approach to
near real-time sentiment analysis on social
networks. Inf. Filter. Retr. 12.
Apache Mahout: Scalable machine learning and data
mining [WWW Document], n.d. URL
http://mahout.apache.org/ (accessed 4.8.15).
Bifet, A., Frank, E., 2010. Sentiment Knowledge
Discovery in Twitter Streaming Data, in:
Proceedings of the 13th International
Conference on Discovery Science, DS’10.
Springer-Verlag, Berlin, Heidelberg, pp. 1–
15.
Bifet, A., Holmes, G., Kirkby, R., Pfahringer, B.,
2010. MOA: Massive Online Analysis. J.
Mach. Learn. Res. 11, 1601–1604.
Breslin, J., Decker, S., 2007. The Future of Social
Networks on the Internet: The Need for
Semantics. IEEE Internet Comput. 11, 86–
90. doi:10.1109/MIC.2007.138
Breslin, J.G., Decker, S., Hauswirth, M., Hynes, G.,
Phuoc, D.L., Passant, A., Polleres, A.,
Rabsch, C., Reynolds, V., 2009. Integrating
Social Networks and Sensor Networks, in:
Proceedings on the W3C Workshop on the
Future of Social Networking.
Calabrese, F., Kloeckl, K., Ratti, C., 2007. Wikicity:
Real-time location-sensitive tools for the
city IEEE Pervasive Computing, 390–413.
Conti, M., Passarella, A., Pezzoni, F., 2011. A model
for the generation of social network graphs.,
in: WOWMOM. IEEE Computer Society,
pp. 1–6.
Fan, M., Khademi, M., 2014. Predicting a Business
Star in Yelp from Its Reviews Text Alone.
CoRR abs/1401.0864.
Giannakopoulos, G., Karkaletsis, V., Vouros, G.A.,
Stamatopoulos, P., 2008. Summarization
system evaluation revisited: N-gram graphs.
TSLP 5.
Godbole, N., Srinivasaiah, M., Skiena, S., 2007.
Large-Scale Sentiment Analysis for News
and Blogs, in: Proceedings of the
International Conference on Weblogs and
Social Media (ICWSM).
GrowSmarter, 2016. Grow Smarter project [WWW
Document].
URL
http://www.growsmarter.eu/home/
Gubbi, J., Buyya, R., Marusic, S., Palaniswami, M.,
2012. Internet of Things (IoT): A Vision,
Architectural Elements, and Future
Directions. CoRR abs/1207.0203.
He, Q., Zhuang, F., Li, J., Shi, Z., 2010. Parallel
Implementation
of
Classification
Algorithms Based on MapReduce, in: Yu, J.,
Greco, S., Lingras, P., Wang, G., Skowron,
A. (Eds.), Rough Set and Knowledge
Technology, Lecture Notes in Computer
Science. Springer Berlin Heidelberg, pp.
655–662.
Hernández-Muñoz, J.M., Vercher, J.B., Muñoz, L.,
Galache, J.A., Presser, M., Gómez, L.A.H.,
Pettersson, J., 2011. Smart Cities at the
Forefront of the Future Internet., in:
Domingue, J., Galis, A., Gavras, A.,
Zahariadis, T.B., Lambert, D., Cleary, F.,
Daras, P., Krco, S., Müller, H., Li, M.-S.,
Schaffers, H., Lotz, V., Alvarez, F., Stiller,
B., Karnouskos, S., Avessta, S., Nilsson, M.
(Eds.), Future Internet Assembly, Lecture
Notes in Computer Science. Springer, pp.
447–462.
Liu, B., Blasch, E., Chen, Y., Shen, D., Chen, G.,
2013. Scalable sentiment classification for
Big Data analysis using Na #x00EF;ve
Bayes Classifier, in: Big Data, 2013 IEEE
International Conference on. pp. 99–104.
doi:10.1109/BigData.2013.6691740
Marz, N., Warren, J., 2015. Big Data: Principles and
best practices of scalable realtime data
systems, 1 edition. ed. Manning
Publications, Westampton.
MEKA: A Multi-label Extension to WEKA [WWW
Document],
n.d.
URL
http://meka.sourceforge.net/
(accessed
3.27.15).
Miluzzo, E., Lane, N.D., Eisenman, S.B., Campbell,
A.T., 2007. CenceMe – Injecting Sensing
Presence
into
Social
Networking
Applications., in: Kortuem, G., Finney, J.,
Lea, R., Sundramoorthy, V. (Eds.),
EuroSSC, Lecture Notes in Computer
Science. Springer, pp. 1–28.
Murty, R., Gosain, A., Tierney, M., Brody, A., Fahad,
A., Bers, J., Welsh, M., 2007. Harvard
University Technical Report. TR‐13‐07.
Page, X.W., Kobsa, A., 2010. Navigating the social
terrain with google latitude. iConference
Urbana‐Champaign, p.174‐178.
Pang, B., Lee, L., 2008. Opinion mining and
sentiment analysis. Found. Trends Inf. Retr.
2, 1–135.
Pang, B., Lee, L., Vaithyanathan, S., 2002. Thumbs
up? Sentiment Classification Using Machine
Learning Techniques, in: emnlp2002.
Philadelphia, Pennsylvania, pp. 79–86.
Perera, C., Zaslavsky, A.B., Christen, P.,
Georgakopoulos, D., 2013. Sensing as a
Service Model for Smart Cities Supported
by Internet of Things. CoRR abs/1307.8198.
Psomakelis, E., Tserpes, K., Anagnostopoulos, D.,
Theodora, V., 2014. Comparing Methods for
Twitter Sentiment Analysis. Proc. 6th Int.
Conf. Knowl. Discov. Inf. Retr. – KDIR
2014 225–232.
RADICAL, 2016. Rapid Deployment for Intelligent
Cities and Living, FP7 EU funded Research
project
[WWW
Document].
URL
http://www.radical-project.eu
Read, J., Martino, L., Olmos, P.M., Luengo, D., 2015.
Scalable multi-output label prediction: From
classifier chains to classifier trellises.
Pattern
Recognit.
48,
2096–2109.
doi:10.1016/j.patcog.2015.01.004
Romero Lankao, P., 2008. Urban areas and climate
change: Review of current issues and trends.
Issues Pap. 2011 Glob. Rep. Hum. Settl.
Sakaki, T., Okazaki, M., Matsuo, Y., 2013. Tweet
analysis for real-time event detection and
earthquake reporting system development.
Knowl. Data Eng. IEEE Trans. On 25, 919–
931.
Sanchez, L., 2010. SmartSantander: Experimenting
The Future Internet in the City of the Future.
Presented at the PIMRC2010, Istanbul,
Turkey.
SmartSantander, 2013. SmartSantander, FP7 EU
funded
Research
project
[WWW
Document].
URL
http://www.smartsantander.eu/
SocIoS, 2013. Exploiting Social Networks for
Building the Future Internet of Services, FP7
EU funded Research project [WWW
Document].
URL
http://www.sociosproject.eu/
Strohbach, M., Ziekow, H., Gazis, V., Akiva, N.,
2015. Towards a big data analytics
framework for IoT and smart city
applications, in: Modeling and Processing
for
Next-Generation
Big-Data
Technologies. Springer, pp. 257–282.
Sundmaeker, H., Guillemin, P., Friess, P., Woelfflé,
S. (Eds.), 2010. Vision and Challenges for
Realising the Internet of Things.
Publications Office of the European Union,
Luxembourg.
Wang, H., Can, D., Kazemzadeh, A., Bar, F.,
Narayanan, S., 2012. A System for Realtime Twitter Sentiment Analysis of 2012
U.S. Presidential Election Cycle., in: ACL
(System Demonstrations). The Association
for Computer Linguistics, pp. 115–120.
Zhao, J., Dong, L., Wu, J., Xu, K., 2012. MoodLens:
an emoticon-based sentiment analysis
system for chinese tweets., in: 0001, Q.Y.,
Agarwal, D., Pei, J. (Eds.), KDD. ACM, pp.
1528–1531.
2017 Proceedings of PICMET ’17: Technology Management for Interconnected World
Analysis on the Demand of Top Talent Introduction
in Big Data and Cloud Computing Field in China
Based on 3-F Method
Zhao Linjia, Huang Yuanxi, Wang Yinqiu, Liu Jia
National Academy of Innovation Strategy, China Association for Science and Technology, Beijing, P.R.China
Abstract—Big data and cloud computing, which can help
China to implement innovation-driven development strategy and
promote industrial transformation and upgrading, is a new and
emerging industrial field in China. Educated, productive and
healthy workforces are necessary factor to develop big data and
cloud computing industry, especially top talents are essential.
Therefore, a three-step method named 3-F has been introduced
to help describing the distribution of top talents globally and
making decision whether they are needed in China. The 3-F
method relies on calculating the brain gain index to analysis the
top talent introduction demand of a country. Firstly, Focus on the
high-frequency keywords of a specific field by retrieving the
highly cited papers. Secondly, using those keywords to Find out
the top talents of this specific field in the Web of Science. Finally,
Figure out the brain gain index to estimate whether a country
need to introduce top talents of a specific field abroad. The result
showed that the brain gain index value of China’s big data and
cloud computing field was 2.61, which means China need to
introduce top talents abroad. Besides P. R. China, those top
talents mainly distributed in the United States, the United
Kingdom, Germany, Netherlands and France.
I. INTRODUCTION
Big data and cloud computing is a new and emerging
industrial field[1], and increasing widely used in China[2-4].
Talents’ experience is a source of technological mastery[5],
essentially for developing and using big data technologies.
Most European states consider the immigration of foreign
workers as an important factor to decelerate the decline of
national workforces[6]. Lots of universities and research
institutes have set up undergraduate and/or postgraduate
courses on data analytics for cultivating talents[7]. EMC
corporation think that vision, talent, and technology are
necessary elements to providing solutions to big data
management and analysis, insuring the big data success[8].
for top talent introduction demand analysis will be discussed.
In section 3, we will analysis the demand of top talent
introduction in big data and cloud computing field in China.
II. METHOD
In general, metering indicators contain the most productive
authors, journals, institutions, and countries, and the
collaboration networks between authors and institutions[11,
12]. Based on the commonly used bibliometrics method, 3-F
method for top talent introduction demand analysis is proposed.
3-F method has three steps:
Firstly, searching the literature database and forming a
high-impact literature collection in a specific field. Focusing on
the high-frequency keywords in the high-impact literature
collection by using the text analysis method as the research
hotspots. Just to be clear, the high-impact literature refers to the
journal literature whose number of cited papers ranked in the
top 1% in the same discipline and in the same year.
Secondly, retrieving those keywords in the Web of Science
to find out where those top talents of this specific field are.
Find the top talents by collected the information about talents’
country distribution, the institutions distribution and so on
through the high-impact literature collection. Among them, the
top talent refers to the first author or the communication author
of the high-impact literatures.
Finlly, Figure out the brain gain index to determine the top
talents introduction demand of a certain country. The brain gain
index is calculated as following formulas:
Bibliometrics research has appeared as early as 1917[9],
and has been proved an effective method for assessing or
identifying talents. Based on analyses of publication volume,
journals and their impact factors, most cited articles and
authors, preferred methods, and represented countries,
Gallardo-Gallardo et. al[10] assess whether talent management
should be approached as an embryonic, growth, or mature
phenomenon.
Iik = (Twk / Tik) / (Pw / Pi)
(1)
Among them, Iik means the brain gain index value of
country (i) in the field (k), Twk means the number of world’s top
talents in the field (k), Tik means the number of country’s (i) top
talents in the field (k), Pw means the world population, Pi
means the country’s (i) population. If Iik was more than 1, that
means the country (i) has less top talents in the field (k),
therefore the talent introduction demand will be relatively
strong. In contrast, if Iik was less than 1, that means the
country’s (i) has greater top talents in the field (k) than the
world average, and the talent introduction demand will not be
so strong.
In this paper, we intend to analysis whether China need to
introduce top talents in the field of big data and cloud
computing by using bibliometrics. In section 2, the 3-F method
Additionally, the literature information mainly from the ISI
Web of Science (SCI, CPCI-S), and the the data analysis and
visualization tools are TDA and Tableau.
978-1-890843-36-6 ©2017 PICMET
2017 Proceedings of PICMET ’17: Technology Management for Interconnected World
TABLE II.
III. CASE STUDY
DATA AND CLOUD COMPUTING FIELD
Using 3-F method to analysis the top talents introduction
demand in the big data and cloud computing field. We
collected the high-impact literatures from January 1, 2006 to
July 31, 2016. The literature Language was English and the
literature type was article. Combining with the above
conditions, we got 546 high-impact literatures in the big data
and cloud computing field. Then the high-frequency keywords
have been obtained (Table 1) and served as the research
hotspots set.
TABLE I.
THE RESEARCH HOTSPOTS OF THE HIGH-IMPACT LITERATURES IN
BIG DATA AND CLOUD COMPUTING FIELD
Order
Keywords
THE NATIONALITY DISTRIBUTION OF TOP TALENTS IN THE BIG
Frequency
Order
Number of the top
talent
268
Country or Region
1
US
2
P. R. China
48
3
UK
47
4
Germany
39
5
Netherlands
28
6
France
27
7
Canada
22
8
Australia
21
9
Italy
19
10
Switzerland
13
Spain
13
12
Japan
10
13
Korea
8
Malaysia
8
15
Singapore
7
New Zealand
7
17
Austria
6
18
Belgium
5
1
cloud computing
48
2
big data
24
3
virtualization
11
4
cloud manufacturing
9
5
internet of things (IoT)
8
6
mobile cloud computing
8
7
bioinformatics
6
8
climate change
6
Sweden
5
9
Hadoop
6
India
5
10
software-defined networking (SDN)
6
Chinese Taipei
5
……
……
TABLE III. THE INSTITUTES DISTRIBUTION OF TOP TALENTS IN THE BIG
At the same time, we displayed the frequency distribution
of research hotspots in the way of cloud chart(fig. 1).
DATA AND CLOUD COMPUTING FIELD
Order
Country or Region
Number of the top talent
1
Harvard University (US)
10
2
Purdue University (US)
7
University of Malaya (Malaysia)
7
University of Maryland (US)
7
Unversity of Melbourne (Australia)
7
University of Missouri (US)
7
7
Oxford Unversity (UK)
6
8
Chinese Academy
(P.R.China)
of
Sciences
ETH Zurich (Switzerland)
Massachusetts
(US)
Fig. 1. The cloud chart of research hotspots that in the field of big data and
cloud computing
Then, we find the information about nationality (Table 2),
institutes (Table 3) of top talents in the high-impact literature
collection. Results showed there were 662 top talents
worldwide in the big data and cloud computing field. The top
ten countries or regions who had the most top talents were the
United States, P.R.China, the United Kindom, Germany, the
Netherlands, France, Canada, Australia, Italy and Switzerland
and Spain tied for the tenth.
General
5
5
Hospital
5
Northwestern University (US)
5
University of British Columbia
(Canada)
5
UC, Berkeley (US)
5
UC, San Diego (US)
5
University of Texas at Austin (US)
5
University of Washington (US)
5
……
2017 Proceedings of PICMET ’17: Technology Management for Interconnected World
From table 2 and 3 we can see that China was in the second
place worldwide. However, China’s top talent is much less than
the United States. In addition, the overall strength of Chinese
research institutions is not strong. So, whether China should
introduce top talents from other countries is need to be
discussed.
According to the formula of the brain gain index, and using
the world population data as well as the Chinese mainland
population data released by the World Bank, the value of the
Chinese brain gain index of big data and cloud computing was
2.61. In comparison, the brain gain index value of the United
States was 0.11. That means China need to introduce top talent
in the field of big data and cloud computing.
IV. CONCLUSION
In the knowledge economy era, the international flow of top
talent has become convenient and frequent. Facing the world’s
top talent shortage, China and the world’s major countries have
developed overseas top talent introduction programs. Until
2007, almost all European countries had introduced some
skillselective migration policies in order to attract the top
talents. To make the overseas top talent introduction programs
more effective and targeted is helpful for occupying the
strategic high ground in the global top talent competition.
This paper improved the traditional talent evaluation
function of bibliometric method, and presented the 3-F analysis
method, which was applied to analyze the demand of top
talents. The 3F method could help the government official to
make decision whether need to introduce top talents to develop
a new industry field and lock these top talents geographic
location.
REFERENCES
[1]
.Xu, B.M., X.G. Ni. Development Trend and Key Technical Progress of
Cloud Computing[J]. Bulletin of the Chinese Academy of Sciences,
2015. 30(2), pp. 170-180.
[2] Xiao, Y., Y. Cheng, Y.J. Fang, Research on Cloud Computing and Its
Application in Big Data Processing of Railway Passenger Flow, in
Iaeds15: International Conference in Applied Engineering and
Management, P. Ren, Y. Li, and H. Song, Editors. 2015, Aidic Servizi
Srl: Milano. pp. 325-330.
[3] Zhu, Y.Q., P. Luo, Y.Y. Huo et. al, Study on Impact and Reform of Big
Data on Higher Education in China, in 2015 3rd International
Conference on Social Science and Humanity, G. Lee and Y. Wu,
Editors. 2015, Information Engineering Research Inst, USA: Newark. p.
155-161.
[4] Wang, X., L.C. Song, G.F. Wang et.al. Operational Climate Prediction
in the Era of Big Data in China: Reviews and Prospects[J]. Journal of
Meteorological Research, 2016. 30(3), pp. 444-456.
[5] Dahlman, C., L. Westphal, Technological effort in industrial
development——An Interpretative Survey of Recent Research[R]. 1982.
[6] Cerna, L., M. Czaika, European Policies to Attract Talent: The Crisis
and Highly Skilled Migration Policy Changes, in High-Skill Migration
and Recession. 2016, Springer. pp. 22-43.
[7] Jin, X., B.W. Wah, X. Cheng et. al. Significance and challenges of big
data research[J]. Big Data Research, 2015. 2(2), pp. 59-64.
[8] Fang, H., Z. Zhang, C.J. Wang et. al. A survey of big data research[J].
IEEE Network, 2015. 29(5), pp. 6-9.
[9] Cole, F.J., Eales, N. B. The history of comparative anatomy[J]. science
Progress, 1917. 11, pp. 578-596.
[10] Gallardo-Gallardo, E., S. Nijs, N. Dries et. al. Towards an understanding
of talent management as a phenomenon-driven field using bibliometric
and content analysis[J]. Human Resource Management Review, 2015.
25, pp. 264-279.
[11] Clarke, B.L. Multiple authorship trends in scientific papers[J]. Science,
1964. 143(3608), pp. 822-824.
[12] Gonzalez-Valiente, C.L., J. Pacheco-Mendoza, R. Arencibia-Jorge. A
review of altmetrics as an emerging discipline for research evaluation[J].
Learned Publishing, 2016. 29(4), pp. 229-238.
Purchase answer to see full
attachment
Why Choose Us
- 100% non-plagiarized Papers
- 24/7 /365 Service Available
- Affordable Prices
- Any Paper, Urgency, and Subject
- Will complete your papers in 6 hours
- On-time Delivery
- Money-back and Privacy guarantees
- Unlimited Amendments upon request
- Satisfaction guarantee
How it Works
- Click on the “Place Order” tab at the top menu or “Order Now” icon at the bottom and a new page will appear with an order form to be filled.
- Fill in your paper’s requirements in the "PAPER DETAILS" section.
- Fill in your paper’s academic level, deadline, and the required number of pages from the drop-down menus.
- Click “CREATE ACCOUNT & SIGN IN” to enter your registration details and get an account with us for record-keeping and then, click on “PROCEED TO CHECKOUT” at the bottom of the page.
- From there, the payment sections will show, follow the guided payment process and your order will be available for our writing team to work on it.