作者机构：
[XiaoYong Tang] College of Information Science and Technology, Hunan Agricultural University;[Fan Wu; Qijie Feng] College of Computer Science and Electronic Engineering, Hunan University
会议名称：
2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS)
会议时间：
August 2019
会议地点：
Zhangjiajie, China
会议论文集名称：
2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS)
关键词：
Spark Streaming, Kafka, Parallel Google S2, Big Data
摘要：
In recent years, with the rise of shared bicycles, shared vehicles and other moving objects, how to quickly find nearby objects has gradually become an urgent problem for people to travel. Aiming at the characteristics of fast, real-time and large data volume, this paper proposes a massive data real-time processing system based on Spark Streaming. The massive data generated in real time is sent to the Kafka cluster through the Flume NG cluster. The Spark platform analyzes the data in Kafka in real time, and saves the processed data to redis through our improved parallelized Google S2 algorithm. In this paper, the architecture and function of the system are introduced in detail, and the real-time processing capability of the system is verified by experiments. Experiments show that under the three nodes, when the amount of data is 300,000, our improved algorithm is 4.2% higher than the traditional method; when the amount of data is 500,000, the time is increased by 24%. At the same time, it also verifies the accurate search ability of Google S2 algorithm. Therefore, the architecture and algorithm designed in this paper can be applied to real-time query of objects under large-scale high concurrency.
摘要：
Crowd density estimation is one of the critical issues in social activities. The traditional solution to this problem is to leverage video surveillance to monitor a crowd. However, this is not accurate for crowd density estimation because it is still hard to identify people from background. In the past few years, more and more people use Wi-Fi enabled smartphones. Smartphones can send Wi-Fi request packets periodically, even when they are not connected to access points. This gives another promising solution to the crowd density estimation even for the public environment. In this paper, we first develop a Wi-Fi monitor detection that can capture smartphone passive Wi-Fi signal information including MAC address and received signal strength indicator. Then, we propose a positioning algorithm based on smartphone passive Wi-Fi probe and a dynamic fingerprint management strategy. In real-world public social activities, a person may have zero, one, two, or multiple smartphones with variant Wi-Fi signals. Therefore, we design a method of computing the probability of a user generating one Wi-Fi signal to identify people population. Finally, we propose a crowd density estimation solution based on Wi-Fi probe packets positioning algorithm. Experiments were conducted in an indoor laboratory class and three public social activities, clearly demonstrated that the proposed solution can effectively and accurately estimate crowd density.
关键词：
Cloud data center;Energy consumption;Job scheduling;Workload prediction;Wavelet neural network
摘要：
Data centers are the backbone of cloud infrastructure platform to support large-scale data processing and storage. More and more business-to-consumer and enterprise applications are based on cloud data center. However, the amount of data center energy consumption is inevitably lead to high operation costs. The aim of this paper is to comprehensive reduce energy consumption of cloud data center servers, network, and cooling systems. We first build an energy efficient cloud data center system including its architecture, job and power consumption model. Then, we combine the linear regression and wavelet neural network techniques into a prediction method, which we call MLWNN, to forecast the cloud data center short-term workload. Third, we propose a heuristic energy efficient job scheduling with workload prediction solution, which is divided into resource management strategy and online energy efficient job scheduling algorithm. Our extensive simulation performance evaluation results clearly demonstrate that our proposed solution has good performance and is very suitable for low workload cloud data center.
摘要：
Recently, computational Grids have proven to be a good solution for processing large-scale, computation intensive problems. However, the heterogeneity, dynamics of resources and diversity of applications requirements have always been important factors affecting their performance. In response to these challenges, this work first builds a Grid job scheduling architecture that can dynamically monitor Grid computing center resources and make corresponding scheduling decisions. Second, a Grid job model is proposed to describe the application requirements. Third, this paper studies the characteristics of commercial interconnection networks used in Grids and forecast job transmission time. Fourth, this paper proposes an application-aware job scheduling mechanism (AJSM) that includes periodic scheduling flow and a heuristic application-aware deadline constraint job scheduling algorithm. The rigorous performance evaluation results clearly demonstrate that the proposed application-aware job scheduling mechanism can successful schedule more Grid jobs than the existing algorithms. For successful scheduled jobs, our proposed AJSM method is the best algorithm for job average processing time and makespan.
