Design and Implementation of a Flexible and Extensible Data Processing Runtime

DC Field Value Language
dc.description학위논문 (석사)-- 서울대학교 대학원 : 공과대학 컴퓨터공학부, 2018. 2. 전병곤.-
dc.description.abstractToday's data analytics applications take a wide variety of characteristics. They are also executed in various resource environments, with many distinct requirements. To face these requirements, many systems have been developed with optimization techniques that are suitable for each system's needs. However, the field of data processing is continuously growing with diverse requirements for job characteristics and resource environments. With current system designs which demonstrate pre-defined runtime behaviors, it is extremely difficult to apply new optimization techniques to them. Onyx is a system that approaches to solve this problem by designing and implementing a flexible and extensible execution runtime. The Onyx execution runtime is designed and implemented around the execution properties that must be flexibly controllable and extensible in order for jobs to be executed under the desired runtime behaviors. It uses a user configurable job representation, Onyx IR, annotated with execution properties which control the underlying runtime behaviors for each job to flexibly execute jobs according to users' requirements. Examples and evaluations show that new optimization techniques are easily applicable to Onyx, which otherwise require a significant amount of engineering effort using current data processing systems.-
dc.description.tableofcontentsChapter 1 Introduction 1

Chapter 2 Background 4
2.1 Data Processing Concepts 4
2.2 Optimizations for Data Processing 5
2.3 Pre-Defined Runtime Behaviors 7
2.4 Execution Properties 10

Chapter 3 Onyx Overview 12
3.1 Onyx IR 12
3.2 Optimization Passes 13
3.3 Submitting to the Execution Runtime 14

Chapter 4 The Execution Runtime 16
4.1 Design 16
4.2 The Processing Backbone 19
4.3 The Flexible and Extensible Execution Properties 21

Chapter 5 Examples 30
5.1 Push Optimization for Small Scale Workloads 30
5.2 Harnessing Transient Resources: Pado 31

Chapter 6 Evaluation 35
6.1 Small Scale Workloads 35
6.1.1 Experimental Setup 35
6.1.2 Results 35
6.1.3 Discussion 37
6.2 Harnessing Transient resources 38
6.2.1 Experimental setup 38
6.2.2 Results 38
6.2.3 Discussion 39

Chapter 7 Conclusion 40
Bibliography 42
국문초록 45
dc.format.extent2826845 bytes-
dc.publisher서울대학교 대학원-
dc.subjectData Processing-
dc.subjectData Processing Framework-
dc.subjectData Analytics-
dc.subjectData Analytics Framework-
dc.subjectData Processing Engine-
dc.subjectData Analytics Engine-
dc.titleDesign and Implementation of a Flexible and Extensible Data Processing Runtime-
dc.contributor.affiliation공과대학 컴퓨터공학부-
Appears in Collections:
College of Engineering/Engineering Practice School (공과대학/대학원)Dept. of Computer Science and Engineering (컴퓨터공학부)Theses (Master's Degree_컴퓨터공학부)
Files in This Item:
  • mendeley

Items in S-Space are protected by copyright, with all rights reserved, unless otherwise indicated.