Evaluation of Online Services Using the MCDM Approach: User Evaluation and Expert Evaluation

Abstract

The recent development of the Internet technology has drawn attention toward online services. In response, many kinds of online services have been developed. These online services have some distinct characteristics compared with those of traditional offline services. First, online services can provide more service functions than offline services, because there is no restriction of time and space in terms of the service provision. Second, customers often have deficient information about online service providers, because online service provision generally takes place between parties who have never transacted with each other before. Third, online services have difficulty providing the spontaneous responses that can be offered in offline services through face-to-face contact. Given these circumstances, the evaluation of online services through continuous monitoring to reliably provide various service functions has been emphasized. In response, much attention has been paid to the evaluation of online services. Many studies have utilized multiple criteria decision-making (MCDM) approaches for evaluating online services by considering various evaluation criteria simultaneously. Although previous studies contributed to improving our initial understanding of the evaluation of online services, they were limited, as they did not consider the characteristics of evaluators and different types of evaluation data. Accordingly, the focus of this study is to suggest an evaluation framework of online services that considers characteristics of evaluators and types of data in the process of evaluation. The study is composed of two modules and three themes. First, module 1 focuses on the issue of user evaluation of online services using MCDM. Specifically, user evaluation approaches can be divided into two types in terms of the data that can be collected from evaluators: evaluations based on customer reviews and evaluations based on surveys. Therefore, different frameworks for user evaluation in terms of data are provided as different themes. Second, module 2 deals with the issue of expert evaluation of online services using MCDM. This module includes the third theme. The objectives of the three themes are specified as follows. Theme #1 aims to suggest a framework for user evaluation of online services using customer reviews. Customer reviews have been recognized as valuable information sources for monitoring and enhancing customer satisfaction levels, and they provide the real voices of users. Each customer review has concise core content focused on important criteria in comparison with survey data, which contains information about evaluation scores for alternatives in terms of various criteria. It can be useful in situations in which we cannot collect survey data. To analyze a large number of customer reviews effectively and evaluate online services, this study combines VIKOR (in Serbian: ViseKriterijumsa Optimizacija I Kompromisno Resenje) and sentiment analysis. The suggested framework of this study mainly consists of two stages: (1) data collection and preprocessing and (2) customer satisfaction measurement. In the first stage, data collection and preprocessing, text mining is utilized to compile customer-review-based dictionaries of attributes and sentiment words. Then, using sentiment analysis, sentiment scores for attributes are calculated for each mobile service. In the second stage, levels of customer satisfaction are measured using VIKOR. For the purpose of illustration, an empirical case study was conducted on customer reviews of mobile application services. Theme #1 contributes to propose a systematic way of utilizing customer reviews in the evaluation process Theme #2 presents a tailored framework for user evaluation of online services using survey data. In this study, fuzzy hierarchical TOPSIS based on E-SERVQUAL (E-S-QUAL) is suggested for the effective evaluation of online services using information gathered from surveys. To put it more concretely, with E-S-QUAL, which is the extended version of SERVQUAL for the measurement of electronic service quality, information about the gap between the expected and perceived service level collected from users can be extracted effectively. In addition, with fuzzy hierarchical TOPSIS, the hierarchical structure of the evaluation criteria between the main dimensions and their sub-criteria can be considered. With this approach, based on the preservation of the core concept of E-S-QUAL, the hierarchy between the main dimensions and their sub-criteria can be captured. Theme #3 focuses on the suggestion of an integrated model consisting of fuzzy DEMATEL and fuzzy VIKOR as a stepwise method of expert evaluation of online services. This theme mainly focuses on the effective approach for making the best use of knowledge of expert opinions collected from surveys. For expert evaluations, more detailed information (e.g., interrelationships among evaluation criteria) is collected. The evaluation process proceeds as follows. First, several online services providing similar functions are selected as alternatives, and service-based and product-based criteria are defined based on a literature review and expert opinions. Second, interrelationships and importance weights of criteria are calculated using fuzzy DEMATEL. Third, the online service is evaluated using fuzzy VIKOR. In this theme, the inevitable and often conflicting relationships among service-based criteria and product-based criteria can be effectively captured to reflect the evaluation of online services.