The source of unprecedented data growth in recent years is many and varied. International Data Corporation (IDC) estimated that the Global Datasphere, a measure of all new data collected, created, and replicated in a year across the globe, will grow from 33 Zettabytes (1021 bytes) in 2018 to 175 by 2025 (Reinsel et al., 2018). A zettabyte is a trillion Gigabytes.
In May 2020, IDC published an update. The estimated growth of data for the year 2020 alone was 59 Zettabytes with a forecast of continued growth through 2024 with a five-year compound annual growth rate (CAGR) of 26%. The COVID-19 pandemic contributes to this figure by causing an abrupt increase in the number of work-from-home employees and rapid digitization (IDC, 2020).
A vast majority of this data is transmitted, processed, and stored at enterprise data centers. The remarkably high rate of new data creation means there need to be more storage systems in the enterprise data centers. These days, enterprise IT infrastructure consists of tens of thousands of physical and virtual servers and their associated hardware like servers, network equipment, and storage systems spread across geographies. The amount of data stored in each data center is in multiple Petabytes (1015 bytes).
Storage systems consist of dedicated servers, storage media, and related software to obtain a high-performance, high availability, and efficiently managed system. The main types of storage media used in data centers are tape drives, magnetic hard drives, and solid-state drives. Enterprise-grade hardware is meant to run continuously, twenty-four hours a day, seven days a week.
Storage systems have developed over decades, improving performance, cutting cost, and, most importantly, enabling modern computing needs by supporting the new types of workloads. The recent industry trend in IT infrastructure management emphasizes automation of the daily repetitive tasks through various commercially available software or homegrown scripts. Another trend in enterprise IT is to leverage public cloud storage for offloading some or most management responsibilities to a third party.
Enterprise data storage systems go through hardware refresh every three to five years to take advantage of newly available features and reduce risks due to aging hardware. These are multi-million-dollar decisions involving implementation and data migration plans that span months to years.
Technology assessment is the evaluation and estimation of the nature, quality, or ability of the technology. It started as a form of public policy research to examine various short and long-term consequences of technology use. Such analysis requires consideration of multiple perspectives and criteria.
Multiple Criteria Decision Making (MCDM) is one of the most widely used decision methodologies in various fields that aim to satisfy the multitude of conflicting objectives in the best possible way. We derive measurements by directly comparing objects. Thomas L. Saaty established that direct comparisons are necessary to establish measurements for intangible properties with no scales of measurement (Saaty, 2008). Methods based on pairwise comparison form a significant part of multiple criteria decision making.
Decision-makers need models that are updated, capturing all significant perspectives and criteria. IT executives in the decision-making positions often supplement their knowledge with advice from experts in the field. This study discusses the use of expert judgment in the assessment of enterprise data storage systems. Expert judgment can be defined as an expert opinion given in the context of a specific decision problem. Expert judgment is a recognized, mature research methodology suitable for assessing emerging technology where benchmarks have not been established or no objective data is available. Expert judgment quantification utilizes rating instruments in the form of a questionnaire to convert informed estimates from the experts to numeric values. This study uses a constant sum pairwise comparison to record one criterion or perspective’s importance versus another. The information that the experts provide becomes data. Conclusions are drawn by combining expert judgments as an aggregation of quantitative estimates.
In this study, we map the process of expert judgment to validate the STORE assessment model for assessing enterprise data storage systems (Shrestha & Sheikh, n.d.). We also used quantification of expert judgment in finding the criteria weights. This study speaks primarily to the decision-makers who fill one of these roles:
· Senior Executives responsible for leadership and technology purchase decisions
· Storage Managers responsible for providing the storage services
· Storage Architects responsible for the design of storage solutions
· Storage Engineers engaged in the implementation of storage solutions
· IT Operations Staff responsible for the daily operations
2. Literature Review
We performed a literature review to understand and develop a scholarly base for the three related research areas: Enterprise IT, Data Storage, and Technology Assessment as shown in Figure 1 (Shrestha & Sheikh, n.d.).
The review enabled identifying gaps in prior research, specifically, the lack of a comprehensive decision model, covering Strategic, Technological, Operational, Regulatory, and Economic (STORE) perspectives. Each of the STORE perspectives was subjectively categorized and considered a preferentially orthogonal dimension. Preferentially orthogonal perspectives are the independent dimensions. The combined dimensions are then deemed to be comprehensive for the assessment of EDSS.
Strategic perspective considers high and low level, short- and long-term goals of the organization. Technological perspective considers various criteria that relate to the capability and efficiency of the storage system. The operational perspective considers how the product will affect the day-to-day operations of the IT function. The regulatory perspective considers the legal aspects of technology implementation. The economic perspective considers the financial aspects of the solution. We derived twenty criteria by grouping the related concepts and categorized them under the five STORE perspectives.
This study expands our literature review to expert judgment quantification and constant sum pairwise comparison in finding criteria weights for multiple criteria decision making. We explore the recent use of these methods in the information technology domain.
Multiple criteria decision making refers to all methods that help designate a preferred alternative and rank alternatives based on subjective preferences, where there is more than one criterion (Ho, 2008). Some authors (Zimmermann, 1991; Chen & Hwang, 1992) categorized MCDM into two categories: 1) Multi-Attribute Decision Making (MADM) problems, where the number of alternatives is predetermined, and 2) Multi-Objective Decision Making (MODM), where they are not.
MCDM has been used in the study of cloud service selection (Rehman et al., 2011), IT infrastructure selection for smart grid (Rezagholizadehl et al., 2013), big data storage selection (Kachaoui & Belangour, 2019), IT disaster recovery site selection (Yang et al., 2015) and justification of IT investments (Borenstein & Betencourt, 2005).
Figure 1. Literature review intersection.
Expert judgment quantification adds substantial value in analyzing complex problems when there are no universally accepted scientific laws or extensive data available. Keeney & Von Winterfeldt (1989) stressed the value of quantifying expert judgments to complement the expert’s qualitative thinking and reasoning. They also highlighted the need for explicit judgments to avoid misinterpretations and misuse.
Expert judgment has been studied in various fields, including cybersecurity (Holm et al., 2014), web development projects (Torrecilla-Salinas et al., 2019), regression models of software effort estimation (Tsunoda et al., 2012), the potential of blockchain in supply chain management (Kopyto et al., 2020) and addressing uncertainty in high technology system design (Chytka et al., 2006).
The constant sum pairwise comparison method is used in the scientific study of preferences, attitudes, and requirements engineering. It reflects the importance or priority attached by a respondent to one entity compared to another. The number of independent pairwise comparisons for n number of entities is n(n − 1)/2. In multiperspective hierarchical decision making, criteria are the lower-level entities to perspectives in the hierarchy. Constant sum pairwise comparison has a relative orientation providing more decision context than binary choices. It is unavoidably more complex than a binary or discrete choice task, resulting in inattention or higher drop-out rates (Skedgel & Regier, 2015).
Pairwise comparison has been used in various fields, including IT infrastructure refresh planning for enterprises (Daim et al., 2011), engineering design (Dym et al., 2002), document ranking algorithms in information retrieval systems (Ozbey & Dincsoy, 2020), intelligent transportation recommendation systems (Borodinov et al., 2020), and image quality assessment (Zhang et al., 2017).
3. Development of Assessment Model
Figure 2 shows the STORE assessment model with five perspectives and twenty criteria in a hierarchy (Shrestha & Sheikh, n.d.). Note that certain aspects of a criterion are covered by others in the model. For example, technical aspects of data security under regulatory perspective are considered in technology features under technological perspective.
Short definitions of the perspectives and criteria:
Strategic Perspective: Strategic perspective considers high and low level, short- and long-term goals of the organization.
Technological Perspective: Technological perspective considers various criteria that relate to the storage system’s capability and efficiency.
Operational Perspective: Operational perspective considers how the product will affect the day-to-day operations of the IT function.
Regulatory Perspective: Regulatory perspective considers the legal compliance aspects of technology implementation. Note that some regulations like GDPR cover more than one criterion in the STORE regulatory perspective.
Economic Perspective: Economic perspective considers the financial aspects of the solution.
Figure 2. STORE assessment model.
Business Strategy: Business strategy is a set of guiding principles that aim to achieve business objectives like service enablement, revenue growth, and cost-saving.
Technology Strategy: Technology strategy explains how we should utilize technology as part of an organization’s business strategy.
Organizational Readiness: Organizational readiness refers to the availability of a skilled workforce and defined processes for technology implementation and operations.
Technology Feature: Technology features of storage systems include policy-based provisioning, orchestration, storage snapshot, and replication.
System Performance: System performance of storage systems includes throughput in IOPS and latency.
System Reliability: The system reliability of a storage system is a measure of performing consistently well. It contributes directly to the high availability of applications supported by the system.
Capacity Management: Capacity management refers to the ease with which capacity is expanded when needed. Capacity management is aided by data reduction techniques like compression and deduplication.
Technological Complexity: Technological complexity refers to a difficulty understanding the storage system and its interaction with other IT components.
Storage Implementation: Storage implementation for an application involves setting up the physical hardware and cables, configuring the device, testing, and migrating data.
Storage Administration: Storage administration tasks include provisioning storage, creating storage units, and performing cleanups.
System Monitoring: System monitoring is essential for the fine-tuning of storage systems and identifying performance bottlenecks.
System Reporting: System reporting is essential for keeping track of how well the system fulfills the needs. It also helps in making decisions related to capacity expansion and others.
Vendor Support: Vendor support is characterized by the ease of creating service requests, communicating with support engineers, and a clear escalation path.
Data Privacy: Data privacy relates to the protection of consumer data. Examples of the regulations that might be applicable are European Union General Data Protection Regulation (GDPR), California Consumer Privacy Act (CCPA), and others.
Data Security: Data security refers to the prevention of unauthorized access. Laws like the Cybersecurity Information Sharing Act (CISA) equip organizations to secure the data from the latest cyber threats.
Data Retention: Data retention means the safe keeping of data for future access. Examples of regulations that might be applicable are the Sarbanes Oxley Act (SOX) and Health Insurance Portability and Accountability Act (HIPPA).
Data Transfer: Data transfer refers to the mobility of data. An example of data transfer regulation is cross-border data transfers under GDPR.
Capital Expense: Capital expenses (CAPEX) consist of purchasing equipment or services towards fixed assets that the company will use beyond the current year.
Operating Expense: Operating expenses (OPEX) refers to the ongoing expenses for the operation and maintenance, including the cost of power, space, and cooling in the data center.
Total Cost of Ownership: Total cost of ownership (TCO) is a holistic view of the enterprise’s expenses over time.
4. Expert Judgment Process
Figure 3 shows the process of expert judgment quantification used in this research study. It involved five steps: Define Decision Problem, Recruit Experts, Design Research Instruments, Collect Expert Judgment, and Analyze Expert Judgment.
4.1. Define Decision Problem
Data storage systems are essential components of enterprise IT operations. Application performances depend heavily on the underlying data storage systems. Storage system implementation and data migration can take years of coordinated effort and tens of millions of dollars in investment. A successful storage implementation can bring new IT capabilities, and a bad case can jeopardize entire enterprise IT stability. Performance issues in the storage systems or a total failure can cause costly service outages. Therefore, the selection of EDSS is a critical decision for IT executives.
Figure 3. Expert judgment process.
In enterprise IT infrastructure, hardware refresh takes place every three to five years. The supplier’s continual development in computer hardware and software technologies provides new features, but it also introduces more variables for the decision-making process. The availability of more options further complicates the process.
IT executives need to assess various storage systems and select the best alternative to fulfill the business needs. Evaluation of an EDSS needs careful consideration of all related perspectives and criteria. A hierarchical model with criteria weights allows decision-makers to apply numeric methods. A multi-criteria decision model to assess the storage systems must be developed for a comprehensive approach to this problem. Experts are needed to validate the assessment model and assign weights subjectively.
4.2. Recruit Experts
The potential participants, experts in EDSS, were contacted initially through email, telephone calls, and in-person interviews to introduce them to the research. Their potential value was determined by expertise on the research topic and ability to answer related questions. All selected experts have demonstrated experience through vocation, education, or both with a minimum of ten years of related experience.
We selected the experts based on their years of related experience working with leading information technology organizations. We formed six panels from the twenty-six participants—one for calculating the perspective weights and one each for criteria weights in the five STORE perspectives. Based on work experience, an expert could be included in more than one panel. Participation was voluntary and, we did not provide any financial incentives for participation.
4.2.1. Expert Qualification
Table 1 shows the expert’s experience in years—total IT experience, and experience in each of the STORE perspectives related to EDSS. An asterisk (*) indicates the experience of fewer than ten years.
Table 1. Work experience of experts in years.
4.2.2. Formation of Expert Panels
From the expert’s pool (Table 1), we formed six panels—one for calculating perspective weights and one each for criteria weights under the five STORE perspectives. Experts with at least ten years of experience in all five perspectives are included in panel 1. Panels 2 through 6 include experts with at least ten years of experience in the related perspectives.
An expert can be included in more than one panel.
Panel 1: Experts in all five STORE perspectives (14 experts).
Expert 2, 4, 6, 7, 8, 11, 12, 14, 15, 20, 21, 23, 24, 26.
Panel 2: Experts in Strategic Perspectives (18 experts).
Expert 2, 4, 5, 6, 7, 8, 10, 11, 12, 13, 14, 15, 19, 20, 21, 23, 24, 26.
Panel 3: Experts in Technological Perspectives (24 experts).
Expert 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19, 20, 21, 22, 23, 24, 25, 26.
Panel 4: Experts in Operational Perspectives (22 experts).
Expert 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 12, 13, 14, 15, 18, 19, 20, 21, 22, 23, 24, 26.
Panel 5: Experts in Regulatory Perspectives (19 experts).
Expert 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 21, 23, 24, 26.
Panel 6: Experts in Economic Perspectives (16 experts).
Expert 2, 4, 6, 7, 8, 10, 11, 12, 14, 15, 17, 20, 21, 23, 24, 26.
4.3. Design Research Instruments
Pairwise comparison is a process in which experts rate a set of criteria, perspectives, or alternatives only two at a time. While this method is time-consuming to elicit all possible combinations and only provides relative data relations, recent research shows that people make better relative judgments than direct estimates (Benini et al., 2017). This study adopts the pairwise comparison method with seven stepped levels to assess solar photovoltaic technologies (Sheikh, 2013).
We selected an elicitation situation of individual experts instead of interactive group or Delphi to avoid potential bias from group dynamics. We chose a web-based form as the mode of communication to capture encoded expert judgments. There were two questionnaires for the experts—the first questionnaire (Appendix A) validated the assessment model, and the second established criteria weights (Appendix B).
4.4. Collect Expert Judgment
We met with each expert through a video chat to explain the research study and question format through screen share. The sessions took about 20 minutes each. We then sent the web link to the experts with the following greeting message:
“This research study aims to develop a comprehensive assessment model to evaluate Enterprise Data Storage Systems (EDSS). EDSS represents the servers, storage media, or appliance used for storing digital data. Some examples of EDSS are Storage Area Network (SAN), Network Attached Storage (NAS), Direct Attached Storage (DAS), Hyperconverged Infrastructure (HCI), and Public Cloud Storage.
You are being asked to participate in this research study because of your EDSS expertise. Your experience working with the leading information technology organizations makes you uniquely valuable to this research.
The questionnaire (I) relates to the validation of the STORE assessment model. Questionnaire (II) has a total of 42 pairwise comparison questions with multiple-choice answers. Questions are not specific to any job, organization, or vendor but EDSS in general. These questionnaires should not take more than 30 minutes.
Informed Consent: Participation in this research activity is voluntary. The participants may withdraw at any time without penalty or loss of benefits. The questionnaires are anonymous. Please do not enter any personally identifiable information.”
4.5. Analyze Expert Judgment
After collecting the expert judgment, we used a quantification scale described below with a constant sum of 100.
Attribute A is four times as important as Attribute B; A = 80 and B = 20
Attribute A is three times as important as Attribute B; A = 75 and B = 25
Attribute A is two times as important as Attribute B; A = 67 and B = 33
Attribute A is equally important as Attribute B; A = 50 and B = 50
Attribute A is one-half times important as Attribute B; A = 33 and B = 67
Attribute A is one-third times important as Attribute B; A = 25 and B = 75
Attribute A is one-fourth times important as Attribute B; A = 20 and B = 80
A score of zero is entered for both A and B when experts do not qualify for the panel. In the next steps, we aggregated the scores to obtain combined values.
4.5.1. Panel 1: Perspective Weights
Table 2 shows the quantified expert judgment on questions 1 through 10 (Appendix B) using the method explained in Section 4.5.
Sum of perspective scores from all pairwise comparisons:
Strategic Perspective = 845 + 789 + 662 + 713 = 3,009
Technological Perspective = 555 + 843 + 696 + 763 = 2,857
Operational Perspective = 611 + 557 + 717 + 777 = 2,662
Regulatory Perspective = 738 + 704 + 683 + 860 = 2,985
Economic Perspective = 687 + 637 + 623 + 540 = 2,487
Total Score = 3,009 + 2,857 + 2,662 + 2,985 + 2,487 = 14,000
4.5.2. Panel 2: Strategic Perspective
Table 3 shows the quantified expert judgment on questions 11 through 13 (Appendix B) using the method explained in Section 4.5.
Sum of criterion scores from all pairwise comparisons:
Business Strategy = 956 + 875 = 1,831
Technology Strategy = 844 + 1,056 = 1,900
Organizational Readiness = 875 + 744 = 1,696
Total Score = 1,831 + 1,900 + 1,696 = 5,400
4.5.3. Panel 3: Technological Perspective
Table 4 shows the quantified expert judgment on questions 14 through 23 (Appendix B) using the method explained in Section 4.5.
Sum of criterion scores from all pairwise comparisons:
Technology Features = 1,177 + 1,048 + 1,267 + 1,252 = 4,744
System Performance = 1,223 + 1,158 + 1,471 + 1,461 = 5,313
System Reliability = 1,352 + 1,242 + 1,491 + 1,571 = 5,656
Capacity Management = 1,133 + 929 + 909 + 1,437 = 4,408
Technological Complexity = 1,148 + 939 + 829 + 963 = 3,879
Total Score = 4,744 + 5,313 + 5,656 + 4,408 + 3,879 = 24,000
4.5.4. Panel 4: Operational Perspective
Table 5 shows the quantified expert judgment on questions 24 through 33 (Appendix B) using the method explained in Section 4.5.
Sum of criterion scores from all pairwise comparisons:
Storage Implementation = 1,020 + 1,011 + 1,091 + 988 = 4,110
Storage Administration = 1,180 + 1,269 + 1,353 + 1,057 = 4,859
System Monitoring = 1,189 + 931 + 1,214 + 1,091 = 4,425
System Reporting = 1,109 + 847 + 986 + 940 = 3,882
Vendor Support = 1,212 + 1,143 + 1,109 + 1,260 = 4,724
Total Score = 4,110 + 4,859 + 4,425 + 3,882 + 4,724 = 22,000
4.5.5. Panel 5: Regulatory Perspective
Table 6 shows the quantified expert judgment on questions 34 through 39 (Appendix B) using the method explained in Section 4.5.
Sum of criterion scores from all pairwise comparisons:
Data Privacy = 937 + 1,080 + 1,068 = 3,085
Data Security = 963 + 1,246 + 1,154 = 3,363
Data Transfer = 820 + 654 + 917 = 2,391
Data Retention = 832 + 746 + 983 = 2,561
Total Score = 3,085 + 3,363 + 2,391 + 2,561 = 11,400
4.5.6. Panel 6: Economic Perspective
Table 7 shows the quantified expert judgment on questions 40 through 42 (Appendix B) using the method explained in Section 4.5.
Sum of criterion scores from all pairwise comparisons:
Capital Expense = 1,011 + 838 = 1,849
Operating Expense = 589 + 860 = 1,449
Total Cost of Ownership = 762 + 740 = 1,502
Total Score = 4,800
Table 3. Quantified expert judgment for criteria related to Strategic Perspective.
Table 6. Quantified expert judgment for criteria related to Regulatory Perspective.
Table 7. Quantified expert judgment for criteria related to Economic Perspective.
We used the following set of equations to obtain the final criteria weights represented by Cji
p represents a perspective.j is the index of perspectives ranging from 1 to J.
c represents a criterion. i is the index of criteria ranging from 1 toI.
The method of constant sum pairwise comparison ensures the following three conditions:
Condition I: Sum of all perspective weights equal to one. Note: pj is the weight of the jth perspective.
Condition II: Sum of all initial criteria weights within each perspective equal to one. Note: is the initial weight of criterion i under perspectivej. Ij is the maximum number of criteria in perspective j.
Condition III: Sum of all final criteria weights equal to one. Note: Cji represents the final weight of criterion i under perspective j.
5.1. Calculation of Perspective Weights
The first column in Table 8 gives the sum of scores for each perspective from all four pairwise comparisons.
Table 8. Calculation of perspective weights.
Total score possible = Number of pairwise comparisons (10) * Number of panelist (14) * 100 = 1,400.
Perspective weight pj = The sum of scores from all pairwise comparisons/Total score possible.
5.2. Calculation of Criteria Weights
Table 9 shows the final result with weights for all twenty criteria, the lowest-level in the STORE hierarchical model.
Table 9. Final results.
Twenty-five out of twenty-six experts responded with “yes” to the first questionnaire for model validation. One of the experts replied “no,” suggesting a new criterion, Data Resiliency under Regulatory Perspective. Data resiliency refers to the speed and quality of recovery in case data is compromised. From a regulatory perspective, this is similar in concept to data retention. If data retention is ensured through periodic backups, resilience is confirmed. Also, we did not find literature with an established data resilience regulations applicable across industries. We conclude that the STORE assessment model for EDSS is complete and validated by the experts.
The perspective weights in the order of higher importance are Strategic Perspective (21.5%), Regulatory Perspective (21.3%), Technological Perspective (20.4%), Operational Perspective (19.0%), and Economic Perspective (17.8%).
The criteria weights in the order of higher importance are Technology Strategy (7.565%), Business Strategy (7.290%), Capital Expense (6.857%), Organizational Readiness (6.645%), Data Security (6.284%), Data Privacy (5.764%), Total Cost of Ownership (5.570%), Operating Expense (5.373%), System Reliability (4.808%), Data Retention (4.785%), System Performance (4.516%), Data Transfer (4.467%), Storage Administration (4.196%), Vendor Support (4.080%), Technology Features (4.032%), System Monitoring (3.822%), Capacity Management (3.747%), Storage Implementation (3.550%), System Reporting (3.353%), and Technological Complexity (3.297%).
A group of twenty-six experts working for large US enterprise IT organizations, each with at least ten years of experience, validated the STORE assessment model. Six expert panels, each with at least ten experts, made constant sum pairwise comparisons to derive weights for the twenty criteria.
Experts put the most weight on Technology Strategy, followed by Business Strategy and Capital Expense. The lowest weight of Technology Complexity indicates that the nature and role of complexity in IT infrastructure management are underestimated or poorly understood.
The results give a reliable understanding of various criteria required to evaluate an enterprise data storage system. However, we caution against over-interpreting the expert judgments in this study and relying too heavily on minor numerical differences.
Contribution to the Body of Knowledge
This original empirical research study validated the STORE assessment model for enterprise data storage systems using expert judgment and calculated criteria weights using constant-sum pairwise comparison.
Assumptions and Limitations
All twenty-six experts worked for large US enterprise IT organizations—this research study considered only the consumer worldview.
A 0 to 100 continuous scale can improve the results’ accuracy compared to the seven stepped levels used in this study. We chose the seven steps at the unanimous request of the experts.
Other worldviews such as a vendor, consulting company, or regulating agencies can be considered in future research. Researchers can expand each criterion to multiple factors as the field evolves, definitions are clear, and more commonly accepted benchmarks are established. Results from this study can be used in studies related to the selection of enterprise data storage systems.
We thank the twenty-six experts who, despite their busy schedules, took time to listen to our research and participated with great enthusiasm.
Q1. Does the STORE assessment model capture all significant perspectives and criteria in evaluating an EDSS?
Q2. If your answer to question 1 is No, please write the new perspectives or criteria to complete the assessment model. (_____________________________________________________)
There is a total of 42 pairwise comparisons with multiple-choice answers. All questions are strictly in the context of EDSS. FigureB1 shows the answer options and screen capture for question 1. The seven answer options are the same for all questions.
Q1. Please rate the importance of Strategic Perspective with respect to Technological Perspective.
Strategic Perspective is FOUR times as important as Technological Perspective.
Strategic Perspective is THREE times as important as Technological Perspective.
Strategic Perspective is TWO times as important as Technological Perspective.
Strategic Perspective is EQUALLY important as Technological Perspective.
Strategic Perspective is ONE-HALF times as important as Technological Perspective.
Strategic Perspective is ONE-THIRD times as important as Technological Perspective.
Strategic Perspective is ONE-FOURTH times as important as Technological Perspective.
Figure B1. Screen capture of Question 1.
Q2. Strategic Perspective with respect to Operational Perspective
Q3. Strategic Perspective with respect to Regulatory Perspective
Q4. Strategic Perspective with respect to Economic Perspective
Q5. Technological Perspective with respect to Operational Perspective
Q6. Technological Perspective with respect to Regulatory Perspective
Q7. Technological Perspective with respect to Economic Perspective
Q8. Operational Perspective with respect to Regulatory Perspective
Q9. Operational Perspective with respect to Economic Perspective
Q10. Regulatory Perspective with respect to Economic Perspective
Q11. Business Strategy with respect to Technology Strategy
Q12. Business Strategy with respect to Operational Readiness
Q13. Technology Strategy with respect to Operational Readiness
Q14. Technology Features with respect to System Performance
Q15. Technology Features with respect to System Reliability
Q16. Technology Features with respect to Capacity Management
Q17. Technology Features with respect to Technological Complexity
Q18. System Performance with respect to System Reliability
Q19. System Performance with respect to Capacity Management
Q20. System Performance with respect to Technological Complexity
Q21. System Reliability with respect to Capacity Management
Q22. System Reliability with respect to Technological Complexity
Q23. Capacity Management with respect to Technological Complexity
Q24. Storage Implementation with respect to Storage Administration
Q25. Storage Implementation with respect to System Monitoring
Q26. Storage Implementation with respect to System Reporting
Q27. Storage Implementation with respect to Vendor Support
Q28. Storage Administration with respect to System Monitoring
Q29. Storage Administration with respect to System Reporting
Q30. Storage Administration with respect to Vendor Support
Q31. System Monitoring with respect to System Reporting
Q32. System Monitoring with respect to Vendor Support
Q33. System Reporting with respect to Vendor Support
Q34. Data Privacy with respect to Data Security
Q35. Data Privacy with respect to Data Transfer
Q36. Data Privacy with respect to Data Retention
Q37. Data Security with respect to Data Transfer
Q38. Data Security with respect to Data Retention
Q39. Data Transfer with respect to Data Retention
Q40. Capital Expense with respect to Operating Expense
Q41. Capital Expense with respect to Total Cost of Ownership
Q42. Operating Expense with respect to Total Cost of Ownership
 Benini, A., Chataigner, P., Noumri, N., Parham, N., Sweeney, J., & Tax, L. (2017). Expert Judgment: The Use of Expert Judgment in Humanitarian Analysis: Theory, Methods and Applications. Geneva: Assessment Capacities Project.
 Borenstein, D., & Betencourt, P. R. B. (2005). A Multi-Criteria Model for the Justification of IT Investments. Information Systems and Operational Research, 43, 1-21.
 Borodinov, A., Agafonov, A., & Myasnikov, V. (2020). A Method of Preference and Utility Elicitation by Pairwise Comparisons and Its Application to Intelligent Transportation Recommendation Systems. 10th International Conference on Information Science and Technology, Bath, London, and Plymouth, 9-15 September 2020, 77-85.
 Chytka, T. M., Conway, B. A., & Unal, R. (2006). An Expert Judgment Approach for Addressing Uncertainty in High Technology System Design. Portland International Conference on Management of Engineering and Technology, 1, 444–449.
 Daim, T. U., Letts, M., Krampits, M., Khamis, R., Dash, P., Monalisa, M., & Justice, J. (2011). IT Infrastructure Refresh Planning for Enterprises: A Business Process Perspective. Business Process Management Journal, 17, 510-525.
 Dym, C. L., Wood, W. H., & Scott, M. J. (2002). Rank Ordering Engineering Designs: Pairwise Comparison Charts and Borda Counts. Research in Engineering Design, 13, 236-242.
 Holm, H., Sommestad, T., Ekstedt, M., & Honeth, N. (2014). Indicators of Expert Judgement and Their Significance: An Empirical Investigation in the Area of Cyber Security. Expert Systems, 31, 299-318.
 Kachaoui, J., & Belangour, A. (2019). A Multi-Criteria Group Decision Making Method for Big Data Storage Selection. In M. Atig, & A. Schwarzmann (Eds.), NETYS 2019: Networked Systems (pp. 381-386). Cham: Springer.
 Keeney, R. L., & Von Winterfeldt, D. (1989). On the Uses of Expert Judgment on Complex Technical Problems. IEEE Transactions on Engineering Management, 36, 83-86.
 Kopyto, M., Lechler, S., von der Gracht, H. A., & Hartmann, E. (2020). Potentials of Blockchain Technology in Supply Chain Management: Long-Term Judgments of an International Expert Panel. Technological Forecasting and Social Change, 161, Article ID: 120330.
 Ozbey, C., & Dincsoy, O. (2020). An Efficient Pairwise Comparison Scheme for Document Ranking. 2020 28th Signal Processing and Communications Applications Conference, Gaziantep, 5-7 October 2020, 1-4.
 Rehman, Z. U., Hussain, F. K., & Hussain, O. K. (2011). Towards Multi-Criteria Cloud Service Selection. Proceedings of 2011 5th International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing, Seoul, 30 June-2 July 2011, 44-48.
 Rezagholizadehl, M., Mehrannii, P., Barzegar, A., Fereidunian, A., Moshiri, B., & Lesani, H. (2013). A Probabilistic Partial Order Theory Approach to IT Infrastructure Selection for Smart Grid. International Conference on Control, Automation and Systems, Gwangju, 20-23 October 2013, 488-493.
 Saaty, T. L. (2008). Relative Measurement and Its Generalization in Decision Making Why Pairwise Comparisons Are Central in Mathematics for the Measurement of Intangible Factors The Analytic Hierarchy/Network Process (To the Memory of my Beloved Friend Professor Sixto Rios Garcia). Revista de la Real Academia de Ciencias Exactas, Fisicas y Naturales. Serie A. Matematicas, 102, 251-318.
 Shrestha, L., & Sheikh, N. J. (n.d.). Multiperspective Assessment of Enterprise Data Storage Systems: Literature Review. Portland International Conference on Management of Engineering and Technology (PICMET).
 Torrecilla-Salinas, C. J., De Troyer, O., Escalona, M. J., & Mejías, M. (2019). A Delphi-Based Expert Judgment Method Applied to the Validation of a Mature Agile Framework for Web Development Projects. Information Technology and Management, 20, 9-40.
 Tsunoda, M., Monden, A., Keung, J., & Matsumoto, K. (2012). Incorporating Expert Judgment into Regression Models of Software Effort Estimation. 2012 19th Asia-Pacific Software Engineering Conference, Hong Kong, 4-7 December 2012, 374-379.
 Yang, C. L., Yuan, B. J. C., & Huang, C. Y. (2015). Key Determinant Derivations for Information Technology Disaster Recovery Site Selection by the Multi-Criterion Decision Making Method. Sustainability, 7, 6149-6188.
 Zhang, Z., Zhau, J., Liu, N., Gu, X., & Zhang, Y. (2017). An Improved Pairwise Comparison Scaling Method for Subjective Image Quality Assessment. 2017 IEEE International Symposium on Broadband Multimedia Systems and Broadcasting, Cagliari, 7-9 June 2017, 1-6.