In the second phase of the MaCuDE project, the Information Systems (IS) task force, in collaboration with the Association for Information Systems (AIS), focused on industry needs related to Big Data Analytics (BDA) and AI and future education to address the needs of these two disciplines. The study was carried out as an open, interview-based field study. It involved interviewing a representative sample of industry leaders making hiring decisions and knowledgeable about the educational needs associated with development and management of BDA and AI based applications.
Methodology
The Phase II MaCuDE IS interview protocol was developed around key changes and issues related to workforce demands covering the new big data analytics environment and applications of artificial intelligence. The sample included 18 interviewees from 16 organizations in multiple industries, who had positions of CIOs, CTOs, product managers, heads of big data consultancy, or head of big data / machine learning teams. All the interviews were then transcribed verbatim and used for coding.
The data analysis was done in three steps. First, open coding of the transcribed interview data was conducted, to identify key topics and issues around big data and related industry needs. NVivo was used to identify emerging concepts and categories in the data, the three authors independently coded the interview transcripts for the distinct first-order categories, discussed and crosschecked the emerging codes, and re-evaluated the coding scheme to capture any missing dimensions or merge overlapping ones. The results were statistically validated. At a next step, axial coding was conducted, to crosscut and relate categories and identify the similarities and differences in the first-order categories. During this analysis, the second-order categories were formed, to represent the dimensions of theoretical interest and uncovered emerging dominant themes. Finally, a three-step member-based validation process was conducted to increase the validity of the analysis.
Key Findings
1. The Big Data and AI environment
The interviews revealed that big data is typically characterized by four key attributes: volume, velocity, variety, and veracity. These characteristics pose new challenges related to data governance and data quality.
New types of data governance needs were commonly identified as a salient aspect of big data. Data governance is in BDA settings more about how to put the data together, how to keep the data quality, how to store and organize the data for further effective use, and how track down and promote the use of the data across different constituencies.
Data quality was heavily emphasized by several interviewees as key element of data governance. This applied especially for participants from construction, finance and insurance, cloud computing, and hospitals and health care. Based on their industrial experience, these interviewees pointed out that the quality of data, not just the size and the speed of collecting data, is a pertinent problem in their practices.
The use of cloud computing is closely intertwined with big data storage and processing, and economic considerations often influence the balance between data quality, governance, and cloud costs.
AI, on the other hand, is generally viewed as the automation of analytics and modeling processes to generate insights and predictions from big data. Machine learning (ML) is considered a subset of AI, focusing on specific analytical techniques.
2. Organizational Transformation
The adoption of BDA and AI is expected to drive three primary organizational outcomes. First, it can lead to changes in products and operations, enabling optimization of performance and the development of new offerings. Second, it can improve decision-making strategies by providing data-driven insights. Third, it can facilitate capability improvement and transformation through learning, automation, and human-machine collaboration.
3. Individual Competencies
The study identified three broad categories of individual competencies essential for success in the BDA and AI landscape.
- Fundamental Environmental Competencies encompass: a) Individual foundational competencies such as teamwork, communication, critical thinking, meta-learning, problem-solving, and systems thinking, and b) Business domain competencies, which include the ability to integrate business and technology, aligning IT with business objectives, identifying business value, and understand the specific business domain.
- Data, Information, and Content Management competencies cover four groups of competencies. Database competencies include foundational database skills, online analytical processing (OLAP), and SQL. Data analytics competencies involve data staging/ETL, cloud-based analytics execution, storytelling, and visualization. Data management competencies encompass data modeling, data architecture, data science lifecycle, and end-to-end data lifecycle management. Business continuity and information assurance competencies include model and data security, as well as privacy and security/ethics.
- Systems Design Competencies comprise five technical competency groups: Individual analytics and programming skills, which include programming and statistics, IT infrastructure competencies involve machine learning, cloud resource management, and architecting for cloud environments. Systems architecture competencies include the use of open-source technologies and understanding data structures, architectures, and governance. AI systems development competencies cover building ML models, understanding the technical foundations of ML, and UI/UX design. IS management competencies, particularly project management, are also essential.
Discussion
The findings suggest that BDA and AI competencies are additive rather than substitutive to existing IS curricula. While core IS topics remain important, there is a need for new emphasis on several areas. These include integrating business and technology skills, understanding data science and analytics lifecycles, developing storytelling and visualization skills, addressing ethics related to BDA/AI, strengthening statistics and mathematical foundations, mastering cloud infrastructure and platform tools, and acquiring AI/ML development skills.
To provide hands-on experience with BDA/AI, IS programs will need to invest in cloud computing resources and support faculty in building relevant technical skills. Specialized undergraduate courses, graduate programs, or certificate offerings may be necessary to cultivate deep expertise in these areas.
From an organizational perspective, expanding IS curricula to comprehensively cover BDA/AI may require trade-offs within the constraints of a business school context. Collaboration across academic units and with industry partners can help IS programs access the necessary skills and resources. Overcoming faculty and student apprehension about mathematical and statistical concepts will be crucial.
As BDA/AI capabilities become integrated into enterprise architectures, IS graduates have the opportunity to bridge the gap between technical experts and business leaders, ensuring that these technologies deliver tangible organizational value. Incorporating design thinking and ethical reasoning throughout the curriculum can equip IS professionals to guide the responsible development of BDA/AI solutions aligned with business goals.
Conclusion
In conclusion, industry demand for BDA/AI skills is accelerating digital transformation across various sectors. IS programs must adapt their curricula and leverage new educational resources to produce graduates who can extract insights from big data, deploy AI/ML models, manage cloud infrastructure, and ultimately lead organizations in harnessing these technologies for competitive advantage. Collaboration between academia and industry, as well as cross-disciplinary partnerships, will be vital to building the future workforce skilled in the tools and techniques of data-driven intelligence. By embracing these changes and proactively addressing the challenges, IS programs can position themselves at the forefront of preparing students for the evolving landscape of BDA and AI in business and society.