Streamlining Industrial Big Data Analytics for Smart Manufacturing

Vibhor Pandhare¹, Soumyabrata Bhattacharjee² and Ram Mohril²

1 Department of Mechanical Engineering, Indian Institute of Technology Bombay, Mumbai, India ²Department of Mechanical Engineering, Indian Institute of Technology Indore, Indore, India

E-mail: vibhorpandhare@iitb.ac.in

Status

Since ancient times, people have recorded their observations. Observations become data when they are stored, processed, and shared through various means. Early tools, such as the Ishango Bone, sufficed for simple data management [1]. As civilisation progressed, so did the data management needs. By 1940, the first data centre appeared at the University of Pennsylvania [2]. The late 2000s saw a surge in internet usage, leading to the rise of ‘Big Data’ [3]. In 2011, Industry 4.0 introduced Big Data to manufacturing, utilising sensors to create smart, interconnected factories [4]. In 2012, ‘Industrial Big Data Analytics’ (IBDA) [5]

emerged to draw real-time insights and improve manufacturing decisions. Early research explored whether IBDA could be applied to tasks such as alerting operators about anomalies, predicting maintenance needs, automating fault diagnosis, supporting shop-floor decision-making, optimising performance, and recommending process improvements [6]. Today, the need for Industrial Big Data Analytics is evident as ever. 98% of manufacturing organisations struggle to extract actionable insights from vast, varied industrial data [7]. In 2024 itself, unplanned downtime cost the world’s largest 500 companies trillions of dollars [8]. For example, around 20% of the unplanned downtime in production lines is due to tool wear-out [9]. Traditionally, industries utilise only 50- 80% of a tool’s total available life [10], wasting a valuable resource. Additionally, the manufacturing sector accounts for 30% of global energy consumption [11], a figure that may increase with faulty equipment. In machining, it is also challenging to detect deviations in the toolpath during ongoing processes, which increases scrap and hampers product quality. Additional concerns include reconfigurability of the production line and shop floor to address the growing demand for customised products, keeping resources unchanged. Thus, manufacturing industries are investing heavily in using data-driven decisions to reduce operating costs and carbon footprints while maximising resource utilisation. On these lines, advancements in IBDA are required to maximise the return on investment (ROI) in smart manufacturing (SM). For example, new frameworks are needed to quickly process a large stream of heterogeneous data for real-time anomaly detection and automated quality control. IBDA may also help optimise toolpaths and machine parameters, reducing scrap and improving quality. Monitoring tool and equipment conditions may minimise unplanned downtimes through predictive as well as prescriptive maintenance. This can further reduce the carbon footprint and energy usage in manufacturing value chains. IBDA can enable dynamic reconfiguration of the production processes, with limited resources, to cater to the growing demand for customised products.

Current and future challenges

Given the crucial role IBDA can play in Smart Manufacturing, challenges for these advancements are multidimensional. Specifically, each element of IBDA brings their own set of challenges, as presented below:

• Industrial: When it comes to industry-related challenges, privacy is one of the prominent concerns in safeguarding intellectual property (IP) amid rising cyber threats [12], as storing and sharing sensitive data risks breaches and unauthorised access, necessitating robust security measures. Heterogeneous data from diverse sources creates integration issues due to varying formats and units [13]. Equipment of the same type and state often generates inconsistent data patterns [14]. Limited failure events in industries lead to imbalanced datasets, lacking sufficient failure data for effective modelling [15]. Legacy machines, with outdated interfaces, are complex to integrate [16]. Licensing restrictions limit sensor integration, hindering comprehensive data collection and analysis for optimising industrial processes. In such cases, even if remote sensors and cameras are deployed, ambient conditions like humidity, temperature, and light hinder their effectiveness. Mobile industrial robots struggle to establish stable reference points, restricting mapping ability in dynamic environments, making human-robot collaboration (HRC) risky [17].

• Big: The ubiquitous and indispensable large volume of heterogeneous industrial data demands robust storage and computational infrastructure. Also, protocols are needed to optimise network redundancy and latency for real-time IBDA on data coming at high velocity.

• Data: Variation of not only data formats, but also units of the data of the same parameter, complicates integration and analysis, demanding sophisticated standardisation techniques. Data ownership disputes arise when multiple stakeholders, such as manufacturers and third-party vendors, claim rights, leading to legal and ethical dilemmas. Also, ensuring data veracity becomes a challenge when its volume, variety and velocity are high.

• Analytics: One of the critical challenges in IBDA is feature learning from noisy datasets, as irrelevant or corrupted data can obscure meaningful patterns, requiring advanced filtering and preprocessing techniques. Using open-source tools raises intellectual property and security concerns, complicating industry adoption [18]. Explainability remains a challenge, as complex models, such as deep learning, often lack transparency, which hinders trust and compliance with regulations. Verification, validation, and uncertainty quantification (VVUQ) are essential for fostering trust in the displayed recommendation, yet difficult, as ensuring model accuracy and quantifying uncertainties in dynamic industrial environments demands rigorous methodologies. These challenges hinder the development of reliable, scalable, and trustworthy analytics, necessitating innovative solutions to advance IBDA.

Advances in science and technology to meet challenges

To address these challenges, systematic and streamlined advancements are needed in science and technology, such as seamless interoperability and context-aware adaptability of computational models. Privacy-preserving techniques for federated learning also need to evolve, with scalable algorithms tackling heterogeneous data and robust defences against adversarial attacks. Enabling cross-company collaboration and edge-optimised frameworks may shrink communication delays for real-time model synchronisation. While transfer learning may adapt models built on open-source datasets to industry-specific manufacturing data, it is essential to safeguard the organisation’s intellectual property. Large language models may be retrained on industry-specific data to respond to queries in manufacturing jargon. This may lead to the development of Industrial-GPT, which provides uncertainty-quantified recommendations from heterogeneous data, making them easily communicable to human operators and preventing decision paralysis [19]. However, when developing Industrial-GPT, utmost care must be taken to protect the organisation’s intellectual property. The simultaneous tackling of multiple challenges through advanced research directions for streamlining Industrial Big Data Analytics for Smart Manufacturing is highlighted in Figure 1.

Figure 1. Research direction to address the challenges for streamlining IDBA for Smart Manufacturing.

Computer vision systems may also be developed to the point where they can operate effectively in extreme environments, including varying light conditions and occlusions, without hindrance. This enables them to learn the process behaviour independently, thereby reducing dependence on data labelling. Moreover, in industry, often the data generated by systems is unlabelled. In such cases, industries may adopt autodidactic digital twins (DTs), capable of real-time, unsupervised, uncertainty-quantified, and explainable decision-making. These DTs may dynamically monitor and optimise process parameters using IBDA to enhance productivity, reduce energy usage, and lower the carbon footprint for sustainable manufacturing. These DTs could also be lightweight enough to run on edge devices, not just in the cloud. Determining the fidelity level and refresh rate of each such system-level DT may help integrate them to get insights into the entire manufacturing plant, at any given point in time, which may lead to further optimised process parameters for each system. DTs could also be coupled with lightweight physics-based models to improve effectiveness amidst the dynamic nature of manufacturing.

Concluding remarks

Smart manufacturing, with IBDA at its core, brings significant advantages over traditional practices, for example, lower operational and maintenance costs, a reduced carbon footprint, and improved product quality and resource utilisation. This realisation has led to greater investment in the field. However, maximising ROI requires overcoming specific challenges as highlighted in Figure 1, solving which needs a systematic, simultaneous and multidisciplinary research approach. While addressing all these challenges will take time, innovative solutions may help bridge the gap between legacy and modern manufacturing systems, enabling organisations to begin realising the benefits of smart manufacturing with minimal intervention. It also needs to be noted that preserving the organisation's IP is paramount, irrespective of what technology is developed. Further, no actionable insight can be derived from industrial data by IBDA, unless the veracity of the data is ensured despite its high volume, variety and velocity. Addressing these challenges could help

streamline the integration of the vast stream of heterogeneous industrial data, while protecting its intellectual property (IP), to derive actionable insights. This approach would enable the organisation to minimise costs and maximise productivity.

Acknowledgements

This work is supported by IITI Research Grants: IITI/YFRSG/2023-24/Phase-III/07 and IITI/ YFRSG-Dream Lab/2023-24/Phase-I/03.

References

[1] Huylebrouck D 2008 The ISShango project Journal of Mathematics and the Arts 2145–52 [2] Stuart B L 2018 Debugging the ENIAC [Scanning Our Past] Proceedings of the IEEE 1062331–45 [3] Halevi G and Moed H 2012 The evolution of big data as a research and scientific topic: Overview of the literature Research Trends 1 [4] Vogel-Heuser B and Hess D 2016 Guest Editorial Industry 4.0–Prerequisites and Visions IEEE Transactions on Automation Science and Engineering 13411–3 [5] Kelly J and Floyer D 2013 The industrial internet and big data analytics: Opportunities and challenges 2016-08-02]. http://wikibon. org/wiki/v/The_Industrial_Internet_ and_Big_Data_Analytics: _Opportunities_and_Challenges. Wikibon [6] Nagorny K, Lima-Monteiro P, Barata J and Colombo A W 2017 Big Data Analysis in Smart Manufacturing: A Review Network and System Sciences 1031–58 [7] Anon 2024 Advanced Manufacturing Report | Hexagon 6–7 [8] Akkerman F, Knofius N, van der Heijden M and Mes M 2025 Solving dual sourcing problems with supply mode dependent failure rates Int J Prod Res [9] Zhang A, Ni J, Wei X, Su Y and Liu X 2024 A tool wear prediction method for free-form surface machining of ball-end mill J Manuf Process 13087–101 [10] Chen N, Liu Z, Xue Z, He L, Zou Y, Chen M and Li L 2025 Intelligent wireless tool wear monitoring system based on chucked tool condition monitoring ring and deep learning Advanced Engineering Informatics 65103176 [11] Keramati Feyz Abadi M M, Liu C, Zhang M, Hu Y and Xu Y 2025 Leveraging AI for energy-efficient manufacturing systems: Review and future prospectives J Manuf Syst 78153–77 [12] Leng J, Li R, Xie J, Zhou X, Li X, Liu Q, Chen X, Shen W and Wang L 2025 Federated learning-empowered smart manufacturing and product lifecycle management: A review Advanced Engineering Informatics 65103179 [13] Yang C, Yu H, Zheng Y, Feng L, Ala-Laurinaho R and Tammi K 2025 A digital twin-driven industrial context-aware system: A case study of overhead crane operation J Manuf Syst 78394–409 [14] De Blasi S, Bahrami M, Engels E and Gepperth A 2024 Safe contextual Bayesian optimization integrated in industrial control for self-learning machines J Intell Manuf 35885–903 [15] Altalhan M, Algarni A and Turki-Hadj Alouane M 2025 Imbalanced Data Problem in Machine Learning: A Review IEEE Access 1313686–99 [16] Schlemitz A and Mezhuyev V 2024 Approaches for data collection and process standardization in smart manufacturing: Systematic literature review J Ind Inf Integr 38100578 [17] Zheng C, Du Y, Xiao J, Sun T, Wang Z, Eynard B and Zhang Y 2025 Semantic map construction approach for human-robot collaborative manufacturing Robot Comput Integr Manuf 91102845 [18] Gowripeddi V V, Sasirekha G V K, Bapat J and Das D 2023 Digital Twin and Ontology based DDoS Attack Detection in a Smart-Factory 4.0 5th International Conference on Artificial Intelligence in Information and Communication, ICAIIC 2023286–91 [19] Sanchit, Bhattacharjee S and Pandhare V 2024 Deriving inferences through natural language from structured datasets for asset lifecycle management IFAC-PapersOnLine 58145–50