Hierarchical Parallel Computing for Machine Learning Data Processing using HPCC's Enterprise Control Language

As part of the ongoing effort to improve neural network implementations on the LexisNexis High-Performance Computing Cluster (HPCC) Systems Platform, a new adaptation designed specifically for the platform's recursive computing environment is presented. This approach systematically addresses distributed and parallelization challenges by optimizing neural network algorithms through the use of the HPCC's ECL programming language. The methodology involves an examination of inefficiencies, with a focus on improving the collection of critical independent variable data. The main objective is to make neural network implementations capable of managing large volumes of Big Data and acting as a robust distributed machine learning system. The data collection process is changed from the conventional Pass-to-All-and-Filter method to a highly efficient, fully parallelized fetching-on-demand strategy by leveraging the robust processing and analytics capabilities of the HPCC Systems Platform. A comprehensive run-time comparison with traditional methods demonstrates significant run-time savings during the learning process, increasing efficiency. This study not only addresses the unique challenges inherent to the HPCC Systems Platform but also provides profound insights into the broader landscape of distributed machine learning, especially in the domain of handling large-scale datasets.