Using machine learning to model the training scalability of convolutional neural networks on clusters of GPUs
![Thumbnail](/xmlui/bitstream/handle/10234/195009/76662.pdf.jpg?sequence=4&isAllowed=y)
View/ Open
Impact
![Google Scholar](/xmlui/themes/Mirage2/images/uji/logo_google.png)
![Microsoft Academico](/xmlui/themes/Mirage2/images/uji/logo_microsoft.png)
Metadata
Show full item recordcomunitat-uji-handle:10234/9
comunitat-uji-handle2:10234/7036
comunitat-uji-handle3:10234/8620
comunitat-uji-handle4:
INVESTIGACIONMetadata
Title
Using machine learning to model the training scalability of convolutional neural networks on clusters of GPUsAuthor (s)
Date
2021-08-30Publisher
SpringerBibliographic citation
Barrachina, S., Castelló, A., Catalán, M. et al. Using machine learning to model the training scalability of convolutional neural networks on clusters of GPUs. Computing 105, 915–934 (2023). https://doi.org/10.1007/s00607-021-00997-9Type
info:eu-repo/semantics/articleVersion
info:eu-repo/semantics/publishedVersionSubject
Abstract
In this work, we build a general piece-wise model to analyze data-parallel (DP) training costs of convolutional neural networks (CNNs) on clusters of GPUs. This general model is based on i) multi-layer perceptrons ... [+]
In this work, we build a general piece-wise model to analyze data-parallel (DP) training costs of convolutional neural networks (CNNs) on clusters of GPUs. This general model is based on i) multi-layer perceptrons (MLPs) in charge of modeling the NVIDIA cuDNN/cuBLAS library kernels involved in the training of some of the state-of-the-art CNNs; and ii) an analytical model in charge of modeling the NVIDIA NCCL Allreduce collective primitive using the Ring algorithm. The CNN training scalability study performed using this model in combination with the Roofline technique on varying batch sizes, node (floating-point) arithmetic performance, node memory bandwidth, network link bandwidth, and cluster dimension unveil some crucial bottlenecks at both GPU and cluster level. To provide evidence of this analysis, we validate the accuracy of the proposed model against a Python library for distributed deep learning training. [-]
Is part of
Computing 105, 915–934 (2023)Funder Name
CRUE-CSIC agreement with Springer Nature | Ministerio de Ciencia, Innovación y Universidades (Spain) | Generalitat Valenciana
Project code
TIN2017-82972-R | Prometeo/2019/109 | Plan GenT project CDEIGENT/2018/014
Project title or grant
Open Access funding provided
Rights
© The Author(s) 2021
info:eu-repo/semantics/openAccess
info:eu-repo/semantics/openAccess
This item appears in the folowing collection(s)
- ICC_Articles [430]