Using machine learning to model the training scalability of convolutional neural networks on clusters of GPUs
Ver/ Abrir
Impacto
Scholar |
Otros documentos de la autoría: Barrachina Mir, Sergio; Castelló, Adrián; Catalán Carbó, Mar; Dolz, Manuel F.; Mestre Miravet, Jose Ignacio
Metadatos
Mostrar el registro completo del ítemcomunitat-uji-handle:10234/9
comunitat-uji-handle2:10234/7036
comunitat-uji-handle3:10234/8620
comunitat-uji-handle4:
INVESTIGACIONMetadatos
Título
Using machine learning to model the training scalability of convolutional neural networks on clusters of GPUsAutoría
Fecha de publicación
2021-08-30Editor
SpringerCita bibliográfica
Barrachina, S., Castelló, A., Catalán, M. et al. Using machine learning to model the training scalability of convolutional neural networks on clusters of GPUs. Computing 105, 915–934 (2023). https://doi.org/10.1007/s00607-021-00997-9Tipo de documento
info:eu-repo/semantics/articleVersión
info:eu-repo/semantics/publishedVersionPalabras clave / Materias
Resumen
In this work, we build a general piece-wise model to analyze data-parallel (DP) training costs of convolutional neural networks (CNNs) on clusters of GPUs. This general model is based on i) multi-layer perceptrons ... [+]
In this work, we build a general piece-wise model to analyze data-parallel (DP) training costs of convolutional neural networks (CNNs) on clusters of GPUs. This general model is based on i) multi-layer perceptrons (MLPs) in charge of modeling the NVIDIA cuDNN/cuBLAS library kernels involved in the training of some of the state-of-the-art CNNs; and ii) an analytical model in charge of modeling the NVIDIA NCCL Allreduce collective primitive using the Ring algorithm. The CNN training scalability study performed using this model in combination with the Roofline technique on varying batch sizes, node (floating-point) arithmetic performance, node memory bandwidth, network link bandwidth, and cluster dimension unveil some crucial bottlenecks at both GPU and cluster level. To provide evidence of this analysis, we validate the accuracy of the proposed model against a Python library for distributed deep learning training. [-]
Publicado en
Computing 105, 915–934 (2023)Entidad financiadora
CRUE-CSIC agreement with Springer Nature | Ministerio de Ciencia, Innovación y Universidades (Spain) | Generalitat Valenciana
Código del proyecto o subvención
TIN2017-82972-R | Prometeo/2019/109 | Plan GenT project CDEIGENT/2018/014
Título del proyecto o subvención
Open Access funding provided
Derechos de acceso
© The Author(s) 2021
info:eu-repo/semantics/openAccess
info:eu-repo/semantics/openAccess
Aparece en las colecciones
- ICC_Articles [427]