William Aidan Good
Undergraduate Research
While in undergrad, I had the opporunity to work with Professor Thiago Serra on studying the effects of neural network pruning. In the ~2 years I worked there we got 2 papers accepted, with a mostly undergraduate team, into prestegious ML conferences such as NeurIPS.
Recall Distortion in Neural Network Pruning
and the Undecayed Pruning Algorithm
Paper acepted into the NeurIPS 2022 Conference. Link to the main paper on arXiv.
Also accepted into the 2022 Sparsity in Neural Networks workshop.
Click here to view a PDF of the Poster
Abstract: Pruning techniques have been successfully used in neural networks to trade accuracy for sparsity.
However, this impact is not uniform: prior work has shown the recall for underrepresented classes in a dataset may be more negatively affected.
In this work, we study such relative distortions in recall by hypothesizing an intensification effect.
In addition, we propose a new pruning algorithm aimed at attenuating such effect.
Through statistical analysis, we have observed that intensification is less severe with our algorithm but nevertheless more pronounced with relatively more difficult tasks, less complex models, and higher pruning ratios.
Getting Away with More Network Pruning:
From Sparsity to Geometry and Linear Regions
Paper acepted into the CPAIOR 2023 Conference. Link to the main paper on arXiv.
Abstract: One surprising trait of neural networks is the extent to which their connections can be pruned with little to no effect on accuracy.
But when we cross a critical level of parameter sparsity, pruning any further leads to a sudden drop in accuracy.
This drop plausibly reflects a loss in model complexity, which we aim to avoid.
In this work, we explore how sparsity also affects the geometry of the linear regions defined by a neural network, and consequently reduces the expected maximum number of linear regions based on the architecture.
We observe that pruning affects accuracy similarly to how sparsity affects the number of linear regions and our proposed bound for the maximum number.
Conversely, we find out that selecting the sparsity across layers to maximize our bound very often improves accuracy in comparison to pruning as much with the same sparsity in all layers, thereby providing us guidance on where to prune.