AI论文集

Surveys

• 1993-TNN-Pruning Algorithms — A survey

• 2017-Proceedings of the IEEE-Efficient Processing of Deep Neural Networks: A Tutorial and Survey [2020 Book: Efficient Processing of Deep Neural Networks]

• 2017.12-A survey of FPGA-based neural network accelerator

• 2018-FITEE-Recent Advances in Efficient Computation of Deep Convolutional Neural Networks

• 2018-IEEE Signal Processing Magazine-Model compression and acceleration for deep neural networks: The principles, progress, and challenges. Arxiv extension

• 2018.8-A Survey on Methods and Theories of Quantized Neural Networks

• 2019-JMLR-Neural Architecture Search: A Survey

• 2020-MLSys-What is the state of neural network pruning

• 2019.02-The State of Sparsity in Deep Neural Networks

• 2021-TPAMI-Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks

• 2021-IJCV-Knowledge Distillation: A Survey

• 2020-Proceedings of the IEEE-Model Compression and Hardware Acceleration for Neural Networks: A Comprehensive Survey

• 2020-Pattern Recognition-Binary neural networks: A survey

• 2021-TPDS-The Deep Learning Compiler: A Comprehensive Survey

• 2021-JMLR-Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networks

• 2022-IJCAI-Recent Advances on Neural Network Pruning at Initialization

• 2021.6-Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better

Papers [Pruning and Quantization]

1980s,1990s

• 1988-NIPS-A back-propagation algorithm with optimal use of hidden units

• 1988-NIPS-Skeletonization: A Technique for Trimming the Fat from a Network via Relevance Assessment

• 1988-NIPS-What Size Net Gives Valid Generalization?

• 1989-NIPS-Dynamic Behavior of Constained Back-Propagation Networks

• 1988-NIPS-Comparing Biases for Minimal Network Construction with Back-Propagation

• 1989-NIPS-Optimal Brain Damage

• 1990-NN-A simple procedure for pruning back-propagation trained neural networks

• 1993-ICNN-Optimal Brain Surgeon and general network pruning

2000s

• 2001-JMLR-Sparse Bayesian learning and the relevance vector machine

• 2007-Book-The minimum description length principle

2011

• 2011-JMLR-Learning with Structured Sparsity

• 2011-NIPSw-Improving the speed of neural networks on CPUs

2013

• 2013-NIPS-Predicting Parameters in Deep Learning

• 2013.08-Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

2014

• 2014-BMVC-Speeding up convolutional neural networks with low rank expansions

• 2014-INTERSPEECH-1-Bit Stochastic Gradient Descent and its Application to Data-Parallel Distributed Training of Speech DNNs

• 2014-NIPS-Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluation

• 2014-NIPS-Do deep neural nets really need to be deep

• 2014.12-Memory bounded deep convolutional networks

2015

• 2015-ICLR-Speeding-up convolutional neural networks using fine-tuned cp-decomposition

• 2015-ICML-Compressing neural networks with the hashing trick

• 2015-INTERSPEECH-A Diversity-Penalizing Ensemble Training Method for Deep Learning

• 2015-BMVC-Data-free parameter pruning for deep neural networks

• 2015-BMVC-Learning the structure of deep architectures using l1 regularization

• 2015-NIPS-Learning both Weights and Connections for Efficient Neural Network

• 2015-NIPS-Binaryconnect: Training deep neural networks with binary weights during propagations

• 2015-NIPS-Structured Transforms for Small-Footprint Deep Learning

• 2015-NIPS-Tensorizing Neural Networks

• 2015-NIPSw-Distilling Intractable Generative Models

• 2015-NIPSw-Federated Optimization:Distributed Optimization Beyond the Datacenter

• 2015-CVPR-Efficient and Accurate Approximations of Nonlinear Convolutional Networks [2016 TPAMI version: Accelerating Very Deep Convolutional Networks for Classification and Detection]

• 2015-CVPR-Sparse Convolutional Neural Networks

• 2015-ICCV-An Exploration of Parameter Redundancy in Deep Networks with Circulant Projections

• 2015.12-Exploiting Local Structures with the Kronecker Layer in Convolutional Networks

2016

• 2016-ICLR-Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding [Best paper!]

• 2016-ICLR-All you need is a good init [Code]

• 2016-ICLR-Data-dependent Initializations of Convolutional Neural Networks [Code]

• 2016-ICLR-Convolutional neural networks with low-rank regularization [Code]

• 2016-ICLR-Diversity networks

• 2016-ICLR-Neural networks with few multiplications

• 2016-ICLR-Compression of deep convolutional neural networks for fast and low power mobile applications

• 2016-ICLRw-Randomout: Using a convolutional gradient norm to win the filter lottery

• 2016-CVPR-Fast algorithms for convolutional neural networks

• 2016-CVPR-Fast ConvNets Using Group-wise Brain Damage

• 2016-BMVC-Learning neural network architectures using backpropagation

• 2016-ECCV-Less is more: Towards compact cnns

• 2016-EMNLP-Sequence-Level Knowledge Distillation

• 2016-NIPS-Learning Structured Sparsity in Deep Neural Networks [Caffe Code]

• 2016-NIPS-Dynamic Network Surgery for Efficient DNNs [Caffe Code]

• 2016-NIPS-Learning the Number of Neurons in Deep Neural Networks

• 2016-NIPS-Memory-Efficient Backpropagation Through Time

• 2016-NIPS-PerforatedCNNs: Acceleration through Elimination of Redundant Convolutions

• 2016-NIPS-LightRNN: Memory and Computation-Efficient Recurrent Neural Networks

• 2016-NIPS-CNNpack: packing convolutional neural networks in the frequency domain

• 2016-ISCA-Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for Convolutional Neural Networks

• 2016-ICASSP-Learning compact recurrent neural networks

• 2016-CoNLL-Compression of Neural Machine Translation Models via Pruning

• 2016.03-Adaptive Computation Time for Recurrent Neural Networks

• 2016.06-Structured Convolution Matrices for Energy-efficient Deep learning

• 2016.06-Deep neural networks are robust to weight binarization and other non-linear distortions

• 2016.06-Hypernetworks

• 2016.07-IHT-Training skinny deep neural networks with iterative hard thresholding methods

• 2016.08-Recurrent Neural Networks With Limited Numerical Precision

• 2016.10-Deep model compression: Distilling knowledge from noisy teachers

• 2016.10-Federated Optimization: Distributed Machine Learning for On-Device Intelligence

• 2016.11-Alternating Direction Method of Multipliers for Sparse Convolutional Neural Networks

2017

• 2017-ICLR-Pruning Filters for Efficient ConvNets [PyTorch Reimpl. #1] [PyTorch Reimpl. #2]

• 2017-ICLR-Pruning Convolutional Neural Networks for Resource Efficient Inference

• 2017-ICLR-Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights [Code]

• 2017-ICLR-Do Deep Convolutional Nets Really Need to be Deep and Convolutional?

• 2017-ICLR-DSD: Dense-Sparse-Dense Training for Deep Neural Networks

• 2017-ICLR-Faster CNNs with Direct Sparse Convolutions and Guided Pruning

• 2017-ICLR-Towards the Limit of Network Quantization

• 2017-ICLR-Loss-aware Binarization of Deep Networks

• 2017-ICLR-Trained Ternary Quantization [Code]

• 2017-ICLR-Exploring Sparsity in Recurrent Neural Networks

• 2017-ICLR-Soft Weight-Sharing for Neural Network Compression [Reddit discussion] [Code]

• 2017-ICLR-Variable Computation in Recurrent Neural Networks

• 2017-ICLR-Training Compressed Fully-Connected Networks with a Density-Diversity Penalty

• 2017-ICML-Theoretical Properties for Neural Networks with Weight Matrices of Low Displacement Rank

• 2017-ICML-Deep Tensor Convolution on Multicores

• 2017-ICML-Delta Networks for Optimized Recurrent Network Computation

• 2017-ICML-Beyond Filters: Compact Feature Map for Portable Deep Model

• 2017-ICML-Combined Group and Exclusive Sparsity for Deep Neural Networks

• 2017-ICML-MEC: Memory-efficient Convolution for Deep Neural Network

• 2017-ICML-Deciding How to Decide: Dynamic Routing in Artificial Neural Networks

• 2017-ICML-ZipML: Training Models with End-to-End Low Precision: The Cans, the Cannots, and a Little Bit of Deep Learning

• 2017-ICML-Analytical Guarantees on Numerical Precision of Deep Neural Networks

• 2017-ICML-Adaptive Neural Networks for Efficient Inference

• 2017-ICML-SplitNet: Learning to Semantically Split Deep Networks for Parameter Reduction and Model Parallelization

• 2017-CVPR-Learning deep CNN denoiser prior for image restoration

• 2017-CVPR-Deep roots: Improving cnn efficiency with hierarchical filter groups

• 2017-CVPR-More is less: A more complicated network with less inference complexity [PyTorch Code]

• 2017-CVPR-All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation

• 2017-CVPR-ResNeXt-Aggregated Residual Transformations for Deep Neural Networks

• 2017-CVPR-Xception: Deep learning with depthwise separable convolutions

• 2017-CVPR-Designing Energy-Efficient CNN using Energy-aware Pruning

• 2017-CVPR-Spatially Adaptive Computation Time for Residual Networks

• 2017-CVPR-Network Sketching: Exploiting Binary Structure in Deep CNNs

• 2017-CVPR-A Compact DNN: Approaching GoogLeNet-Level Accuracy of Classification and Domain Adaptation

• 2017-ICCV-Channel pruning for accelerating very deep neural networks [Caffe Code]

• 2017-ICCV-Learning efficient convolutional networks through network slimming [PyTorch Code]

• 2017-ICCV-ThiNet: A filter level pruning method for deep neural network compression [Project] [Caffe Code] [2018 TPAMI version]

• 2017-ICCV-Interleaved group convolutions

• 2017-ICCV-Coordinating Filters for Faster Deep Neural Networks [Caffe Code]

• 2017-ICCV-Performance Guaranteed Network Acceleration via High-Order Residual Quantization

• 2017-NIPS-Net-trim: Convex pruning of deep neural networks with performance guarantee [Code] (Journal version: 2020-SIAM-Fast Convex Pruning of Deep Neural Networks)

• 2017-NIPS-Runtime neural pruning

• 2017-NIPS-Learning to Prune Deep Neural Networks via Layer-wise Optimal Brain Surgeon [Code]

• 2017-NIPS-Federated Multi-Task Learning

• 2017-NIPS-Towards Accurate Binary Convolutional Neural Network

• 2017-NIPS-Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations

• 2017-NIPS-TernGrad: Ternary Gradients to Reduce Communication in Distributed Deep Learning

• 2017-NIPS-Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networks

• 2017-NIPS-Training Quantized Nets: A Deeper Understanding

• 2017-NIPS-The Reversible Residual Network: Backpropagation Without Storing Activations [Code]

• 2017-NIPS-Compression-aware Training of Deep Networks

• 2017-FPGA-ESE: efficient speech recognition engine with compressed LSTM on FPGA [Best paper!]

• 2017-AISTATS-Communication-Efficient Learning of Deep Networks from Decentralized Data

• 2017-ICASSP-Accelerating Deep Convolutional Networks using low-precision and sparsity

• 2017-NNs-Nonredundant sparse feature extraction using autoencoders with receptive fields clustering

• 2017.02-The Power of Sparsity in Convolutional Neural Networks

• 2017.07-Stochastic, Distributed and Federated Optimization for Machine Learning

• 2017.05-Structural Compression of Convolutional Neural Networks Based on Greedy Filter Pruning

• 2017.07-Extremely Low Bit Neural Network: Squeeze the Last Bit Out with ADMM

• 2017.11-GPU Kernels for Block-Sparse Weights [Code] (OpenAI)

• 2017.11-Block-sparse recurrent neural networks

2018

• 2018-AAAI-Auto-balanced Filter Pruning for Efficient Convolutional Neural Networks

• 2018-AAAI-Deep Neural Network Compression with Single and Multiple Level Quantization

• 2018-AAAI-Dynamic Deep Neural Networks_Optimizing Accuracy-Efficiency Trade-offs by Selective Execution

• 2018-ICLRo-Training and Inference with Integers in Deep Neural Networks

• 2018-ICLR-Rethinking the Smaller-Norm-Less-Informative Assumption in Channel Pruning of Convolution Layers

• 2018-ICLR-N2N learning: Network to Network Compression via Policy Gradient Reinforcement Learning

• 2018-ICLR-Model compression via distillation and quantization

• 2018-ICLR-Towards Image Understanding from Deep Compression Without Decoding

• 2018-ICLR-Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training

• 2018-ICLR-Mixed Precision Training of Convolutional Neural Networks using Integer Operations

• 2018-ICLR-Mixed Precision Training

• 2018-ICLR-Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracy

• 2018-ICLR-Loss-aware Weight Quantization of Deep Networks

• 2018-ICLR-Alternating Multi-bit Quantization for Recurrent Neural Networks

• 2018-ICLR-Adaptive Quantization of Neural Networks

• 2018-ICLR-Variational Network Quantization

• 2018-ICLR-Espresso: Efficient Forward Propagation for Binary Deep Neural Networks

• 2018-ICLR-Learning to share: Simultaneous parameter tying and sparsification in deep learning

• 2018-ICLR-Learning Sparse Neural Networks through L0 Regularization

• 2018-ICLR-WRPN: Wide Reduced-Precision Networks

• 2018-ICLR-Deep rewiring: Training very sparse deep networks

• 2018-ICLR-Efficient sparse-winograd convolutional neural networks [Code]

• 2018-ICLR-Learning Intrinsic Sparse Structures within Long Short-term Memory

• 2018-ICLR-Multi-scale dense networks for resource efficient image classification

• 2018-ICLR-Compressing Word Embedding via Deep Compositional Code Learning

• 2018-ICLR-Learning Discrete Weights Using the Local Reparameterization Trick

• 2018-ICLR-Training wide residual networks for deployment using a single bit for each weight

• 2018-ICLR-The High-Dimensional Geometry of Binary Neural Networks

• 2018-ICLRw-To Prune, or Not to Prune: Exploring the Efficacy of Pruning for Model Compression (Similar topic: 2018-NIPSw-nip in the bud, 2018-NIPSw-rethink)

• 2018-CVPR-Context-Aware Deep Feature Compression for High-Speed Visual Tracking

• 2018-CVPR-NISP: Pruning Networks using Neuron Importance Score Propagation

• 2018-CVPR-Condensenet: An efficient densenet using learned group convolutions [Code]

• 2018-CVPR-Shift: A zero flop, zero parameter alternative to spatial convolutions

• 2018-CVPR-Explicit Loss-Error-Aware Quantization for Low-Bit Deep Neural Networks

• 2018-CVPR-Interleaved structured sparse convolutional neural networks

• 2018-CVPR-Towards Effective Low-bitwidth Convolutional Neural Networks

• 2018-CVPR-CLIP-Q: Deep Network Compression Learning by In-Parallel Pruning-Quantization

• 2018-CVPR-Blockdrop: Dynamic inference paths in residual networks

• 2018-CVPR-Nestednet: Learning nested sparse structures in deep neural networks

• 2018-CVPR-Stochastic downsampling for cost-adjustable inference and improved regularization in convolutional networks

• 2018-CVPR-Wide Compression: Tensor Ring Nets

• 2018-CVPR-Learning Compact Recurrent Neural Networks With Block-Term Tensor Decomposition

• 2018-CVPR-Learning Time/Memory-Efficient Deep Architectures With Budgeted Super Networks

• 2018-CVPR-HydraNets: Specialized Dynamic Architectures for Efficient Inference

• 2018-CVPR-SYQ: Learning Symmetric Quantization for Efficient Deep Neural Networks

• 2018-CVPR-Towards Effective Low-Bitwidth Convolutional Neural Networks

• 2018-CVPR-Two-Step Quantization for Low-Bit Neural Networks

• 2018-CVPR-Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference

• 2018-CVPR-“Learning-Compression” Algorithms for Neural Net Pruning

• 2018-CVPR-PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning [Code]

• 2018-CVPR-MorphNet: Fast & Simple Resource-Constrained Structure Learning of Deep Networks [Code]

• 2018-CVPR-ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices

• 2018-CVPRw-Squeezenext: Hardware-aware neural network design

• 2018-IJCAI-Efficient DNN Neuron Pruning by Minimizing Layer-wise Nonlinear Reconstruction Error

• 2018-IJCAI-Soft Filter Pruning for Accelerating Deep Convolutional Neural Networks [PyTorch Code]

• 2018-IJCAI-Where to Prune: Using LSTM to Guide End-to-end Pruning

• 2018-IJCAI-Accelerating Convolutional Networks via Global & Dynamic Filter Pruning

• 2018-IJCAI-Optimization based Layer-wise Magnitude-based Pruning for DNN Compression

• 2018-IJCAI-Progressive Blockwise Knowledge Distillation for Neural Network Acceleration

• 2018-IJCAI-Complementary Binary Quantization for Joint Multiple Indexing

• 2018-ICML-Compressing Neural Networks using the Variational Information Bottleneck

• 2018-ICML-DCFNet: Deep Neural Network with Decomposed Convolutional Filters

• 2018-ICML-Deep k-Means Re-Training and Parameter Sharing with Harder Cluster Assignments for Compressing Deep Convolutions

• 2018-ICML-Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization

• 2018-ICML-High Performance Zero-Memory Overhead Direct Convolutions

• 2018-ICML-Kronecker Recurrent Units

• 2018-ICML-Weightless: Lossy weight encoding for deep neural network compression

• 2018-ICML-StrassenNets: Deep learning with a multiplication budget

• 2018-ICML-Learning Compact Neural Networks with Regularization

• 2018-ICML-WSNet: Compact and Efficient Networks Through Weight Sampling

• 2018-ICML-Gradually Updated Neural Networks for Large-Scale Image Recognition [Code]

• 2018-ICML-On the Optimization of Deep Networks: Implicit Acceleration by Overparameterization

• 2018-ICML-Understanding and simplifying one-shot architecture search

• 2018-ECCV-A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multipliers [Code]

• 2018-ECCV-Coreset-Based Neural Network Compression

• 2018-ECCV-Data-Driven Sparse Structure Selection for Deep Neural Networks [MXNet Code]

• 2018-ECCV-Training Binary Weight Networks via Semi-Binary Decomposition

• 2018-ECCV-Learning Compression from Limited Unlabeled Data

• 2018-ECCV-Constraint-Aware Deep Neural Network Compression

• 2018-ECCV-Sparsely Aggregated Convolutional Networks

• 2018-ECCV-Deep Expander Networks: Efficient Deep Networks from Graph Theory [Code]

• 2018-ECCV-SparseNet-Sparsely Aggregated Convolutional Networks [Code]

• 2018-ECCV-Ask, acquire, and attack: Data-free uap generation using class impressions

• 2018-ECCV-Netadapt: Platform-aware neural network adaptation for mobile applications

• 2018-ECCV-Clustering Convolutional Kernels to Compress Deep Neural Networks

• 2018-ECCV-Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm

• 2018-ECCV-Extreme Network Compression via Filter Group Approximation

• 2018-ECCV-Convolutional Networks with Adaptive Inference Graphs

• 2018-ECCV-SkipNet: Learning Dynamic Routing in Convolutional Networks [Code]

• 2018-ECCV-Value-aware Quantization for Training and Inference of Neural Networks

• 2018-ECCV-LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks

• 2018-ECCV-AMC: AutoML for Model Compression and Acceleration on Mobile Devices

• 2018-ECCV-Piggyback: Adapting a single network to multiple tasks by learning to mask weights

• 2018-BMVCo-Structured Probabilistic Pruning for Convolutional Neural Network Acceleration

• 2018-BMVC-Efficient Progressive Neural Architecture Search

• 2018-BMVC-Igcv3: Interleaved lowrank group convolutions for efficient deep neural networks

• 2018-NIPS-Discrimination-aware Channel Pruning for Deep Neural Networks

• 2018-NIPS-Frequency-Domain Dynamic Pruning for Convolutional Neural Networks

• 2018-NIPS-ChannelNets: Compact and Efficient Convolutional Neural Networks via Channel-Wise Convolutions

• 2018-NIPS-DropBlock: A regularization method for convolutional networks

• 2018-NIPS-Constructing fast network through deconstruction of convolution

• 2018-NIPS-Learning Versatile Filters for Efficient Convolutional Neural Networks [Code]

• 2018-NIPS-Moonshine: Distilling with cheap convolutions

• 2018-NIPS-HitNet: hybrid ternary recurrent neural network

• 2018-NIPS-FastGRNN: A Fast, Accurate, Stable and Tiny Kilobyte Sized Gated Recurrent Neural Network

• 2018-NIPS-Training DNNs with Hybrid Block Floating Point

• 2018-NIPS-Reversible Recurrent Neural Networks

• 2018-NIPS-Synaptic Strength For Convolutional Neural Network

• 2018-NIPS-Learning sparse neural networks via sensitivity-driven regularization

• 2018-NIPS-Multi-Task Zipping via Layer-wise Neuron Sharing

• 2018-NIPS-A Linear Speedup Analysis of Distributed Deep Learning with Sparse and Quantized Communication

• 2018-NIPS-Gradient Sparsification for Communication-Efficient Distributed Optimization

• 2018-NIPS-GradiVeQ: Vector Quantization for Bandwidth-Efficient Gradient Aggregation in Distributed CNN Training

• 2018-NIPS-ATOMO: Communication-efficient Learning via Atomic Sparsification

• 2018-NIPS-Norm matters: efficient and accurate normalization schemes in deep networks

• 2018-NIPS-Sparsified SGD with memory

• 2018-NIPS-Pelee: A Real-Time Object Detection System on Mobile Devices

• 2018-NIPS-Scalable methods for 8-bit training of neural networks

• 2018-NIPS-TETRIS: TilE-matching the TRemendous Irregular Sparsity

• 2018-NIPS-Training deep neural networks with 8-bit floating point numbers

• 2018-NIPS-Multiple instance learning for efficient sequential data classification on resource-constrained devices

• 2018-NIPS-Sparse dnns with improved adversarial robustness

• 2018-NIPSw-Pruning neural networks: is it time to nip it in the bud?

• 2018-NIPSw-Rethinking the Value of Network Pruning [2019 ICLR version] [PyTorch Code]

• 2018-NIPSw-Structured Pruning for Efficient ConvNets via Incremental Regularization [2019 IJCNN version] [Caffe Code]

• 2018-NIPSw-Adaptive Mixture of Low-Rank Factorizations for Compact Neural Modeling

• 2018-NIPSw-Learning Sparse Networks Using Targeted Dropout [OpenReview] [Code]

• 2018-WACV-Recovering from Random Pruning: On the Plasticity of Deep Convolutional Neural Networks

• 2018.05-Compression of Deep Convolutional Neural Networks under Joint Sparsity Constraints

• 2018.05-AutoPruner: An End-to-End Trainable Filter Pruning Method for Efficient Deep Model Inference

• 2018.10-A Closer Look at Structured Pruning for Neural Network Compression [Code]

• 2018.11-Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs

• 2018.11-PydMobileNet: Improved Version of MobileNets with Pyramid Depthwise Separable Convolution

2019

• 2019-MLSys-Towards Federated Learning at Scale: System Design

• 2019-MLsys-To compress or not to compress: Understanding the Interactions between Adversarial Attacks and Neural Network Compression

• 2019-ICLR-Slimmable Neural Networks [Code]

• 2019-ICLR-Defensive Quantization: When Efficiency Meets Robustness

• 2019-ICLR-Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters [Code]

• 2019-ICLR-ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware [Code]

• 2019-ICLR-SNIP: Single-shot Network Pruning based on Connection Sensitivity

• 2019-ICLR-Non-vacuous Generalization Bounds at the ImageNet Scale: a PAC-Bayesian Compression Approach

• 2019-ICLR-Dynamic Channel Pruning: Feature Boosting and Suppression

• 2019-ICLR-Energy-Constrained Compression for Deep Neural Networks via Weighted Sparse Projection and Layer Input Masking

• 2019-ICLR-RotDCF: Decomposition of Convolutional Filters for Rotation-Equivariant Deep Networks

• 2019-ICLR-Dynamic Sparse Graph for Efficient Deep Learning

• 2019-ICLR-Big-Little Net: An Efficient Multi-Scale Feature Representation for Visual and Speech Recognition

• 2019-ICLR-Data-Dependent Coresets for Compressing Neural Networks with Applications to Generalization Bounds

• 2019-ICLR-Learning Recurrent Binary/Ternary Weights

• 2019-ICLR-Double Viterbi: Weight Encoding for High Compression Ratio and Fast On-Chip Reconstruction for Deep Neural Network

• 2019-ICLR-Relaxed Quantization for Discretized Neural Networks

• 2019-ICLR-Integer Networks for Data Compression with Latent-Variable Models

• 2019-ICLR-Minimal Random Code Learning: Getting Bits Back from Compressed Model Parameters

• 2019-ICLR-Analysis of Quantized Models

• 2019-ICLR-DARTS: Differentiable Architecture Search [Code]

• 2019-ICLR-Graph HyperNetworks for Neural Architecture Search

• 2019-ICLR-Learnable Embedding Space for Efficient Neural Architecture Compression [Code]

• 2019-ICLR-Efficient Multi-Objective Neural Architecture Search via Lamarckian Evolution

• 2019-ICLR-SNAS: stochastic neural architecture search

• 2019-AAAIo-A layer decomposition-recomposition framework for neuron pruning towards accurate lightweight networks

• 2019-AAAI-Balanced Sparsity for Efficient DNN Inference on GPU [Code]

• 2019-AAAI-CircConv: A Structured Convolution with Low Complexity

• 2019-AAAI-Regularized Evolution for Image Classifier Architecture Search

• 2019-AAAI-Universal Approximation Property and Equivalence of Stochastic Computing-Based Neural Networks and Binary Neural Networks

• 2019-WACV-DAC: Data-free Automatic Acceleration of Convolutional Networks

• 2019-ASPLOS-Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimization

• 2019-CVPRo-HAQ: hardware-aware automated quantization

• 2019-CVPRo-Filter Pruning via Geometric Median for Deep Convolutional Neural Networks Acceleration [Code]

• 2019-CVPR-All You Need is a Few Shifts: Designing Efficient Convolutional Neural Networks for Image Classification

• 2019-CVPR-Importance Estimation for Neural Network Pruning [Code]

• 2019-CVPR-HetConv Heterogeneous Kernel-Based Convolutions for Deep CNNs

• 2019-CVPR-Fully Learnable Group Convolution for Acceleration of Deep Neural Networks

• 2019-CVPR-Towards Optimal Structured CNN Pruning via Generative Adversarial Learning

• 2019-CVPR-ChamNet: Towards Efficient Network Design through Platform-Aware Model Adaptation

• 2019-CVPR-Partial Order Pruning: for Best Speed/Accuracy Trade-off in Neural Architecture Search [Code]

• 2019-CVPR-Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation [Code]

• 2019-CVPR-MnasNet: Platform-Aware Neural Architecture Search for Mobile [Code]

• 2019-CVPR-MFAS: Multimodal Fusion Architecture Search

• 2019-CVPR-A Neurobiological Evaluation Metric for Neural Network Model Search

• 2019-CVPR-Fast Neural Architecture Search of Compact Semantic Segmentation Models via Auxiliary Cells

• 2019-CVPR-Efficient Neural Network Compression [Code]

• 2019-CVPR-T-Net: Parametrizing Fully Convolutional Nets with a Single High-Order Tensor

• 2019-CVPR-Centripetal SGD for Pruning Very Deep Convolutional Networks with Complicated Structure [Code]

• 2019-CVPR-DSC: Dense-Sparse Convolution for Vectorized Inference of Convolutional Neural Networks

• 2019-CVPR-DupNet: Towards Very Tiny Quantized CNN With Improved Accuracy for Face Detection

• 2019-CVPR-ECC: Platform-Independent Energy-Constrained Deep Neural Network Compression via a Bilinear Regression Model

• 2019-CVPR-Variational Convolutional Neural Network Pruning

• 2019-CVPR-Accelerating Convolutional Neural Networks via Activation Map Compression

• 2019-CVPR-Compressing Convolutional Neural Networks via Factorized Convolutional Filters

• 2019-CVPR-Deep Virtual Networks for Memory Efficient Inference of Multiple Tasks

• 2019-CVPR-Exploiting Kernel Sparsity and Entropy for Interpretable CNN Compression

• 2019-CVPR-MBS: Macroblock Scaling for CNN Model Reduction

• 2019-CVPR-On Implicit Filter Level Sparsity in Convolutional Neural Networks

• 2019-CVPR-Structured Pruning of Neural Networks With Budget-Aware Regularization

• 2019-CVPRo-Neural Rejuvenation: Improving Deep Network Training by Enhancing Computational Resource Utilization [Code]

• 2019-ICML-Approximated Oracle Filter Pruning for Destructive CNN Width Optimization [Code]

• 2019-ICML-EigenDamage: Structured Pruning in the Kronecker-Factored Eigenbasis [PyTorch Code]

• 2019-ICML-Zero-Shot Knowledge Distillation in Deep Networks [Code]

• 2019-ICML-LegoNet: Efficient Convolutional Neural Networks with Lego Filters [Code]

• 2019-ICML-EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks [Code]

• 2019-ICML-Collaborative Channel Pruning for Deep Networks

• 2019-ICML-Training CNNs with Selective Allocation of Channels

• 2019-ICML-NAS-Bench-101: Towards Reproducible Neural Architecture Search [Code]

• 2019-ICML-Learning fast algorithms for linear transforms using butterfly factorizations

• 2019-ICMLw-Towards Learning of Filter-Level Heterogeneous Compression of Convolutional Neural Networks [Code] (AutoML workshop)

• 2019-IJCAI-Play and Prune: Adaptive Filter Pruning for Deep Model Compression

• 2019-BigComp-Towards Robust Compressed Convolutional Neural Networks

• 2019-ICCV-Rethinking ImageNet Pre-training

• 2019-ICCV-Universally Slimmable Networks and Improved Training Techniques

• 2019-ICCV-MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning [Code]

• 2019-ICCV-Progressive Differentiable Architecture Search: Bridging the Depth Gap between Search and Evaluation [Code]

• 2019-ICCV-Data-Free Quantization through Weight Equalization and Bias Correction

• 2019-ICCV-ACNet: Strengthening the Kernel Skeletons for Powerful CNN via Asymmetric Convolution Blocks

• 2019-ICCV-Adversarial Robustness vs. Model Compression, or Both? [PyTorch Code]

• 2019-NIPS-Global Sparse Momentum SGD for Pruning Very Deep Neural Networks

• 2019-NIPS-Model Compression with Adversarial Robustness: A Unified Optimization Framework

• 2019-NIPS-AutoPrune: Automatic Network Pruning by Regularizing Auxiliary Parameters

• 2019-NIPS-Double Quantization for Communication-Efficient Distributed Optimization

• 2019-NIPS-Focused Quantization for Sparse CNNs

• 2019-NIPS-E2-Train: Training State-of-the-art CNNs with Over 80% Energy Savings

• 2019-NIPS-MetaQuant: Learning to Quantize by Learning to Penetrate Non-differentiable Quantization

• 2019-NIPS-Random Projections with Asymmetric Quantization

• 2019-NIPS-Network Pruning via Transformable Architecture Search [Code]

• 2019-NIPS-Point-Voxel CNN for Efficient 3D Deep Learning [Code]

• 2019-NIPS-Gate Decorator: Global Filter Pruning Method for Accelerating Deep Convolutional Neural Networks [PyTorch Code]

• 2019-NIPS-A Mean Field Theory of Quantized Deep Networks: The Quantization-Depth Trade-Off

• 2019-NIPS-Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification and Local Computations

• 2019-NIPS-Post training 4-bit quantization of convolutional networks for rapid-deployment

• 2019-PR-Filter-in-Filter: Improve CNNs in a Low-cost Way by Sharing Parameters among the Sub-filters of a Filter

• 2019-PRL-BDNN: Binary Convolution Neural Networks for Fast Object Detection

• 2019-TNNLS-Towards Compact ConvNets via Structure-Sparsity Regularized Filter Pruning [Code]

• 2019.03-Network Slimming by Slimmable Networks: Towards One-Shot Architecture Search for Channel Numbers [Code]

• 2019.03-Single Path One-Shot Neural Architecture Search with Uniform Sampling

• 2019.04-Resource Efficient 3D Convolutional Neural Networks

• 2019.04-Meta Filter Pruning to Accelerate Deep Convolutional Neural Networks

• 2019.04-Knowledge Squeezed Adversarial Network Compression

• 2019.05-Dynamic Neural Network Channel Execution for Efficient Training

• 2019.06-AutoGrow: Automatic Layer Growing in Deep Convolutional Networks

• 2019.06-BasisConv: A method for compressed representation and learning in CNNs

• 2019.06-BlockSwap: Fisher-guided Block Substitution for Network Compression

• 2019.06-Separable Layers Enable Structured Efficient Linear Substitutions [Code]

• 2019.06-Butterfly Transform: An Efficient FFT Based Neural Architecture Design

• 2019.06-A Taxonomy of Channel Pruning Signals in CNNs

• 2019.08-Adversarial Neural Pruning with Latent Vulnerability Suppression

• 2019.09-Training convolutional neural networks with cheap convolutions and online distillation

• 2019.09-Pruning from Scratch

• 2019.11-Adversarial Interpolation Training: A Simple Approach for Improving Model Robustness

• 2019.11-A Programmable Approach to Model Compression [Code]

2020

• 2020-AAAI-Pconv: The missing but desirable sparsity in dnn weight pruning for real-time execution on mobile devices

• 2020-AAAI-Channel Pruning Guided by Classification Loss and Feature Importance

• 2020-AAAI-Pruning from Scratch

• 2020-AAAI-Harmonious Coexistence of Structured Weight Pruning and Ternarization for Deep Neural Networks

• 2020-AAAI-AutoCompress: An Automatic DNN Structured Pruning Framework for Ultra-High Compression Rates

• 2020-AAAI-DARB: A Density-Adaptive Regular-Block Pruning for Deep Neural Networks

• 2020-AAAI-Real-Time Object Tracking via Meta-Learning: Efficient Model Adaptation and One-Shot Channel Pruning

• 2020-AAAI-Dynamic Network Pruning with Interpretable Layerwise Channel Selection

• 2020-AAAI-Reborn Filters: Pruning Convolutional Neural Networks with Limited Data

• 2020-AAAI-Layerwise Sparse Coding for Pruned Deep Neural Networks with Extreme Compression Ratio

• 2020-AAAI-Sparsity-inducing Binarized Neural Networks

• 2020-AAAI-Structured Sparsification of Gated Recurrent Neural Networks

• 2020-AAAI-Hierarchical Knowledge Squeezed Adversarial Network Compression

• 2020-AAAI-Embedding Compression with Isotropic Iterative Quantization

• 2020-ICLR-Comparing Rewinding and Fine-tuning in Neural Network Pruning [Code]

• 2020-ICLR-Lookahead: A Far-sighted Alternative of Magnitude-based Pruning [Code]

• 2020-ICLR-Dynamic Model Pruning with Feedback

• 2020-ICLR-Provable Filter Pruning for Efficient Neural Networks

• 2020-ICLR-Data-Independent Neural Pruning via Coresets

• 2020-ICLR-FSNet: Compression of Deep Convolutional Neural Networks by Filter Summary

• 2020-ICLR-Probabilistic Connection Importance Inference and Lossless Compression of Deep Neural Networks

• 2020-ICLR-Neural Epitome Search for Architecture-Agnostic Network Compression

• 2020-ICLR-One-Shot Pruning of Recurrent Neural Networks by Jacobian Spectrum Evaluation

• 2020-ICLR-DeepHoyer: Learning Sparser Neural Network with Differentiable Scale-Invariant Sparsity Measures [Code]

• 2020-ICLR-Dynamic Sparse Training: Find Efficient Sparse Network From Scratch With Trainable Masked Layers

• 2020-ICLR-Scalable Model Compression by Entropy Penalized Reparameterization

• 2020-ICLR-A Signal Propagation Perspective for Pruning Neural Networks at Initialization

• 2020-CVPR-GhostNet: More Features from Cheap Operations [Code]

• 2020-CVPR-Filter Grafting for Deep Neural Networks

• 2020-CVPR-Low-rank Compression of Neural Nets: Learning the Rank of Each Layer

• 2020-CVPR-Structured Compression by Weight Encryption for Unstructured Pruning and Quantization

• 2020-CVPR-Learning Filter Pruning Criteria for Deep Convolutional Neural Networks Acceleration

• 2020-CVPR-APQ: Joint Search for Network Architecture, Pruning and Quantization Policy

• 2020-CVPR-Group Sparsity: The Hinge Between Filter Pruning and Decomposition for Network Compression [Code]

• 2020-CVPR-Neural Network Pruning With Residual-Connections and Limited-Data

• 2020-CVPR-Multi-Dimensional Pruning: A Unified Framework for Model Compression

• 2020-CVPR-Discrete Model Compression With Resource Constraint for Deep Neural Networks

• 2020-CVPR-Automatic Neural Network Compression by Sparsity-Quantization Joint Learning: A Constrained Optimization-Based Approach

• 2020-CVPR-Low-Rank Compression of Neural Nets: Learning the Rank of Each Layer

• 2020-CVPR-The Knowledge Within: Methods for Data-Free Model Compression

• 2020-CVPR-GAN Compression: Efficient Architectures for Interactive Conditional GANs [Code]

• 2020-CVPR-Few Sample Knowledge Distillation for Efficient Network Compression

• 2020-CVPR-Fast sparse convnets

• 2020-CVPR-Structured Multi-Hashing for Model Compression

• 2020-CVPRo-AdderNet: Do We Really Need Multiplications in Deep Learning? [Code]

• 2020-CVPRo-Towards Efficient Model Compression via Learned Global Ranking [Code]

• 2020-CVPRo-HRank: Filter Pruning Using High-Rank Feature Map [Code]

• 2020-CVPRo-DaST: Data-free Substitute Training for Adversarial Attacks [Code]

• 2020-ICML-PENNI: Pruned Kernel Sharing for Efficient CNN Inference [Code]

• 2020-ICML-Operation-Aware Soft Channel Pruning using Differentiable Masks

• 2020-ICML-DropNet: Reducing Neural Network Complexity via Iterative Pruning

• 2020-ICML-Network Pruning by Greedy Subnetwork Selection

• 2020-ICML-AutoGAN-Distiller: Searching to Compress Generative Adversarial Networks

• 2020-ICML-Soft Threshold Weight Reparameterization for Learnable Sparsity [PyTorch Code]

• 2020-EMNLP-Structured Pruning of Large Language Models [Code]

• 2020-NIPS-Pruning neural networks without any data by iteratively conserving synaptic flow

• 2020-NIPS-Neuron-level Structured Pruning using Polarization Regularizer

• 2020-NIPS-SCOP: Scientific Control for Reliable Neural Network Pruning

• 2020-NIPS-Directional Pruning of Deep Neural Networks

• 2020-NIPS-Storage Efficient and Dynamic Flexible Runtime Channel Pruning via Deep Reinforcement Learning

• 2020-NIPS-Pruning Filter in Filter

• 2020-NIPS-HYDRA: Pruning Adversarially Robust Neural Networks

• 2020-NIPS-Movement Pruning: Adaptive Sparsity by Fine-Tuning

• 2020-NIPS-Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot

• 2020-NIPS-Position-based Scaled Gradient for Model Quantization and Pruning

• 2020-NIPS-The Generalization-Stability Tradeoff In Neural Network Pruning

• 2020-NIPS-FleXOR: Trainable Fractional Quantization

• 2020-NIPS-Adaptive Gradient Quantization for Data-Parallel SGD

• 2020-NIPS-Robust Quantization: One Model to Rule Them All

• 2020-NIPS-HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks

• 2020-NIPS-Efficient Exact Verification of Binarized Neural Networks

• 2020-NIPS-Ultra-Low Precision 4-bit Training of Deep Neural Networks

• 2020-NIPS-Path Sample-Analytic Gradient Estimators for Stochastic Binary Networks

2021

• 2021-WACV-CAP: Context-Aware Pruning for Semantic Segmentation [Code]

• 2021-AAAI-Few Shot Network Compression via Cross Distillation

• 2021-AAAI-Conditional Channel Pruning for Automated Model Compression [Code]

• 2021-ICLR-Neural Pruning via Growing Regularization [PyTorch Code]

• 2021-ICLR-Network Pruning That Matters: A Case Study on Retraining Variants

• 2021-ICLR-ChipNet: Budget-Aware Pruning with Heaviside Continuous Approximations

• 2021-ICLR-A Gradient Flow Framework For Analyzing Network Pruning (Spotlight)

• 2021-CVPR-Towards Compact CNNs via Collaborative Compression

• 2021-CVPR-Manifold Regularized Dynamic Network Pruning

• 2021-CVPR-Learnable Companding Quantization for Accurate Low-bit Neural Networks

• 2021-CVPR-Diversifying Sample Generation for Accurate Data-Free Quantization

• 2021-CVPR-Zero-shot Adversarial Quantization [Oral] [Code]

• 2021-CVPR-Network Quantization with Element-wise Gradient Scaling [Project]

• 2021-ICML-Group Fisher Pruning for Practical Network Compression [Code]

• 2021-ICML-Accelerate CNNs from Three Dimensions: A Comprehensive Pruning Framework

• 2021-ICML-A Probabilistic Approach to Neural Network Pruning

• 2021-ICML-On the Predictability of Pruning Across Scales

• 2021-ICML-Sparsifying Networks via Subdifferential Inclusion

• 2021-ICML-Selfish Sparse RNN Training [Code]

• 2021-ICML-Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training [Code]

• 2021-ICML-Training Adversarially Robust Sparse Networks via Bayesian Connectivity Sampling

• 2021-ICML-ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training

• 2021-ICML-Leveraging Sparse Linear Layers for Debuggable Deep Networks

• 2021-ICML-PHEW: Constructing Sparse Networks that Learn Fast and Generalize Well without Training Data

• 2021-ICML-BASE Layers: Simplifying Training of Large, Sparse Models [Code]

• 2021-ICML-Dense for the Price of Sparse: Improved Performance of Sparsely Initialized Networks via a Subspace Offset

• 2021-ICML-I-BERT: Integer-only BERT Quantization

• 2021-ICML-Training Quantized Neural Networks to Global Optimality via Semidefinite Programming

• 2021-ICML-Differentiable Dynamic Quantization with Mixed Precision and Adaptive Resolution

• 2021-ICML-Communication-Efficient Distributed Optimization with Quantized Preconditioners

• 2021-NIPS-Aligned Structured Sparsity Learning for Efficient Image Super-Resolution [Code] (Spotlight!)

• 2021-NIPS-Scatterbrain: Unifying Sparse and Low-rank Attention [Code]

• 2021.5-Dynamical Isometry: The Missing Ingredient for Neural Network Pruning

2022

• 2022-ICLR-Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models

• 2022-NIPS-Pruning has a disparate impact on model accuracy

• 2022-NIPS-Pruning Neural Networks via Coresets and Convex Geometry: Towards No Assumptions

• 2022-NIPS-Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruning

• 2022-NIPS-Data-Efficient Structured Pruning via Submodular Optimization

• 2022-NIPS-A Fast Post-Training Pruning Framework for Transformers

• 2022-NIPS-SAViT: Structure-Aware Vision Transformer Pruning via Collaborative Optimization

• 2022-NIPS-Recall Distortion in Neural Network Pruning and the Undecayed Pruning Algorithm

• 2022-NIPS-Structural Pruning via Latency-Saliency Knapsack

• 2022-NIPS-Sparse Probabilistic Circuits via Pruning and Growing

• 2022-NIPS-Back Razor: Memory-Efficient Transfer Learning by Self-Sparsified Backpropogation

• 2022-NIPS-SInGE: Sparsity via Integrated Gradients Estimation of Neuron Relevance

• 2022-NIPS-VTC-LFC: Vision Transformer Compression with Low-Frequency Components

• 2022-NIPS-Weighted Mutual Learning with Diversity-Driven Model Compression

• 2022-NIPS-Resource-Adaptive Federated Learning with All-In-One Neural Composition

• 2022-NIPS-Controlled Sparsity via Constrained Optimization or: How I Learned to Stop Tuning Penalties and Love Constraints

• 2022-NIPS-On Measuring Excess Capacity in Neural Networks

• 2022-NIPS-Prune and distill: similar reformatting of image information along rat visual cortex and deep neural networks

• 2022-NIPS-Deep Compression of Pre-trained Transformer Models

• 2022-NIPS-Sparsity in Continuous-Depth Neural Networks

• 2022-NIPS-Spartan: Differentiable Sparsity via Regularized Transportation

• 2022-NIPS-Accelerated Projected Gradient Algorithms for Sparsity Constrained Optimization Problems

• 2022-NIPS-Feature Learning in L2-regularized DNNs: Attraction/Repulsion and Sparsity

• 2022-NIPS-Learning Best Combination for Efficient N:M Sparsity

• 2022-NIPS-Accelerating Sparse Convolution with Column Vector-Wise Sparsity

• 2022-NIPS-Differentially Private Model Compression

• 2022-NIPS-Transformers meet Stochastic Block Models: Attention with Data-Adaptive Sparsity and Cost

• 2022-NIPS-A Win-win Deal: Towards Sparse and Robust Pre-trained Language Models

• 2022-NIPS-Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach

• 2022-NIPS-Learning sparse features can lead to overfitting in neural networks

• 2022-NIPS-Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis

• 2022-NIPS-EfficientFormer: Vision Transformers at MobileNet Speed

• 2022-NIPS-Revisiting Sparse Convolutional Model for Visual Recognition

• 2022-NIPS-Efficient Spatially Sparse Inference for Conditional GANs and Diffusion Models

• 2022-NIPS-A Theoretical View on Sparsely Activated Networks

• 2022-NIPS-Dynamic Sparse Network for Time Series Classification: Learning What to “See”

• 2022-NIPS-Spatial Pruned Sparse Convolution for Efficient 3D Object Detection

• 2022-NIPS-Sparse Structure Search for Delta Tuning

• 2022-NIPS-Beyond L1: Faster and Better Sparse Models with skglm

• 2022-NIPS-On the Representation Collapse of Sparse Mixture of Experts

• 2022-NIPS-M³ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design

• 2022-NIPS-On-Device Training Under 256KB Memory

• 2022-NIPS-Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning

---

Papers [Actual Acceleration via Sparsity]

• 2018-ICML-Efficient Neural Audio Synthesis

• 2018-NIPS-Tetris: Tile-matching the tremendous irregular sparsity

• 2021.4-Accelerating Sparse Deep Neural Networks (White paper from NVIDIA)

• 2021-ICLR-Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch [Code]

• 2021-NIPS-Channel Permutations for N: M Sparsity [Code: NVIDIA ASP]

• 2021-NIPS-Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks

• 2021-ICLR-Learning N:M fine-grained structured sparse neural networks from scratch [Code] [Slides]

• 2022-NIPS-UDC: Unified DNAS for Compressible TinyML Models for Neural Processing Units

---

Papers [Lottery Ticket Hypothesis (LTH)]

For LTH and other Pruning at Initialization papers, please refer to Awesome-Pruning-at-Initialization.

---

Papers [Bayesian Compression]

• 1995-Neural Computation-Bayesian Regularisation and Pruning using a Laplace Prior

• 1997-Neural Networks-Regularization with a Pruning Prior

• 2015-NIPS-Bayesian dark knowledge

• 2017-NIPS-Bayesian Compression for Deep Learning [Code]

• 2017-ICML-Variational dropout sparsifies deep neural networks

• 2017-NIPSo-Structured Bayesian Pruning via Log-Normal Multiplicative Noise

• 2017-ICMLw-Bayesian Sparsification of Recurrent Neural Networks

• 2020-NIPS-Bayesian Bits: Unifying Quantization and Pruning

Papers [Knowledge Distillation (KD)]

Before 2014

• 1996-Born again trees (proposed compressing neural networks and multipletree predictors by approximating them with a single tree)

• 2006-SIGKDD-Model compression

• 2010-ML-A theory of learning from different domains

2014

• 2014-NIPS-Do deep nets really need to be deep?

• 2014-NIPSw-Distilling the Knowledge in a Neural Network [Code]

2016

• 2016-ICLR-Net2net: Accelerating learning via knowledge transfer

• 2016-ECCV-Accelerating convolutional neural networks with dominant convolutional kernel and knowledge pre-regression

2017

• 2017-ICLR-Paying more attention to attention: Improving the performance of convolutional neural networksvia attention transfer

• 2017-ICLR-Do deep convolutional nets really need to be deep and convolutional?

• 2017-CVPR-A gift from knowledge distillation: Fast optimization, network minimization and transfer learning

• 2017-BMVC-Adapting models to signal degradation using distillation

• 2017-NIPS-Sobolev training for neural networks

• 2017-NIPS-Learning efficient object detection models with knowledge distillation

• 2017-NIPSw-Data-Free Knowledge Distillation for Deep Neural Networks [Code]

• 2017.07-Like What You Like: Knowledge Distill via Neuron Selectivity Transfer

• 2017.10-Knowledge Projection for Deep Neural Networks

• 2017.11-Distilling a Neural Network Into a Soft Decision Tree

• 2017.12-Data Distillation: Towards Omni-Supervised Learning

2018

• 2018-AAAI-DarkRank: Accelerating Deep Metric Learning via Cross Sample Similarities Transfer

• 2018-AAAI-Dynamic deep neural networks: Optimizing accuracy-efficiency trade-offs by selective execution

• 2018-AAAI-Rocket Launching: A Universal and Efficient Framework for Training Well-performing Light Net

• 2018-AAAI-Adversarial Learning of Portable Student Networks

• 2018-AAAI-Knowledge Distillation in Generations: More Tolerant Teachers Educate Better Students

• 2018-ICLR-Large scale distributed neural network training through online distillation

• 2018-CVPR-Deep mutual learning

• 2018-ICML-Born-Again Neural Networks

• 2018-IJCAI-Better and Faster: Knowledge Transfer from Multiple Self-supervised Learning Tasks via Graph Distillation for Video Classification

• 2018-ECCV-2018-ECCV-Learning deep representations with probabilistic knowledge transfer [Code]

• 2018-ECCV-Graph adaptive knowledge transfer for unsupervised domain adaptation

• 2018-SIGKDD-Towards Evolutionary Compression

• 2018-NIPS-KDGAN: knowledge distillation with generative adversarial networks [2019 TPAMI version]

• 2018-NIPS-Knowledge Distillation by On-the-Fly Native Ensemble

• 2018-NIPS-Paraphrasing Complex Network: Network Compression via Factor Transfer

• 2018-NIPSw-Variational Mutual Information Distillation for Transfer Learning workshop: continual learning

• 2018-NIPSw-Transparent Model Distillation

• 2018.03-Interpreting Deep Classifier by Visual Distillation of Dark Knowledge

• 2018.11-Dataset Distillation [Code]

• 2018.12-Learning Student Networks via Feature Embedding

• 2018.12-Few Sample Knowledge Distillation for Efficient Network Compression

2019

• 2019-AAAI-Knowledge Distillation with Adversarial Samples Supporting Decision Boundary

• 2019-AAAI-Knowledge Transfer via Distillation of Activation Boundaries Formed by Hidden Neurons [Code]

• 2019-AAAI-Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks [Code]

• 2019-CVPR-Knowledge Representing: Efficient, Sparse Representation of Prior Knowledge for Knowledge Distillation

• 2019-CVPR-Knowledge Distillation via Instance Relationship Graph

• 2019-CVPR-Variational Information Distillation for Knowledge Transfer

• 2019-CVPR-Learning Metrics from Teachers Compact Networks for Image Embedding [Code]

• 2019-ICCV-A Comprehensive Overhaul of Feature Distillation

• 2019-ICCV-Similarity-Preserving Knowledge Distillation

• 2019-ICCV-Correlation Congruence for Knowledge Distillation

• 2019-ICCV-Data-Free Learning of Student Networks

• 2019-ICCV-Learning Lightweight Lane Detection CNNs by Self Attention Distillation [Code]

• 2019-ICCV-Attention bridging network for knowledge transfer

• 2019-NIPS-Zero-shot Knowledge Transfer via Adversarial Belief Matching [Code] (spotlight)

• 2019.05-DistillHash: Unsupervised Deep Hashing by Distilling Data Pairs

2020

• 2020-ICLR-Contrastive Representation Distillation [Code]

• 2020-AAAI-A Knowledge Transfer Framework for Differentially Private Sparse Learning

• 2020-AAAI-Uncertainty-aware Multi-shot Knowledge Distillation for Image-based Object Re-identification

• 2020-AAAI-Improved Knowledge Distillation via Teacher Assistant

• 2020-AAAI-Knowledge Distillation from Internal Representations

• 2020-AAAI-Distilling Knowledge from Well-informed Soft Labels for Neural Relation Extraction

• 2020-AAAI-Online Knowledge Distillation with Diverse Peers

• 2020-AAAI-Ultrafast Video Attention Prediction with Coupled Knowledge Distillation

• 2020-AAAI-Graph Few-shot Learning via Knowledge Transfer

• 2020-AAAI-Diversity Transfer Network for Few-Shot Learning

• 2020-AAAI-Few Shot Network Compression via Cross Distillation

• 2020-ICLR-Knowledge Consistency between Neural Networks and Beyond

• 2020-ICLR-Contrastive Representation Distillation [Code]

• 2020-ICLR-BlockSwap: Fisher-guided Block Substitution for Network Compression on a Budget

• 2020-ICLR-Ensemble Distribution Distillation

• 2020-CVPR-Collaborative Distillation for Ultra-Resolution Universal Style Transfer [Code]

• 2020-CVPR-Explaining Knowledge Distillation by Quantifying the Knowledge

• 2020-CVPR-Self-training with Noisy Student improves ImageNet classification [Code]

• 2020-CVPR-Neural Networks Are More Productive Teachers Than Human Raters: Active Mixup for Data-Efficient Knowledge Distillation From a Blackbox Model

• 2020-CVPR-Heterogeneous Knowledge Distillation Using Information Flow Modeling

• 2020-CVPR-Creating Something From Nothing: Unsupervised Knowledge Distillation for Cross-Modal Hashing

• 2020-CVPR-Revisiting Knowledge Distillation via Label Smoothing Regularization

• 2020-CVPR-Distilling Knowledge From Graph Convolutional Networks

• 2020-CVPR-MineGAN: Effective Knowledge Transfer From GANs to Target Domains With Few Images [Code]

• 2020-CVPRo-Dreaming to Distill: Data-Free Knowledge Transfer via DeepInversion [Code]

• 2020-CVPR-Online Knowledge Distillation via Collaborative Learning

• 2020-CVPR-Distilling Cross-Task Knowledge via Relationship Matching

• 2020-CVPR-Data-Free Knowledge Amalgamation via Group-Stack Dual-GAN

• 2020-CVPR-Regularizing Class-Wise Predictions via Self-Knowledge Distillation

• 2020-ICML-Feature-map-level Online Adversarial Knowledge Distillation

• 2020-NIPS-Self-Distillation as Instance-Specific Label Smoothing

• 2020-NIPS-Ensemble Distillation for Robust Model Fusion in Federated Learning

• 2020-NIPS-Self-Distillation Amplifies Regularization in Hilbert Space

• 2020-NIPS-MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers

• 2020-NIPS-Residual Distillation: Towards Portable Deep Neural Networks without Shortcuts

• 2020-NIPS-Kernel Based Progressive Distillation for Adder Neural Networks

• 2020-NIPS-Agree to Disagree: Adaptive Ensemble Knowledge Distillation in Gradient Space

• 2020-NIPS-Task-Oriented Feature Distillation

• 2020-NIPS-Comprehensive Attention Self-Distillation for Weakly-Supervised Object Detection

• 2020-NIPS-Distributed Distillation for On-Device Learning

• 2020-NIPS-Knowledge Distillation in Wide Neural Networks: Risk Bound, Data Efficiency and Imperfect Teacher

• 2020.12-Knowledge Distillation Thrives on Data Augmentation

• 2020.12-Multi-head Knowledge Distillation for Model Compression

2021

• 2021-AAAI-Cross-Layer Distillation with Semantic Calibration [Code]

• 2021-ICLR-Distilling Knowledge from Reader to Retriever for Question Answering

• 2021-ICLR-Improve Object Detection with Feature-based Knowledge Distillation: Towards Accurate and Efficient Detectors

• 2021-ICLR-Knowledge distillation via softmax regression representation learning [Code]

• 2021-ICLR-Knowledge Distillation as Semiparametric Inference

• 2021-ICLR-Is Label Smoothing Truly Incompatible with Knowledge Distillation: An Empirical Study

• 2021-ICLR-Rethinking Soft Labels for Knowledge Distillation: A Bias–Variance Tradeoff Perspective

• 2021-CVPR-Refine Myself by Teaching Myself: Feature Refinement via Self-Knowledge Distillation [PyTorch Code]

• 2021-CVPR-Complementary Relation Contrastive Distillation

• 2021-CVPR-Distilling Knowledge via Knowledge Review [Code]

• 2021-ICML-KD3A: Unsupervised Multi-Source Decentralized Domain Adaptation via Knowledge Distillation

• 2021-ICML-A statistical perspective on distillation

• 2021-ICML-Training data-efficient image transformers & distillation through attention

• 2021-ICML-Zero-Shot Knowledge Distillation from a Decision-Based Black-Box Model

• 2021-ICML-Data-Free Knowledge Distillation for Heterogeneous Federated Learning

• 2021-ICML-Simultaneous Similarity-based Self-Distillation for Deep Metric Learning

• 2021-NIPS-Slow Learning and Fast Inference: Efficient Graph Similarity Computation via Knowledge Distillation [Code]

2022

• 2022-ECCV-R2L: Distilling Neural Radiance Field to Neural Light Field for Efficient Novel View Synthesis Code

• 2022-NIPS-An Analytical Theory of Curriculum Learning in Teacher-Student Networks

Papers [AutoML (NAS etc.)]

• 2016.11-Neural architecture search with reinforcement learning

• 2019-CVPR-Searching for A Robust Neural Architecture in Four GPU Hours [Code]

• 2019-CVPR-FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search

• 2019-CVPR-RENAS: Reinforced Evolutionary Neural Architecture Search

• 2019-NIPS-Meta Architecture Search

• 2019-NIPS-SpArSe: Sparse Architecture Search for CNNs on Resource-Constrained Microcontrollers

• 2020-NIPS-Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation

• 2020-NIPS-Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search

• 2020-NIPS-Theory-Inspired Path-Regularized Differential Network Architecture Search

• 2020-NIPS-ISTA-NAS: Efficient and Consistent Neural Architecture Search by Sparse Coding

• 2020-NIPS-Semi-Supervised Neural Architecture Search

• 2020-NIPS-Bridging the Gap between Sample-based and One-shot Neural Architecture Search with BONAS

• 2020-NIPS-Does Unsupervised Architecture Representation Learning Help Neural Architecture Search?

• 2020-NIPS-Differentiable Neural Architecture Search in Equivalent Space with Exploration Enhancement

• 2020-NIPS-CLEARER: Multi-Scale Neural Architecture Search for Image Restoration

• 2020-NIPS-A Study on Encodings for Neural Architecture Search

• 2020-NIPS-Auto-Panoptic: Cooperative Multi-Component Architecture Search for Panoptic Segmentation

• 2020-NIPS-Hierarchical Neural Architecture Search for Deep Stereo Matching

Papers [Interpretability]

• 2010-JMLR-How to explain individual classification decisions

• 2015-PLOS ONE-On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation

• 2015-CVPR-Learning to generate chairs with convolutional neural networks

• 2015-CVPR-Understanding deep image representations by inverting them [2016 IJCV version: Visualizing deep convolutional neural networks using natural pre-images]

• 2016-CVPR-Inverting Visual Representations with Convolutional Networks

• 2016-KDD-“Why Should I Trust You?”: Explaining the Predictions of Any Classifier

• 2016-ICMLw-The Mythos of Model Interpretability

• 2017-NIPSw-The (Un)reliability of saliency methods

• 2017-DSP-Methods for interpreting and understanding deep neural networks

• 2018-ICML-Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors

• 2018-CVPR-Deep Image Prior [Code]

• 2018-NIPSs-Sanity Checks for Saliency Maps

• 2018-NIPSs-Human-in-the-Loop Interpretability Prior

• 2018-NIPS-To Trust Or Not To Trust A Classifier [Code]

• 2019-AISTATS-Interpreting Black Box Predictions using Fisher Kernels

• 2019.05-Luck Matters: Understanding Training Dynamics of Deep ReLU Networks

• 2019.05-Adversarial Examples Are Not Bugs, They Are Features

• 2019.06-The Generalization-Stability Tradeoff in Neural Network Pruning

• 2019.06-One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers

• 2019-Book-Interpretable Machine Learning

Workshops

• 2017-ICML Tutorial: interpretable machine learning

• 2018-ICML Workshop: Efficient Credit Assignment in Deep Learning and Reinforcement Learning

• CDNNRIA Workshop (Compact Deep Neural Network Representation with Industrial Applications): 1st-2018-NIPSw, 2nd-2019-ICMLw

• LLD Workshop (Learning with Limited Data): 1st-2017-NIPSw, 2nd-2019-ICLRw

• WHI (Worshop on Human Interpretability in Machine Learning): 1st-2016-ICMLw, 2nd-2017-ICMLw, 3rd-2018-ICMLw

• NIPS-18 Workshop on Systems for ML and Open Source Software

• MLPCD Workshop (Machine Learning on the Phone and other Consumer Devices): 2nd-2018-NIPSw

• Workshop on Bayesian Deep Learning

• 2020 CVPR Workshop on NAS

Books & Courses

• TinyML and Efficient Deep Learning @MIT by Prof. Song Han

Lightweight DNN Engines/APIs

• NNPACK

• DMLC: Tensor Virtual Machine (TVM): Open Deep Learning Compiler Stack

• Tencent: NCNN

• Xiaomi: MACE, Mobile AI Benchmark

• Alibaba: MNN blog (in Chinese)

• Baidu: Paddle-Slim, Paddle-Mobile, Anakin

• Microsoft: ELL, AutoML tool NNI

• Facebook: Caffe2/PyTorch

• Apple: CoreML (iOS 11+)

• Google: ML-Kit, NNAPI (Android 8.1+), TF-Lite

• Qualcomm: Snapdragon Neural Processing Engine (SNPE), Adreno GPU SDK

• Huawei: HiAI

• ARM: Tengine

• Related: DAWNBench: An End-to-End Deep Learning Benchmark and Competition

Related Repos and Websites

• Awesome-NAS

• Awesome-Pruning

• Awesome-Knowledge-Distillation

• MS AI-System open course

• caffe-int8-convert-tools

• Neural-Networks-on-Silicon

• Embedded-Neural-Network

• model_compression

• model-compression (in Chinese)

• Efficient-Segmentation-Networks

• AutoML NAS Literature

• Papers with code

• ImageNet Benckmark

• Self-supervised ImageNet Benckmark

• NVIDIA Blog with Sparsity Tag

源地址：https://github.com/MingSun-Tse/Efficient-Deep-Learning