Ветров Дмитрий Петрович

On the Periodic Behavior of Neural Network Training with Batch Normalization and Weight Decay

2021 · CHAPTER · en

Leveraging Recursive Gumbel-Max Trick for Approximate Inference in Combinatorial Spaces

2021 · CHAPTER · en

Structured latent variables allow incorporating meaningful prior knowledge into deep learning models. However, learning with such variables remains challenging because of their discrete nature. Nowadays, the standard learning approach is to define a latent variable as a perturbed algorithm output and to use a differentiable surrogate for training. In general, the surrogate puts additional constraints on the model and inevitably leads to biased gradients. To alleviate these shortcomings, we extend the Gumbel-Max trick to define distributions over structured domains. We avoid the differentiable surrogates by leveraging the score function estimators for optimization. In particular, we highlight a family of recursive algorithms with a common feature we call stochastic invariant. The feature allows us to construct reliable gradient estimates and control variates without additional constraints on the model. In our experiments, we consider various structured latent variable models and achieve results competitive with relaxation-based counterparts.

Low-Variance Black-Box Gradient Estimates for the Plackett-Luce Distribution

2020 · CHAPTER · en

Greedy Policy Search: A Simple Baseline for Learnable Test-Time Augmentation

2020 · CHAPTER · en

Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics

2020 · CHAPTER · en

Structured Sparsification of Gated Recurrent Neural Networks

2020 · CHAPTER · en

On Power Laws in Deep Ensembles

2020 · CHAPTER · en

User-controllable Multi-texture Synthesis with Generative Adversarial Networks

2020 · CHAPTER · en

We propose a novel multi-texture synthesis model based on generative adversarial networks (GANs) with a user-controllable mechanism. The user control ability allows to explicitly specify the texture which should be generated by the model. This property follows from using an encoder part which learns a latent representation for each texture from the dataset. To ensure a dataset coverage, we use an adversarial loss function that penalizes for incorrect reproductions of a given texture. In experiments, we show that our model can learn descriptive texture manifolds for large datasets and from raw data such as a collection of high-resolution photos. We show our unsupervised learning pipeline may help segmentation models. Moreover, we apply our method to produce 3D textures and show that it outperforms existing baselines.

Involutive MCMC: a Unifying Framework

2020 · CHAPTER · en

Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learning

2020 · CHAPTER · en

Uncertainty estimation and ensembling methods go hand-in-hand. Uncertainty estimation is one of the main benchmarks for assessment of ensembling performance. At the same time, deep learning ensembles have provided state-of-the-art results in uncertainty estimation. In this work, we focus on in-domain uncertainty for image classification. We explore the standards for its quantification and point out pitfalls of existing metrics. Avoiding these pitfalls, we perform a broad study of different ensembling techniques. To provide more insight in this study, we introduce the deep ensemble equivalent score (DEE) and show that many sophisticated ensembling techniques are equivalent to an ensemble of only few independently trained networks in terms of test performance.