Беломестный Денис Витальевич

From Dirichlet to Rubin: Optimistic Exploration in RL without Bonuses

2022 · CHAPTER · en

Statistical inference for scale mixture models via Mellin transform approach

2022 · PREPRINT · en

This paper deals with statistical inference for the scale mixture models. We study an estimation approach based on the Mellin – Stielt- jes transform that can be applied to both discrete and absolute contin- uous mixing distributions. The accuracy of the corresponding estimate is analysed in terms of its expected pointwise error. As an important technical result, we prove the analogue of the Berry – Esseen inequal- ity for the Mellin transforms. The proposed statistical approach is illustrated by numerical examples.

Nonparametric Bayesian volatility estimation for gamma-driven stochastic differential equations

2022 · ARTICLE · en

We study a nonparametric Bayesian approach to estimation of the volatility function of a stochastic differential equation driven by a gamma process. The volatility function is modelled a priori as piecewise constant, and we specify a gamma prior on its values. This leads to a straightforward procedure for posterior inference via an MCMC procedure. We give theoretical performance guarantees (minimax optimal contraction rates for the posterior) for the Bayesian estimate in terms of the regularity of the unknown volatility function. We illustrate the method on synthetic and real data examples.

Optimistic Posterior Sampling for Reinforcement Learning with Few Samples and Tight Guarantees

2022 · CHAPTER · en

Variance Reduction for Policy-Gradient Methods via Empirical Variance Minimization

2022 · PREPRINT · en

Policy-gradient methods in Reinforcement Learning(RL) are very universal and widely applied in practice but their performance suffers from the high variance of the gradient estimate. Several procedures were proposed to reduce it including actor-critic(AC) and advantage actor-critic(A2C) methods. Recently the approaches have got new perspective due to the introduction of Deep RL: both new control variates(CV) and new sub-sampling procedures became available in the setting of complex models like neural networks. The vital part of CV-based methods is the goal functional for the training of the CV, the most popular one is the least-squares criterion of A2C. Despite its practical success, the criterion is not the only one possible. In this paper we for the first time investigate the performance of the one called Empirical Variance(EV). We observe in the experiments that not only EV-criterion performs not worse than A2C but sometimes can be considerably better. Apart from that, we also prove some theoretical guarantees of the actual variance reduction under very general assumptions and show that A2C least-squares goal functional is an upper bound for EV goal. Our experiments indicate that in terms of variance reduction EV-based methods are much better than A2C and allow stronger variance reduction.

О разработке прикладных решений на основе искусственного интеллекта для обеспечения технологической безопасности

2022 · ARTICLE · ru

Основной миссией Исследовательского центра в сфере искусственного интеллекта НИУ ВШЭ (Центра ИИ) являются развитие и внедрение технологий искусственного интеллекта в различные сферы жизни человека и общества, отрасли науки и секторы экономики. В рамках деятельности Центра ИИ разрабатываются новые технологии искусственного интеллекта, позволяющие расширить область применения искусственного интеллекта; создаются программные инструменты и средства для применения искусственного интеллекта в отраслях науки и бизнеса, разрабатывается открытая программная библиотека методов искусственного интеллекта для решения задач, имеющих высокую социальную значимость.

Density deconvolution under general assumptions on the distribution of measurement errors

2021 · ARTICLE · en

In this paper we study the problem of density deconvolution under general assumptions on the measurement error distribution. Typically deconvolution estimators are constructed using Fourier transform techniques, and it is assumed that the characteristic function of the measurement errors does not have zeros on the real line. This assumption is rather strong and is not fulfilled in many cases of interest. In this paper we develop a methodology for constructing optimal density deconvolution estimators in the general setting that covers vanishing and non--vanishing characteristic functions of the measurement errors. We derive upper bounds on the risk of the proposed estimators and provide sufficient conditions under which zeros of the corresponding characteristic function have no effect on estimation accuracy. Moreover, we show that the derived conditions are also necessary in some specific problem instances.

Variance reduction for dependent sequences with applications to Stochastic Gradient MCMC

2021 · ARTICLE · en

In this paper we propose a novel and practical variance reduction approach for additive functionals of dependent sequences. Our approach combines the use of control variates with the minimisation of an empirical variance estimate. We analyse finite sample properties of the proposed method and derive finite-time bounds of the excess asymptotic variance to zero. We apply our methodology to Stochastic Gradient MCMC (SGMCMC) methods for Bayesian inference on large data sets and combine it with existing variance reduction methods for SGMCMC. We present empirical results carried out on a number of benchmark examples showing that our variance reduction method achieves significant improvement as compared to state-of-the-art methods at the expense of a moderate increase of computational overhead.

Fourier transform MCMC, heavy-tailed distributions, and geometric ergodicity

2021 · ARTICLE · en

Markov Chain Monte Carlo methods become increasingly popular in applied mathematics as a tool for numerical integration with respect to complex and high-dimensional distributions. However, application of MCMC methods to heavy-tailed distributions and distributions with analytically intractable densities turns out to be rather problematic. In this paper, we propose a novel approach towards the use of MCMC algorithms for distributions with analytically known Fourier transforms and, in particular, heavy-tailed distributions. The main idea of the proposed approach is to use MCMC methods in Fourier domain to sample from a density proportional to the absolute value of the underlying characteristic function. A subsequent application of the Parseval’s formula leads to an efficient algorithm for the computation of integrals with respect to the underlying density. We show that the resulting Markov chain in Fourier domain may be geometrically ergodic even in the case of heavy-tailed original distributions. We illustrate our approach by several numerical examples including multivariate elliptically contoured stable distributions.

UVIP: Model-Free Approach to Evaluate Reinforcement Learning Algorithms

2021 · ARTICLE · en

Policy evaluation is an important instrument for the comparison of different algorithms in Reinforcement Learning (RL). Yet even a precise knowledge of the value function $V^{\pi}$ corresponding to a policy $\pi$ does not provide reliable information on how far is the policy $\pi$ from the optimal one. We present a novel model-free upper value iteration procedure ({\sf UVIP}) that allows us to estimate the suboptimality gap $V^{\star}(x) - V^{\pi}(x)$ from above and to construct confidence intervals for \(V^\star\). Our approach relies on upper bounds to the solution of the Bellman optimality equation via martingale approach. We provide theoretical guarantees for {\sf UVIP} under general assumptions and illustrate its performance on a number of benchmark RL problems.