Statistical estimation of conditional Shannon entropy

Steklov Mathematical Institute of Russian Academy of Sciences, Moscow, Russia.

^* Corresponding author: bulinski@yandex.ru

Received: 28 June 2018
Accepted: 28 November 2018

Abstract

The new estimates of the conditional Shannon entropy are introduced in the framework of the model describing a discrete response variable depending on a vector of d factors having a density w.r.t. the Lebesgue measure in ℝ^d. Namely, the mixed-pair model (X, Y ) is considered where X and Y take values in ℝ^d and an arbitrary finite set, respectively. Such models include, for instance, the famous logistic regression. In contrast to the well-known Kozachenko–Leonenko estimates of unconditional entropy the proposed estimates are constructed by means of the certain spacial order statistics (or k-nearest neighbor statistics where k = k_n depends on amount of observations n) and a random number of i.i.d. observations contained in the balls of specified random radii. The asymptotic unbiasedness and L²-consistency of the new estimates are established under simple conditions. The obtained results can be applied to the feature selection problem which is important, e.g., for medical and biological investigations.