Project details for Information Theoretical Estimators

Screenshot Information Theoretical Estimators 0.20

by szzoli - November 21, 2012, 13:55:09 CET [ Project Homepage BibTeX BibTeX for corresponding Paper Download ]

view (93 today), download ( 9 today ), 2 subscriptions

Description:

ITE can estimate Shannon-, Rényi-, Tsallis entropy; generalized variance, kernel canonical correlation analysis, kernel generalized variance, Hilbert-Schmidt independence criterion, Shannon-, L2-, Rényi-, Tsallis mutual information, copula-based kernel dependency, multivariate version of Hoeffding's Phi; complex variants of entropy and mutual information; L2-, Rényi-, Tsallis-, Kullback-Leibler divergence; Hellinger-, Bhattacharyya distance, maximum mean discrepancy, and J-distance.

ITE offers solution methods for

  • Independent Subspace Analysis (ISA) and
  • its extensions to different linear-, controlled-, post nonlinear-, complex valued-, partially observed models, as well as to systems with nonparametric source dynamics.

ITE is

  • written in Matlab/Octave,
  • multi-platform (tested extensively on Windows and Linux),
  • free and open source (released under the GNU GPLv3(>=) license).
Changes to previous version:
  • Two Shannon entropy estimators based on the distance (KL divergence) from the uniform/Gaussian distributions: added.

  • Shannon entropy estimator based on Voronoi regions: added.

BibTeX Entry: Download
Corresponding Paper BibTeX Entry: Download
URL: Project Homepage
Supported Operating Systems: Linux, Windows
Data Formats: Matlab, Octave
Tags: Entropy, Mutual Information, Divergence, Independent Subspace Analysis, Separation Principles, Independent Process Analysis
Archive: download here

Other available revisons

Version Changelog Date
0.60
  • Quick test on the Tsallis divergence: introduced.

  • Pearson chi square divergence estimation in the exponential family (MLE + analytical formula): added.

June 3, 2014, 00:17:33
0.59
  • Adaptive partitioning based Shannon mutual information estimators: added.

  • Quick tests: updated to handle the new estimators.

May 16, 2014, 22:13:48
0.58
  • 3-way interaction indices based on the embedding of the (i) Lancaster interaction and (ii) 'joint - product of the marginals' signed measures to a RKHS: added.

  • Quick tests: updated to cover the new estimators.

April 29, 2014, 22:08:50
0.57
  • Kullback-Leibler divergence estimation based on maximum likelihood estimation + analytical formula in the chosen exponential family: added.

  • A new sampling based entropy estimator with KDE correction on the left/right sides: added.

  • Quick tests: updated with the new estimators.

April 10, 2014, 18:35:22
0.56
  • Distribution regression (supervised entropy learning, aerosol optical depth prediction based on satellite images): added.

  • MMD distance computation based on U-statistics, expected kernel: upgraded to cover new kernels (exponential, Cauchy, Matern, polynomial, rational quadratic, inverse multiquadratic).

March 27, 2014, 22:06:54
0.55
  • Shannon entropy and cross-entropy estimation based on maximum likelihood estimation + analytical formula in the chosen exponential family: added.

  • Quick tests: updated with the new estimators.

March 8, 2014, 00:18:00
0.54
  • Renyi and Tsallis entropy estimation based on maximum likelihood estimation + analytical formula in the exponential family: added.

  • Quick tests: updated according to the new estimators.

February 24, 2014, 18:28:32
0.53
  • f-divergence estimation based on second-order Taylor expansion + Pearson chi square divergence: added.

  • Shannon mutual information estimation based on KL divergence: added.

  • Quick tests: updated with the new estimators.

February 2, 2014, 14:04:19
0.52
  • Sharma-Mittal divergence estimation: added using (i) maximum likelihood estimation + analytical formula in the exponential family, (ii) k-nearest neighbors.

  • Quick test for (i) Sharma-Mittal divergence, (ii) Shannon mutual information: added.

  • Normal variables: added to the Pearson chi square divergence quick test.

January 10, 2014, 00:02:47
0.51
  • ITE has been accepted for publication in JMLR; citing information: added.

  • Block-MMD (maximum mean discrepancy) estimator: added.

  • 'Extreme large' k in kNN based estimators: an overflow issue discovered + corrected.

  • +Some refactorization, documentation upgrade.

December 29, 2013, 17:36:00
0.50
  • Entropy and Kullback-Leibler divergence estimation based on power spectral density representation and Szegő's theorem: added.

  • Different noisy examples have been added to the image registration quick test.

December 18, 2013, 20:53:28
0.49
  • MMD (maximum mean discrepancy) estimation based on U- and V-statistics: incomplete Cholesky decomposition based accelerations added.

  • Refactorization; improved navigation in the documentation.

December 1, 2013, 16:44:15
0.48

Sharma-Mittal entropy estimation based on

  • k-nearest neighbors (S={k}): added.

  • maximum likehood estimation + analytical value in the exponential family: added.

November 11, 2013, 19:26:39
0.47
  • Chi-square mutual information estimation based on Pearson chi-square divergence: added.

  • Shannon entropy estimation based on an alternative linearly corrected spacing method: added.

November 1, 2013, 19:29:12
0.46
  • Phi-entropy (f-entropy) estimation based on the spacing method: added.

  • Pearson chi square divergence (chi square distance) estimation based on k-nearest neighbors: added.

October 21, 2013, 19:31:52
0.45
  • Exponentiated Jensen-Tsallis kernel-1 estimation based on Tsallis entropy: added.

  • Exponentiated Jensen-Tsallis kernel-2 estimation based on Jensen-Tsallis divergence: added.

October 9, 2013, 21:44:21
0.44
  • Exponentiated Jensen-Renyi kernel-1 estimation based on Renyi entropy: added.

  • Exponentiated Jensen-Renyi kernel-2 estimation based on Jensen-Renyi divergence: added.

October 1, 2013, 18:16:38
0.43
  • Exponentiated Jensen-Shannon kernel estimation: added.

  • Jensen-Tsallis kernel estimation: added.

September 20, 2013, 19:56:45
0.42
  • High-level information theoretical estimators: 'eval' changed to 'function handles' -- speeds up computations.

  • Cost object initialization: now allows setting field values (alpha, number of kNN-s, ...) through its argument. => possibility to override default values, automatic inheritence in meta estimators.

  • Quick tests introduced: consistency of the estimators (analytical vs. estimated value), positive semi-definiteness of Gram matrices determined by distribution kernels, image registration.

  • Refactorization; documentation: improved.

September 7, 2013, 16:46:42
0.41
  • Probability product kernel estimation based on k-nearest neighbors: added,

  • Jensen-Shannon kernel estimation: added.

July 12, 2013, 21:33:42
0.40
  • Bhattacharyya kernel estimation based on k-nearest neighbors: added,

  • Expected kernel estimation: added,

  • Kernel on distributions (K) object type: added.

June 23, 2013, 13:13:27
0.39
  • Symmetric Bregman distance estimation based on nonsymmetric Bregman distance: added,

  • Symmetric Bregman distance estimation using the k-nearest neighbor method: added.

June 12, 2013, 13:12:52
0.38
  • Jensen-Tsallis divergence estimation: added,

  • Bregman distance estimation: added.

June 1, 2013, 10:20:00
0.37
  • K divergence estimation: added,

  • L divergence estimation: added,

  • kNN squared distance computation: refined.

May 12, 2013, 15:35:39
0.36
  • Jensen-Rényi divergence estimation: added,

  • Jensen-Shannon divergence estimation: added.

April 26, 2013, 18:45:27
0.35

An alternative Jacobi optimization based ICA solution with general entropy/mutual information estimators: added; The method extends the RADICAL ICA scheme to general objectives.

April 2, 2013, 10:37:51
0.34

Jacobi optimization based ICA solution with general entropy/mutual information estimators: added. The method extends the SWICA scheme to general objectives.

March 22, 2013, 11:41:05
0.33

Two one-dimensional Shannon entropy estimators based on the maximum entropy method: added.

March 6, 2013, 09:44:47
0.32
  • ICA and ISA structures: introduced for unified treatment of the estimators. It will also enable embedding of general ICA optimization algorithms such as the Jacobi method.

  • 'stepwiseLS' mAR estimator: deleted.

  • 'kdpee.c': MSVC does not provide log2. A more elegant solution: added.

February 25, 2013, 12:42:40
0.31
  • EASI (equivariant adaptive separation via independence) real/complex ICA method: added.

  • Adaptive (k-d) partitioning based Shannon entropy estimation: added.

February 9, 2013, 11:31:39
0.30
  • Upper tail dependence via conditional Spearman's rho: added.

  • Multivariate conditional version of Spearman's rho weighting the lower tail: added.

January 25, 2013, 15:11:25
0.29
  • Lower tail dependence via conditional Spearman's rho: added.

  • Multivariate conditional version of Spearman's rho weighting the lower tail: added.

January 13, 2013, 11:38:17
0.28
  • Multivariate extension of Blomqvist's beta (medial correlation coefficient): added.

  • Average pairwise Spearman's rho: added.

January 2, 2013, 22:51:22
0.27
  • Approximate correntropy independence measure estimator: added.

  • Correntropy induced metric, centered correntropy induced metric estimators: added.

  • Correntropy, centered correntropy, correntropy coefficient estimators: added.

  • Handling of identically constant random variables in distance correlation computation: included.

December 28, 2012, 14:41:57
0.26
  • Distance covariance estimation via HSIC (Hilbert-Schmidt independence criterion): added.

  • Energy distance estimation via MMD (maximum mean discrepancy): added.

  • Energy distance estimation: added.

  • We computed the square of distance correlation: sqrt included.

December 22, 2012, 14:12:33
0.25

Distance covariance, distance correlation estimation: added.

December 15, 2012, 10:20:06
0.24

MMD (maximum mean discrepancy) estimation based on U- and V-statistics: added.

December 12, 2012, 12:16:05
0.23
  • Three multivariate extensions of Spearman's rho (Spearman's rank correlation coefficient): added.

  • Association (A) cost object type: added.

December 7, 2012, 18:33:26
0.22
  • Cauchy-Schwartz and Euclidean distance based divergence estimators: added.

  • Cauchy-Schwartz and Euclidean distance based quadratic mutual information estimators: added.

December 1, 2012, 13:33:31
0.21
  • Kullback-Leibler divergence estimator based on cross-entropy and entropy: added.

  • Cross-entropy estimation based on k-nearest neighbors: added.

  • Cross cost object type: added.

November 25, 2012, 20:56:32
0.20
  • Two Shannon entropy estimators based on the distance (KL divergence) from the uniform/Gaussian distributions: added.

  • Shannon entropy estimator based on Voronoi regions: added.

November 21, 2012, 13:55:09
0.19
  • Two k-nearest neighbor based Kullback-Leibler divergence estimators: added.

  • compute_CDSS.cpp: 'sqrt(T)' -> 'sqrt(double(T))', to increase compatibility with compilers.

November 21, 2012, 13:47:05
0.18

8 sample spacing based 1d Shannon/Rényi entropy estimators: added.

November 10, 2012, 12:53:37
0.17
  • Edgeworth expansion based Shannon entropy estimator: accelerated (C++ alternative).
  • 'Tsallis entropy <- Renyi entropy' meta estimator: added.
November 6, 2012, 22:18:43
0.16
  • Edgeworth expansion based Shannon entropy estimator: added.
  • Lookup table for the underlying H/I/D estimation formulas: added (documentation).
November 2, 2012, 16:03:25
0.15
  • The Hellinger and Bhattacharyya distances are now available in ITE. They can be estimated via k-nearest neighbor methods.

  • A '/'->'*' typo: corrected in 'DL2_kNN_k_estimation.m'.

October 29, 2012, 17:27:28
0.14
  • Monte-Carlo simulation to compute the additive constants in Rényi entropy estimation: added.
  • some accelerations/compatibility enhancements: performed.
October 29, 2012, 12:31:34
0.13
  • Tsallis entropy is now available in ITE. It can be estimated via k-nearest neighbors.
  • A '/'->'*' typo: corrected in 'HRenyi_kNN_k_estimation.m'.
October 27, 2012, 22:14:18
0.12
  • Schweizer-Wolff's sigma and kappa: added.
  • Hoeffding's Phi computation: scaled-up.
October 27, 2012, 21:21:27
0.11

multivariate version of Hoeffding's Phi: added.

October 20, 2012, 23:30:17
0.1

Initial Announcement on mloss.org.

October 11, 2012, 07:47:43

Comments

No one has posted any comments yet. Perhaps you'd like to be the first?

Leave a comment

You must be logged in to post comments.