It happens that KL(p,q) = cross-entropy(p,q) - entropy(p). If we assume the distribution of data is p, and the classifier says that the distribution is q. Ideally, we would want the KL divergence = 0. When KL divergence = 0, the two divergences are identical and the classifiers outputs are identical to the distribution of the data, which means the machine learning algorithm is making the best guesses that it can make. In information theory, the Kullback-Leibler (KL) divergence measures how “different” two probability distributions are. While the cross-entropy loss can be interpreted via information theory. Hence they are the first choice for Support Vector Machine.

The hinge loss leads to better accuracy and sparsity at the cost of less sensitivity regarding probabilities. Margins in Hinge Loss is the smallest distance between the line (or hyperplane) and data that separates our points into classes and defines our classification. It instead, punishes the misclassification, leading it to be good at estimating margins. However, Hinge loss does not help in probabilistic estimation. There are various loss types starting from cross-entropy to mean squared error, huber loss and the hinge loss(L-2 regularization loss)Ĭross-entropy loss (logarithmic loss) leads to well behaved probabilistic outputs that is allowing us to find the maximum likelihood estimate of our model’s parameters. For any machine learning model, the weights are learned from minimizing the loss function. They give us an objective to measure the performance of the model. For any machine learning model, loss functions are a key part of it.

হায়রোগ্লিফের দেশে by Anirban Ghosh