Break apart cross entropy

docs/classifiers/logistic-regression.md

@@ -15,15 +15,15 @@ A linear classifier that uses the logistic (*sigmoid*) function to estimate the
 | 3 | l2Penalty | 1e-4 | float | The amount of L2 regularization applied to the weights of the output layer. |
 | 4 | epochs | 1000 | int | The maximum number of training epochs. i.e. the number of times to iterate over the entire training set before terminating. |
 | 5 | minChange | 1e-4 | float | The minimum change in the training loss necessary to continue training. |
-| 6 | costFn | CrossEntropy | ClassificationLoss | The function that computes the loss associated with an erroneous activation during training. |
+| 6 | costFn | BinaryCrossEntropy | ClassificationLoss | The function that computes the loss associated with an erroneous activation during training. |
 
 ## Example
 ```php
 use Rubix\ML\Classifiers\LogisticRegression;
 use Rubix\ML\NeuralNet\Optimizers\Adam;
-use Rubix\ML\NeuralNet\CostFunctions\CrossEntropy;
+use Rubix\ML\NeuralNet\CostFunctions\BinaryCrossEntropy;
 
-$estimator = new LogisticRegression(64, new Adam(0.001), 1e-4, 100, 1e-4, new CrossEntropy());
+$estimator = new LogisticRegression(64, new Adam(0.001), 1e-4, 100, 1e-4, new BinaryCrossEntropy());
 ```
 
 ## Additional Methods

docs/classifiers/multilayer-perceptron.md

@@ -21,7 +21,7 @@ A multiclass feed-forward neural network classifier with user-defined hidden lay
 | 6 | evalInterval | 3 | int | The number of epochs to train before evaluating the model using the holdout set. |
 | 7 | window | 5 | int | The number of epochs without improvement in the validation score to wait before considering an early stop. |
 | 8 | holdOut | 0.1 | float | The proportion of training samples to use for internal validation. Set to 0 to disable. |
-| 9 | costFn | CrossEntropy | ClassificationLoss | The function that computes the loss associated with an erroneous activation during training. |
+| 9 | costFn | MulticlassCrossEntropy | ClassificationLoss | The function that computes the loss associated with an erroneous activation during training. |
 | 10 | metric | FBeta | Metric | The validation metric used to score the generalization performance of the model during training. |
 
 ## Example
@@ -33,7 +33,7 @@ use Rubix\ML\NeuralNet\Layers\Activation;
 use Rubix\ML\NeuralNet\Layers\PReLU;
 use Rubix\ML\NeuralNet\ActivationFunctions\LeakyReLU;
 use Rubix\ML\NeuralNet\Optimizers\Adam;
-use Rubix\ML\NeuralNet\CostFunctions\CrossEntropy;
+use Rubix\ML\NeuralNet\CostFunctions\MulticlassCrossEntropy;
 use Rubix\ML\CrossValidation\Metrics\MCC;
 
 $estimator = new MultilayerPerceptron([
@@ -45,7 +45,7 @@ $estimator = new MultilayerPerceptron([
     new Dropout(0.3),
     new Dense(50),
     new PReLU(),
-], 128, new Adam(0.001), 1000, 1e-3, 10, 3, 0.1, new CrossEntropy(), new MCC());
+], 128, new Adam(0.001), 1000, 1e-3, 10, 3, 0.1, new MulticlassCrossEntropy(), new MCC());
 ```
 
 ## Additional Methods

docs/classifiers/softmax-classifier.md

@@ -15,15 +15,15 @@ A multiclass generalization of [Logistic Regression](logistic-regression.md) usi
 | 3 | alpha | 1e-4 | float | The amount of L2 regularization applied to the weights of the output layer. |
 | 4 | epochs | 1000 | int | The maximum number of training epochs. i.e. the number of times to iterate over the entire training set before terminating. |
 | 5 | minChange | 1e-4 | float | The minimum change in the training loss necessary to continue training. |
-| 6 | costFn | CrossEntropy | ClassificationLoss | The function that computes the loss associated with an erroneous activation during training. |
+| 6 | costFn | BinaryCrossEntropy | ClassificationLoss | The function that computes the loss associated with an erroneous activation during training. |
 
 ## Example
 ```php
 use Rubix\ML\Classifiers\SoftmaxClassifier;
 use Rubix\ML\NeuralNet\Optimizers\Momentum;
-use Rubix\ML\NeuralNet\CostFunctions\CrossEntropy;
+use Rubix\ML\NeuralNet\CostFunctions\MulticlassCrossEntropy;
 
-$estimator = new SoftmaxClassifier(256, new Momentum(0.001), 1e-4, 300, 1e-4, new CrossEntropy());
+$estimator = new SoftmaxClassifier(256, new Momentum(0.001), 1e-4, 300, 1e-4, new BinaryCrossEntropy());
 ```
 
 ## Additional Methods

docs/neural-network/cost-functions/binary-cross-entropy.md

@@ -0,0 +1,18 @@
+<span style="float:right;"><a href="https://github.com/RubixML/ML/blob/master/src/NeuralNet/CostFunctions/BinaryCrossEntropy.php">[source]</a></span>
+
+# Binary Cross Entropy
+Binary Cross Entropy (or *log loss*) measures the performance of a binary classification model whose output is a probability value between 0 and 1. Cross-entropy loss increases as the predicted probability diverges from the actual label. So predicting a probability of .012 when the actual observation label is 1 would be bad and result in a high loss value. A perfect score would have a log loss of 0.
+
+$$
+Binary\ Cross\ Entropy = -\frac{1}{N}\sum_{i=1}^N[y_i\log(p_i) + (1-y_i)\log(1-p_i)]
+$$
+
+## Parameters
+This cost function does not have any parameters.
+
+## Example
+```php
+use Rubix\ML\NeuralNet\CostFunctions\BinaryCrossEntropy;
+
+$costFunction = new BinaryCrossEntropy();
+```

docs/neural-network/cost-functions/multiclass-cross-entropy.md

@@ -0,0 +1,18 @@
+<span style="float:right;"><a href="https://github.com/RubixML/ML/blob/master/src/NeuralNet/CostFunctions/MulticlassCrossEntropy.php">[source]</a></span>
+
+# Multiclass Cross Entropy
+Multiclass Cross Entropy measures the performance of a multiclass classification model whose output is a probability distribution over the possible classes. Cross-entropy loss increases as the predicted probability distribution diverges from the actual distribution.
+
+$$
+Multiclass\ Cross\ Entropy = -\frac{1}{N}\sum_{i=1}^N\sum_{c=1}^C y_{i,c}\log(p_{i,c})
+$$
+
+## Parameters
+This cost function does not have any parameters.
+
+## Example
+```php
+use Rubix\ML\NeuralNet\CostFunctions\MulticlassCrossEntropy;
+
+$costFunction = new MulticlassCrossEntropy();
+```

mkdocs.yml

@@ -176,7 +176,8 @@ nav:
         - SiLU: neural-network/activation-functions/silu.md
         - Thresholded ReLU: neural-network/activation-functions/thresholded-relu.md
       - Cost Functions:
-        - Cross Entropy: neural-network/cost-functions/cross-entropy.md
+        - Binary Cross Entropy: neural-network/cost-functions/binary-cross-entropy.md
+        - Multiclass Cross Entropy: neural-network/cost-functions/multiclass-cross-entropy.md
         - Huber Loss: neural-network/cost-functions/huber-loss.md
         - Least Squares: neural-network/cost-functions/least-squares.md
         - Relative Entropy: neural-network/cost-functions/relative-entropy.md

src/Classifiers/LogisticRegression.php

@@ -28,7 +28,7 @@
 use Rubix\ML\Specifications\DatasetIsLabeled;
 use Rubix\ML\Specifications\DatasetIsNotEmpty;
 use Rubix\ML\Specifications\SpecificationChain;
-use Rubix\ML\NeuralNet\CostFunctions\CrossEntropy;
+use Rubix\ML\NeuralNet\CostFunctions\BinaryCrossEntropy;
 use Rubix\ML\Specifications\DatasetHasDimensionality;
 use Rubix\ML\NeuralNet\CostFunctions\ClassificationLoss;
 use Rubix\ML\Specifications\LabelsAreCompatibleWithLearner;
@@ -163,7 +163,7 @@ public function __construct(
         $this->l2Penalty = $l2Penalty;
         $this->epochs = $epochs;
         $this->minChange = $minChange;
-        $this->costFn = $costFn ?? new CrossEntropy();
+        $this->costFn = $costFn ?? new BinaryCrossEntropy();
     }
 
     /**

src/Classifiers/MultilayerPerceptron.php

@@ -30,9 +30,10 @@
 use Rubix\ML\Specifications\DatasetIsLabeled;
 use Rubix\ML\Specifications\DatasetIsNotEmpty;
 use Rubix\ML\Specifications\SpecificationChain;
-use Rubix\ML\NeuralNet\CostFunctions\CrossEntropy;
+use Rubix\ML\NeuralNet\CostFunctions\MulticlassCrossEntropy;
 use Rubix\ML\Specifications\DatasetHasDimensionality;
 use Rubix\ML\NeuralNet\CostFunctions\ClassificationLoss;
+use Rubix\ML\NeuralNet\CostFunctions\BinaryCrossEntropy;
 use Rubix\ML\Specifications\LabelsAreCompatibleWithLearner;
 use Rubix\ML\Specifications\EstimatorIsCompatibleWithMetric;
 use Rubix\ML\Specifications\SamplesAreCompatibleWithEstimator;
@@ -243,6 +244,10 @@ public function __construct(
                 . " between 0 and 0.5, $holdOut given.");
         }
 
+        if ($costFn and $costFn instanceof BinaryCrossEntropy) {
+            throw new InvalidArgumentException('Not compatible with binary cross entropy.');
+        }
+
         if ($metric) {
             EstimatorIsCompatibleWithMetric::with($this, $metric)->check();
         }
@@ -255,7 +260,7 @@ public function __construct(
         $this->evalInterval = $evalInterval;
         $this->window = $window;
         $this->holdOut = $holdOut;
-        $this->costFn = $costFn ?? new CrossEntropy();
+        $this->costFn = $costFn ?? new MulticlassCrossEntropy();
         $this->metric = $metric ?? new FBeta();
     }
 

src/Classifiers/SoftmaxClassifier.php

@@ -25,7 +25,8 @@
 use Rubix\ML\Specifications\DatasetIsLabeled;
 use Rubix\ML\Specifications\DatasetIsNotEmpty;
 use Rubix\ML\Specifications\SpecificationChain;
-use Rubix\ML\NeuralNet\CostFunctions\CrossEntropy;
+use Rubix\ML\NeuralNet\CostFunctions\MulticlassCrossEntropy;
+use Rubix\ML\NeuralNet\CostFunctions\BinaryCrossEntropy;
 use Rubix\ML\Specifications\DatasetHasDimensionality;
 use Rubix\ML\NeuralNet\CostFunctions\ClassificationLoss;
 use Rubix\ML\Specifications\LabelsAreCompatibleWithLearner;
@@ -154,12 +155,16 @@ public function __construct(
                 . " greater than 0, $minChange given.");
         }
 
+        if ($costFn and $costFn instanceof BinaryCrossEntropy) {
+            throw new InvalidArgumentException('Not compatible with binary cross entropy.');
+        }
+
         $this->batchSize = $batchSize;
         $this->optimizer = $optimizer ?? new Adam();
         $this->l2Penalty = $l2Penalty;
         $this->epochs = $epochs;
         $this->minChange = $minChange;
-        $this->costFn = $costFn ?? new CrossEntropy();
+        $this->costFn = $costFn ?? new MulticlassCrossEntropy();
     }
 
     /**

src/NeuralNet/CostFunctions/CrossEntropy.php → src/NeuralNet/CostFunctions/BinaryCrossEntropy.php

@@ -13,21 +13,21 @@
 use const Rubix\ML\EPSILON;
 
 /**
- * Cross Entropy
+ * Binary Cross Entropy
  *
- * Cross Entropy, or log loss, measures the performance of a classification model
- * whose output is a probability value between 0 and 1. Cross-entropy loss
- * increases as the predicted probability diverges from the actual label. So
- * predicting a probability of .012 when the actual observation label is 1 would
- * be bad and result in a high loss value. A perfect score would have a log loss
- * of 0.
+ * Binary Cross Entropy, or log loss, measures the performance of a binary
+ * classification model whose output is a probability value between 0 and 1.
+ * Cross-entropy loss increases as the predicted probability diverges from the
+ * actual label. So predicting a probability of .012 when the actual observation
+ * label is 1 would be bad and result in a high loss value. A perfect score
+ * would have a log loss of 0.
  *
  * @category    Machine Learning
  * @package     Rubix/ML
  * @author      Andrew DalPino
  * @author      Samuel Akopyan <leumas.a@gmail.com>
  */
-class CrossEntropy implements ClassificationLoss
+class BinaryCrossEntropy implements ClassificationLoss
 {
     use AssertsShapes;
 
@@ -42,7 +42,7 @@ public function __construct()
     /**
      * Compute the loss score.
      *
-     * L(y, ŷ) = -Σ(y * log(ŷ)) / n
+     * L(y, ŷ) = -Σ(y * log(ŷ) + (1 - y) * log(1 - ŷ)) / n
      *
      * @param NDArray $output The output of the network
      * @param NDArray $target The target values
@@ -52,12 +52,17 @@ public function compute(NDArray $output, NDArray $target) : float
     {
         $this->assertSameShape($output, $target);
 
-        // Clip values to avoid log(0)
-        $output = NumPower::clip($output, EPSILON, 1.0);
+        $output = NumPower::clip($output, EPSILON, 1.0 - EPSILON);
+        $target = NumPower::clip($target, EPSILON, 1.0 - EPSILON);
 
         $logOutput = NumPower::log($output);
+        $logOneMinusOutput = NumPower::log(NumPower::subtract(1.0, $output));
+        $oneMinusTarget = NumPower::subtract(1.0, $target);
+
         $product = NumPower::multiply($target, $logOutput);
-        $negated = NumPower::multiply($product, -1.0);
+        $product2 = NumPower::multiply($oneMinusTarget, $logOneMinusOutput);
+        $sum = NumPower::add($product, $product2);
+        $negated = NumPower::multiply($sum, -1.0);
 
         return NumPower::mean($negated);
     }
@@ -75,13 +80,10 @@ public function differentiate(NDArray $output, NDArray $target) : NDArray
     {
         $this->assertSameShape($output, $target);
 
-        // Numerator = ŷ - y (calculate before clipping to preserve zeros)
         $numerator = NumPower::subtract($output, $target);
 
-        // Clip values to avoid division by zero
         $output = NumPower::clip($output, EPSILON, 1.0 - EPSILON);
 
-        // Denominator = ŷ * (1 - ŷ)
         $oneMinusOutput = NumPower::subtract(1.0, $output);
         $denominator = NumPower::multiply($output, $oneMinusOutput);
         $denominator = NumPower::clip($denominator, EPSILON, 1.0);
@@ -96,6 +98,6 @@ public function differentiate(NDArray $output, NDArray $target) : NDArray
      */
     public function __toString() : string
     {
-        return 'Cross Entropy';
+        return 'Binary Cross Entropy';
     }
 }

src/NeuralNet/CostFunctions/MulticlassCrossEntropy.php

@@ -0,0 +1,91 @@
+<?php
+
+declare(strict_types=1);
+
+namespace Rubix\ML\NeuralNet\CostFunctions;
+
+use NDArray;
+use NumPower;
+use Rubix\ML\Specifications\ExtensionIsLoaded;
+use Rubix\ML\Specifications\ExtensionMinimumVersion;
+use Rubix\ML\Specifications\SpecificationChain;
+use Rubix\ML\Traits\AssertsShapes;
+use const Rubix\ML\EPSILON;
+
+/**
+ * Multiclass Cross Entropy
+ *
+ * Multiclass Cross Entropy measures the performance of a multiclass
+ * classification model whose output is a probability distribution over the
+ * possible classes. Cross-entropy loss increases as the predicted probability
+ * distribution diverges from the actual distribution.
+ *
+ * @category    Machine Learning
+ * @package     Rubix/ML
+ * @author      Andrew DalPino
+ * @author      Samuel Akopyan <leumas.a@gmail.com>
+ */
+class MulticlassCrossEntropy implements ClassificationLoss
+{
+    use AssertsShapes;
+
+    public function __construct()
+    {
+        SpecificationChain::with([
+            new ExtensionIsLoaded('RubixNumPower'),
+            new ExtensionMinimumVersion('RubixNumPower', '0.7.0'),
+        ])->check();
+    }
+
+    /**
+     * Compute the loss score.
+     *
+     * L(y, ŷ) = -Σ(y * log(ŷ)) / n
+     *
+     * @param NDArray $output The output of the network
+     * @param NDArray $target The target values
+     * @return float
+     */
+    public function compute(NDArray $output, NDArray $target) : float
+    {
+        $this->assertSameShape($output, $target);
+
+        $output = NumPower::clip($output, EPSILON, 1.0);
+
+        $logOutput = NumPower::log($output);
+        $product = NumPower::multiply($target, $logOutput);
+        $negated = NumPower::multiply($product, -1.0);
+
+        return NumPower::mean($negated);
+    }
+
+    /**
+     * Calculate the gradient of the cost function with respect to the output.
+     *
+     * ∂L/∂ŷ = -y / ŷ
+     *
+     * @param NDArray $output The output of the network
+     * @param NDArray $target The target values
+     * @return NDArray
+     */
+    public function differentiate(NDArray $output, NDArray $target) : NDArray
+    {
+        $this->assertSameShape($output, $target);
+
+        $output = NumPower::clip($output, EPSILON, 1.0);
+
+        $negated = NumPower::multiply($target, -1.0);
+
+        return NumPower::divide($negated, $output);
+    }
+
+    /**
+     * Return the string representation of the object.
+     *
+     * @return string
+     */
+    public function __toString() : string
+    {
+        return 'Multiclass Cross Entropy';
+    }
+}