Release v. 0.1.8

.github/workflows/github-actions-test-simple.yml

@@ -35,6 +35,7 @@ jobs:
           cd tests
           pytest test_feature_metrics.py
           pytest test_package.py
+          pytest test_drop_redundant_columns.py
 
       - name: Upload test results
         uses: actions/upload-artifact@v4

.gitignore

@@ -24,6 +24,9 @@ share/python-wheels/
 *.egg
 MANIFEST
 
+# Claude Code
+.claude/
+
 # IntelliJ
 .idea/
 
@@ -54,8 +57,13 @@ src/features/lexicons/certainty.txt
 examples/vector_data/*
 examples/output/*
 node_modules/
+*.csv
+# but always track test fixture CSVs so they aren't silently dropped
+!tests/data/cleaned_data/*.csv
+*.log
 
 # testing
 /output
 /vector_data
-test.py
+test.py
+test.ipynb

docs/.readthedocs.yaml

@@ -16,8 +16,12 @@ build:
     # golang: "1.19"
   jobs:
     pre_install: # Stuff in src/requirements.txt depends on en_core_web_sm, which in turn depends on spacy
-      - pip install spacy==3.7.2 
-      - bash -c "python3 -m spacy download en_core_web_sm"
+      - pip install "numpy<2.0.0" "spacy==3.7.2"
+      # Install the model wheel directly instead of `spacy download`, which
+      # resolves the model version via a runtime compatibility lookup that
+      # can fail and build a malformed (404) URL. en_core_web_sm-3.7.1 is the
+      # release compatible with spacy 3.7.2.
+      - pip install https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl
     post_install: # Install NLTK resources after the install step
       - python3 -m nltk.downloader nps_chat
       - python3 -m nltk.downloader punkt

docs/build/html/.buildinfo

@@ -1,4 +1,4 @@
 # Sphinx build info version 1
 # This file hashes the configuration used when building these files. When it is not found, a full rebuild will be done.
-config: d7678f479036f3220c73480ec4f2c467
+config: 9a01a2cd3d4384710101b4a99edd7683
 tags: 645f666f9bcd5a90fca523b33c5a78b7

docs/build/html/_sources/examples.rst.txt

@@ -91,10 +91,11 @@ Now we are ready to call the FeatureBuilder on our data. All we need to do is de
 		speaker_id_col = "speaker_nickname",
 		message_col = "message",
 		timestamp_col = "timestamp",
-		grouping_keys = ["batch_num", "round_num"],
+		grouping_keys = ["batch_num", "round_num"], # NOTE: This example demonstrates grouping. Use conversation_id_col if you have a single conversation identifier.
 		vector_directory = "./vector_data/",
 		output_file_base = "jury_output",
-		turns = True
+		turns = True, # NOTE: This defaults to False. Decide whether you want to combine successive 'utterances' by the same person as a 'turn.'
+		drop_redundant_columns = True # NOTE: This defaults to False. When True, highly correlated and sparse feature columns are dropped from the output.
 	)
 	jury_feature_builder.featurize()
 
@@ -219,6 +220,12 @@ Regenerating Vector Cache
 
 	* By default, **we assume that, if your output file is named the same, that the underlying vectors are the same**. If this isn't true, you should set **regenerate_vectors = True** in order to clear out the cache and re-generate the RoBERTa and SBERT outputs.
 
+
+Generating Vectors using GPU
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+By default, we use the CPU to generate sentence vectors and cached RoBERTa sentimets. To override this feature and use a GPU when available (which will speed up the computation of the vectors), turn ``use_gpu`` to True.
+
+
 Custom Features
 ~~~~~~~~~~~~~~~~~
 
@@ -235,18 +242,12 @@ Custom Features
 
     * You can chose to add any of these features depending on your preference.
 
-Analyzing First Percentage (%)
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-* The **analyze_first_pct** parameter allows you to "cut off" and separately analyze the first X% of a conversation, in case you wish to separately study different sections of a conversation as it progresses. For example, you may be interested in knowing how the attributes of the first 50% of a conversation differ from the attributes of the entire conversation. Then you can sepcify the following:
-
-	.. code-block:: python
+Analyzing First Percentage (%) [Deprecated]
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-		analyze_first_pct: [0.5, 1.0]
+.. warning::
 
-	* This will first analyze the first 50% of each conversation, and then analyze the full conversation.
-
-	* By default, we will simply analyze 100% of each conversation.
+	**Deprecated as of v.0.1.8.** The **analyze_first_pct** parameter (and its underlying ``get_first_pct_of_chat`` method) has been removed. To analyze only a portion of a conversation, subset your input dataframe before passing it to the FeatureBuilder.
 
 Named Entity Recognition
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -344,6 +345,42 @@ Important Notes and Caveats
 - **[NOTE 2]** Be careful when choosing the "sum" aggregation method, as it is not always appropriate to use the "sum" as an aggregation function. While it is a sensible choice for utterance-level attributes that are *countable* (for example, the total number of words, or other lexical wordcounts), it is a less sensible choice for others (for example, it does not make sense to sum sentiment scores for each utterance in a conversation). Consequently, using the "sum" feature will come with an associated warning.
 - **[NOTE 3]** In addition to aggregating from the utterance (chat) level to the conversation level, we also aggregate from the speaker (user) level to the conversation level, using the same methods specified in ``convo_methods`` to do so.
 
+.. _reducing_redundant_features:
+
+Reducing Redundant Features
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+* **New in v.0.1.8.** The FeatureBuilder generates a large number of features, and many of them are highly correlated with one another or are sparsely populated (containing mostly missing or zero values). The following parameters allow the FeatureBuilder to automatically detect groups of redundant features and retain only one representative per group, as well as drop columns that are dominated by missing (NA) or zero values.
+
+* The **drop_redundant_columns** parameter is the main "switch" controlling this behavior. It defaults to **False**, meaning that no columns are dropped; instead, the FeatureBuilder simply *logs* which columns it identified as redundant. When set to **True**, the redundant columns are actually removed from the chat-, user-, and conversation-level outputs.
+
+	* Note that only features generated by the FeatureBuilder are considered. Your original input columns (metadata, outcome variables, and other non-numeric columns) are always preserved untouched.
+
+* The remaining parameters control *how* redundancy is detected:
+
+	* **corr_thresh** (default ``0.9``): the minimum absolute Spearman correlation at which two numeric features are treated as redundant. Features whose correlations meet this threshold are clustered into groups, and one representative (the column with the most valid data and highest variance) is kept from each group.
+
+	* **min_group_size** (default ``2``): the minimum number of correlated columns required to form a redundancy group.
+
+	* **min_na_ratio** (default ``0.3``): numeric columns whose fraction of missing (NA) values exceeds this threshold are flagged (and dropped, if **drop_redundant_columns** is **True**).
+
+	* **min_zero_ratio** (default ``0.9``): numeric columns whose fraction of zero values exceeds this threshold are flagged (and dropped, if **drop_redundant_columns** is **True**).
+
+	* **treat_zero_as_na** (default ``True``): if **True**, zeros are treated as missing values when computing redundancy metrics and selecting the representative column for each group.
+
+* **Example: dropping correlated features.** To actually drop redundant columns (rather than just logging them), set **drop_redundant_columns = True**. Recall that this parameter **defaults to False**, so you must opt in. The example below keeps only one representative from each group of features correlated at an absolute Spearman correlation of 0.9 or higher:
+
+	.. code-block:: python
+
+		jury_feature_builder = FeatureBuilder(
+			input_df = juries_df,
+			grouping_keys = ["batch_num", "round_num"],
+			output_file_base = "jury_output",
+			drop_redundant_columns = True,  # Defaults to False; set to True to drop redundant columns rather than only logging them.
+			corr_thresh = 0.9               # Cluster features correlated at >= 0.9 (absolute Spearman) and keep one representative per group.
+		)
+		jury_feature_builder.featurize()
+
 Cumulative Grouping
 ~~~~~~~~~~~~~~~~~~~~
 

docs/build/html/_static/pygments.css

@@ -6,9 +6,9 @@ span.linenos.special { color: #000000; background-color: #ffffc0; padding-left:
 .highlight .hll { background-color: #ffffcc }
 .highlight { background: #f8f8f8; }
 .highlight .c { color: #3D7B7B; font-style: italic } /* Comment */
-.highlight .err { border: 1px solid #FF0000 } /* Error */
+.highlight .err { border: 1px solid #F00 } /* Error */
 .highlight .k { color: #008000; font-weight: bold } /* Keyword */
-.highlight .o { color: #666666 } /* Operator */
+.highlight .o { color: #666 } /* Operator */
 .highlight .ch { color: #3D7B7B; font-style: italic } /* Comment.Hashbang */
 .highlight .cm { color: #3D7B7B; font-style: italic } /* Comment.Multiline */
 .highlight .cp { color: #9C6500 } /* Comment.Preproc */
@@ -25,34 +25,34 @@ span.linenos.special { color: #000000; background-color: #ffffc0; padding-left:
 .highlight .gp { color: #000080; font-weight: bold } /* Generic.Prompt */
 .highlight .gs { font-weight: bold } /* Generic.Strong */
 .highlight .gu { color: #800080; font-weight: bold } /* Generic.Subheading */
-.highlight .gt { color: #0044DD } /* Generic.Traceback */
+.highlight .gt { color: #04D } /* Generic.Traceback */
 .highlight .kc { color: #008000; font-weight: bold } /* Keyword.Constant */
 .highlight .kd { color: #008000; font-weight: bold } /* Keyword.Declaration */
 .highlight .kn { color: #008000; font-weight: bold } /* Keyword.Namespace */
 .highlight .kp { color: #008000 } /* Keyword.Pseudo */
 .highlight .kr { color: #008000; font-weight: bold } /* Keyword.Reserved */
 .highlight .kt { color: #B00040 } /* Keyword.Type */
-.highlight .m { color: #666666 } /* Literal.Number */
+.highlight .m { color: #666 } /* Literal.Number */
 .highlight .s { color: #BA2121 } /* Literal.String */
 .highlight .na { color: #687822 } /* Name.Attribute */
 .highlight .nb { color: #008000 } /* Name.Builtin */
-.highlight .nc { color: #0000FF; font-weight: bold } /* Name.Class */
-.highlight .no { color: #880000 } /* Name.Constant */
-.highlight .nd { color: #AA22FF } /* Name.Decorator */
+.highlight .nc { color: #00F; font-weight: bold } /* Name.Class */
+.highlight .no { color: #800 } /* Name.Constant */
+.highlight .nd { color: #A2F } /* Name.Decorator */
 .highlight .ni { color: #717171; font-weight: bold } /* Name.Entity */
 .highlight .ne { color: #CB3F38; font-weight: bold } /* Name.Exception */
-.highlight .nf { color: #0000FF } /* Name.Function */
+.highlight .nf { color: #00F } /* Name.Function */
 .highlight .nl { color: #767600 } /* Name.Label */
-.highlight .nn { color: #0000FF; font-weight: bold } /* Name.Namespace */
+.highlight .nn { color: #00F; font-weight: bold } /* Name.Namespace */
 .highlight .nt { color: #008000; font-weight: bold } /* Name.Tag */
 .highlight .nv { color: #19177C } /* Name.Variable */
-.highlight .ow { color: #AA22FF; font-weight: bold } /* Operator.Word */
-.highlight .w { color: #bbbbbb } /* Text.Whitespace */
-.highlight .mb { color: #666666 } /* Literal.Number.Bin */
-.highlight .mf { color: #666666 } /* Literal.Number.Float */
-.highlight .mh { color: #666666 } /* Literal.Number.Hex */
-.highlight .mi { color: #666666 } /* Literal.Number.Integer */
-.highlight .mo { color: #666666 } /* Literal.Number.Oct */
+.highlight .ow { color: #A2F; font-weight: bold } /* Operator.Word */
+.highlight .w { color: #BBB } /* Text.Whitespace */
+.highlight .mb { color: #666 } /* Literal.Number.Bin */
+.highlight .mf { color: #666 } /* Literal.Number.Float */
+.highlight .mh { color: #666 } /* Literal.Number.Hex */
+.highlight .mi { color: #666 } /* Literal.Number.Integer */
+.highlight .mo { color: #666 } /* Literal.Number.Oct */
 .highlight .sa { color: #BA2121 } /* Literal.String.Affix */
 .highlight .sb { color: #BA2121 } /* Literal.String.Backtick */
 .highlight .sc { color: #BA2121 } /* Literal.String.Char */
@@ -67,9 +67,9 @@ span.linenos.special { color: #000000; background-color: #ffffc0; padding-left:
 .highlight .s1 { color: #BA2121 } /* Literal.String.Single */
 .highlight .ss { color: #19177C } /* Literal.String.Symbol */
 .highlight .bp { color: #008000 } /* Name.Builtin.Pseudo */
-.highlight .fm { color: #0000FF } /* Name.Function.Magic */
+.highlight .fm { color: #00F } /* Name.Function.Magic */
 .highlight .vc { color: #19177C } /* Name.Variable.Class */
 .highlight .vg { color: #19177C } /* Name.Variable.Global */
 .highlight .vi { color: #19177C } /* Name.Variable.Instance */
 .highlight .vm { color: #19177C } /* Name.Variable.Magic */
-.highlight .il { color: #666666 } /* Literal.Number.Integer.Long */
+.highlight .il { color: #666 } /* Literal.Number.Integer.Long */

docs/build/html/_static/searchtools.js

@@ -328,14 +328,13 @@ const Search = {
     for (const [title, foundTitles] of Object.entries(allTitles)) {
       if (title.toLowerCase().trim().includes(queryLower) && (queryLower.length >= title.length/2)) {
         for (const [file, id] of foundTitles) {
-          const score = Math.round(Scorer.title * queryLower.length / title.length);
-          const boost = titles[file] === title ? 1 : 0;  // add a boost for document titles
+          let score = Math.round(100 * queryLower.length / title.length)
           normalResults.push([
             docNames[file],
             titles[file] !== title ? `${titles[file]} > ${title}` : title,
             id !== null ? "#" + id : "",
             null,
-            score + boost,
+            score,
             filenames[file],
           ]);
         }