Vnbdev#3
Closed
sangyu wants to merge 6 commits intosangyu:feat_pooled_variance_hedges_g_for_delta_gfrom
Closed
Conversation
…t data
Remove overly aggressive NaN filtering in _check_errors() that was causing
data truncation when using wide-format paired data with different group sizes.
Problem:
When loading wide-format paired data created by concatenating DataFrames of
different lengths (e.g., 20, 10, and 40 samples), the package was removing
ALL rows with ANY NaN value across ALL columns. This truncated all groups
to the size of the smallest group.
Root Cause:
In _check_errors() method, the code had:
elif x is None and y is None:
self.__output_data.dropna(inplace=True)
This removed entire rows if they had NaN in ANY column, affecting all groups
even though NaN values were structural (from DataFrame concatenation) and not
actual missing data points.
Solution:
Removed the problematic elif block from _check_errors(). The downstream code
in _get_plot_data() already handles NaN values correctly by:
1. Using pd.melt() which preserves all non-NaN values
2. Calling dropna(subset=[self.__yvar]) which only removes rows with NaN in
the measurement column, not across all columns
Testing:
- Added test_33_multi_paired_different_sizes() to verify groups with 20, 10,
and 40 samples are preserved correctly
Add new blog post notebook and figure for the DABEST2 preprint.
Update _quarto.yml to exclude tests/ from Quarto rendering
Fix a bug when receiving wide-format paired data with different group sizes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.