Welcome to TransparentAI’s documentation!¶

This Python library was developed by Nathan Lauga a french Data Scientist.

If you find spelling or grammar mistakes that hurt your eyes, I apologize in advance. But do not hesitate to report them on the issues GitHub page from the library here : https://github.com/Nathanlauga/transparentai/issues

Quick start¶

Install the library with pypi :

>>> pip install transparentai

Installation¶

Install TransparentAI with PyPI :

>>> pip install transparentai

Or by cloning GitHub repository :

>>> git clone https://github.com/Nathanlauga/transparentai.git
>>> cd transparentai
>>> python setup.py install

Getting started with TransparentAI¶

This page will show you some code to start with the TransparentAI library.

In this section I created a binary classifier based on Adult dataset. The following variables will be used :

variable	description
data	Adult dataset as DataFrame
clf	Classifier model
y_true	True labels for train set
y_true_valid	True labels for valid set
y_pred	Predictions labels for train set
y_pred_valid	Predictions labels for valid set
df_valid	Dataframe for valid set
X_train	Features for train set
X_valid	Features for valid set

Is my model biased ?¶

>>> privileged_group = {
    # For gender attribute Male peoples are considered to be privileged
    'gender':['Male'],
    # For marital-status attribute Married peoples are considered to be privileged
    'marital-status': lambda x: 'Married' in x,
    # For race attribute White peoples are considered to be privileged
    'race':['White']
}

>>> from transparentai import fairness
>>> fairness.model_bias(y_true_valid, y_pred_valid, df_valid, privileged_group)
{
    "gender": {
        "statistical_parity_difference": -0.07283528047741014,
        "disparate_impact": 0.4032473042703101,
        "equal_opportunity_difference": -0.04900038770381182,
        "average_odds_difference": -0.026173142849183567
    },
    "marital-status": {
        "statistical_parity_difference": -0.11667610209029305,
        "disparate_impact": 0.27371312304160633,
        "equal_opportunity_difference": 0.08345535064884008,
        "average_odds_difference": 0.03867329810319946
    },
    "race": {
        "statistical_parity_difference": -0.0420778376239787,
        "disparate_impact": 0.5964166117990216,
        "equal_opportunity_difference": -0.0004408949904296522,
        "average_odds_difference": -0.002870373184105955
    }
}

This metrics can be not easy to understand so you can use the returns_text=True so that you can get ths insight :

>>> fairness_txt = fairness.model_bias(y_true_valid, y_pred_valid, df_valid, privileged_group, returns_text=True)
>>> print(fairness_txt['gender'])
The privileged group is predicted with the positive output 7.28% more often than the unprivileged group. This is considered to be fair.
The privileged group is predicted with the positive output 2.48 times more often than the unprivileged group. This is considered to be not fair.
For a person in the privileged group, the model predict a correct positive output 4.90% more often than a person in the unprivileged group. This is considered to be fair.
For a person in the privileged group, the model predict a correct positive output or a correct negative output 2.62% more often than a person in the unprivileged group. This is considered to be fair.
The model has 3 fair metrics over 4 (75%).

And if you like to get visual help use the plot_bias function :

>>> privileged_group = {'gender': ['Male']}
>>> from transparentai import fairness
>>> fairness.plot_bias(y_true_valid, y_pred_valid, df_valid, privileged_group, with_text=True)

_images/fairness.plot_bias_binary_classifier.png

How can I explain my model ?¶

>>> from transparentai.models import explainers
>>> explainer = explainers.ModelExplainer(clf, X_train, model_type='tree')

>>> explainer.explain_global_influence(X_train, nsamples=1000)
{
    'age': 0.08075649984055841,
    'fnlwgt': 0.05476459574744569,
    'education-num': 0.08048316800088552,
    'capital-gain': 0.06879137962639843,
    'capital-loss': 0.018367250661071737,
    'hours-per-week': 0.06009733425389803
}

>>> explainer.plot_global_explain()

_images/explainer.plot_global_explain_binary_classifier.png

>>> explainer.plot_local_explain(X_valid.iloc[0])

_images/explainer.plot_local_explain_binary_classifier.png

What’s my model performance ?¶

>>> from transparentai.models import classification

>>> # You can use custom function with lambda
>>> metrics = ['accuracy', 'roc_auc', 'f1', 'recall', 'precision', lambda y_true, y_pred: sum(y_true-y_pred)]
>>> classification.compute_metrics(y_true_valid, y_pred_valid, metrics)
{
    'accuracy': 0.812011415808413,
    'roc_auc': 0.8272860034692258,
    'f1': 0.5682530635508691,
    'recall': 0.5244608100999474,
    'precision': 0.6200248756218906,
    'custom_1': 586
}

>>> classification.plot_performance(y_true, y_pred, y_true_valid, y_pred_valid)

_images/classification.plot_performance_binary_classifier.png

What is in my data ?¶

>>> from transparentai.datasets import variable
>>> variable.plot_variable(data['age'])

>>> variable.plot_variable(data['capital-loss'], legend=data['income'], ylog=True)

_images/variable.plot_variable_capital_loss.png

>>> variable.plot_variable(data['workclass'])

_images/variable.plot_variable_workclass.png

The birthdate column was generated based on the age column.

>>> variable.plot_variable(data['birthdate'], legend=data['income'])

_images/variable.plot_variable_birthdate.png

How can I know the model is still good over time ?¶

timestamp variable was generated randomly, it represents the time of the prediction.

>>> from transparentai import monitoring
>>> monitoring.plot_monitoring(y_true, y_pred, timestamp, interval='month', classification=True)

_images/plot_monitoring_binary_classifier.png

Is my model sustainable ?¶

Estimate your training CO2 consumption.

>>> from transparentai import sustainable
>>> sustainable.estimate_co2(hours=24, location='France', watts=250)
3.18437946896484

Evaluate your training kWh consumption.

>>> from transparentai import sustainable
>>> kWh, clf = sustainable.evaluate_kWh(clf.fit, X, Y, verbose=True)
Location:                                                                 France
Baseline wattage:                                                     4.79 watts
Process wattage:                                                     18.45 watts
--------------------------------------------------------------------------------
-------------------------------  Final Readings  -------------------------------
--------------------------------------------------------------------------------
Average baseline wattage:                                             3.53 watts
Average total wattage:                                               16.04 watts
Average process wattage:                                             12.51 watts
Process duration:                                                        0:00:07
--------------------------------------------------------------------------------
-------------------------------   Energy Data    -------------------------------
--------------------------------------------------------------------------------
                              Energy mix in France
Coal:                                                                      3.12%
Petroleum:                                                                16.06%
Natural Gas:                                                              33.56%
Low Carbon:                                                               47.26%
--------------------------------------------------------------------------------
-------------------------------    Emissions     -------------------------------
--------------------------------------------------------------------------------
Effective emission:                                              1.32e-05 kg CO2
Equivalent miles driven:                                          5.39e-12 miles
Equivalent minutes of 32-inch LCD TV watched:                   8.14e-03 minutes
Percentage of CO2 used in a US household/day:                          4.33e-12%
--------------------------------------------------------------------------------
------------------------- Assumed Carbon Equivalencies -------------------------
--------------------------------------------------------------------------------
Coal:                                                      995.725971 kg CO2/MWh
Petroleum:                                                816.6885263 kg CO2/MWh
Natural gas:                                              743.8415916 kg CO2/MWh
Low carbon:                                                         0 kg CO2/MWh
--------------------------------------------------------------------------------
-------------------------     Emissions Comparison     -------------------------
--------------------------------------------------------------------------------
                      Quantities below expressed in kg CO2
        US                      Europe                  Global minus US/Europe
Max:    Wyoming        2.85e-05 Kosovo         2.93e-05 Mongolia        2.86e-05
Median: Tennessee      1.40e-05 Ukraine        2.04e-05 Korea, South    2.34e-05
Min:    Vermont        8.00e-07 Iceland        5.26e-06 Bhutan          3.26e-06
--------------------------------------------------------------------------------
--------------------------------------------------------------------------------
Process used:                                                       3.10e-05 kWh

Do I use safe packages ?¶

>>> import transparentai.utils as utils
>>> utils.check_packages_security(full_report=True)
+==============================================================================+
|                                                                              |
|                               /$$$$$$            /$$                         |
|                              /$$__  $$          | $$                         |
|           /$$$$$$$  /$$$$$$ | $$  \__//$$$$$$  /$$$$$$   /$$   /$$           |
|          /$$_____/ |____  $$| $$$$   /$$__  $$|_  $$_/  | $$  | $$           |
|         |  $$$$$$   /$$$$$$$| $$_/  | $$$$$$$$  | $$    | $$  | $$           |
|          \____  $$ /$$__  $$| $$    | $$_____/  | $$ /$$| $$  | $$           |
|          /$$$$$$$/|  $$$$$$$| $$    |  $$$$$$$  |  $$$$/|  $$$$$$$           |
|         |_______/  \_______/|__/     \_______/   \___/   \____  $$           |
|                                                          /$$  | $$           |
|                                                         |  $$$$$$/           |
|  by pyup.io                                              \______/            |
|                                                                              |
+==============================================================================+
| REPORT                                                                       |
| checked 77 packages, using default DB                                        |
+==============================================================================+
| No known security vulnerabilities found.                                     |
+==============================================================================+

List of preformated metrics¶

Evaluation metrics¶

If you use the compute_metrics function in the classification.compute_metrics or regression.compute_metrics functions there are preformated metrics.

You can see details of the code in the documentation page : transparentai.models

This is the list :

Problem type	metric name
classification	`'accuracy'`
classification	`'balanced_accuracy'`
classification	`'average_precision'`
classification	`'brier_score'`
classification	`'f1'`
classification	`'f1_micro'`
classification	`'f1_macro'`
classification	`'f1_weighted'`
classification	`'f1_samples'`
classification	`'log_loss'`
classification	`'precision'`
classification	`'precision_micro'`
classification	`'recall'`
classification	`'recall_micro'`
classification	`'true_positive_rate'`
classification	`'false_positive_rate'`
classification	`'jaccard'`
classification	`'matthews_corrcoef'`
classification	`'roc_auc'`
classification	`'roc_auc_ovr'`
classification	`'roc_auc_ovo'`
classification	`'roc_auc_ovr_weighted'`
classification	`'roc_auc_ovo_weighted'`
classification	`'true_positives'`
classification	`'false_positives'`
classification	`'false_negatives'`
classification	`'true_negatives'`
classification	`'confusion_matrix'`
regression	`'max_error'`
regression	`'mean_absolute_error'`
regression	`'mean_squared_error'`
regression	`'root_mean_squared_error'`
regression	`'mean_squared_log_error'`
regression	`'median_absolute_error'`
regression	`'r2'`
regression	`'mean_poisson_deviance'`
regression	`'mean_gamma_deviance'`

`transparentai.datasets`¶

Variable submodule¶

transparentai.datasets.variable.variable.describe_number(arr)[source]¶

Descriptive statistics about a number array.

Returned statistics:

Count of valid values
Count of missing values
Mean
Mode
Min
Quantitle 25%
Median
Quantile 75%
Max

Parameters:	arr (array like) – Array of value to get desriptive statistics from
Raises:	TypeError: – arr is not an array like TypeError: – arr is not a number array

transparentai.datasets.variable.variable.describe_datetime(arr, format='%Y-%m-%d')[source]¶

Descriptive statistics about a datetime array.

Returned statistics:

Count of valid values
Count of missing values
Count of unique values
Most common value
Min
Mean
Max

Parameters:	arr (array like) – Array of value to get desriptive statistics from format (str) – String format for datetime value
Raises:	TypeError: – arr is not an array like TypeError: – arr is not a datetime array

transparentai.datasets.variable.variable.describe_object(arr)[source]¶

Descriptive statistics about an object array.

Returned statistics:

Count of valid values
Count of missing values
Count of unique values
Most common value

Parameters:	arr (array like) – Array of value to get desriptive statistics from
Raises:	TypeError: – arr is not an array like TypeError: – arr is not an object array

transparentai.datasets.variable.variable.describe(arr)[source]¶

Descriptive statistics about an array. Depending on the detected dtype (number, date, object) it returns specific stats.

Common statistics for all dtype (using describe_common):

Count of valid values
Count of missing values

Number statistics (using describe_number):

Mean
Mode
Min
Quantitle 25%
Median
Quantile 75%
Max

Datetime statistics (using describe_datetime):

Count of unique values
Most common value
Min
Mean
Max

Object statistics (using describe_datetime):

Count of unique values
Most common value

Parameters:	arr (array like) – Array of value to get desriptive statistics from
Returns:	Dictionnary with descriptive statistics
Return type:	dict
Raises:	TypeError: – arr is not an array like

transparentai.datasets.variable.correlation.compute_correlation(df, nrows=None, max_cat_val=100)[source]¶

Computes differents correlations matrix for three cases and merge them:

numerical to numerical (using Pearson coeff)
categorical to categorical (using Cramers V & Chi square)
numerical to categorical (discrete) (using Point Biserial)

/!\ ==== Caution ==== /!\

This matrix has a default : the cramers_v_corr is scale from 0 to 1, but the others are from to -1 to 1. Be sure to understand this.

Pearson coeff Wikipedia definition :

In statistics, the Pearson correlation coefficient, also referred to as Pearson’s r, the Pearson product-moment correlation coefficient (PPMCC) or the bivariate correlation, is a statistic that measures linear correlation between two variables X and Y. It has a value between +1 and −1, where 1 is total positive linear correlation, 0 is no linear correlation, and −1 is total negative linear correlation (that the value lies between -1 and 1 is a consequence of the Cauchy–Schwarz inequality). It is widely used in the sciences.

Cramers V Wikipedia definition :

In statistics, Cramér’s V (sometimes referred to as Cramér’s phi and denoted as φc) is a measure of association between two nominal variables, giving a value between 0 and +1 (inclusive). It is based on Pearson’s chi-squared statistic and was published by Harald Cramér in 1946.

Point Biserial Wikipedia definition :

The point biserial correlation coefficient (rpb) is a correlation coefficient used when one variable (e.g. Y) is dichotomous; Y can either be “naturally” dichotomous, like whether a coin lands heads or tails, or an artificially dichotomized variable. In most situations it is not advisable to dichotomize variables artificially[citation needed]. When a new variable is artificially dichotomized the new dichotomous variable may be conceptualized as having an underlying continuity. If this is the case, a biserial correlation would be the more appropriate calculation.

Parameters:	df (pd.DataFrame) – pandas Dataframe with values to compute correlation nrows (None or int or float (default None)) – If not None reduce the data to a sample of nrows if int else if float reduce to len(df) * nrows max_cat_val (int or None (default 100)) – Number max of unique values in a categorical feature if there are more distinct values than this number then the feature is ignored
Returns:	Correlation matrix computed with Pearson coeff for numerical features to numerical features, Cramers V for categorical features to categorical features and Point Biserial for categorical features to numerical features
Return type:	pd.DataFrame
Raises:	TypeError: – Must provide a pandas DataFrame representing the data

transparentai.datasets.variable.correlation.compute_cramers_v_corr(df)[source]¶

Computes Cramers V correlation for a dataframe.

Cramers V Wikipedia definition :

In statistics, Cramér’s V (sometimes referred to as Cramér’s phi and denoted as φc) is a measure of association between two nominal variables, giving a value between 0 and +1 (inclusive). It is based on Pearson’s chi-squared statistic and was published by Harald Cramér in 1946.

Parameters:	df (pd.DataFrame) – pandas Dataframe with values to compute Cramers V correlation
Returns:	Correlation matrix computed for Cramers V coeff
Return type:	pd.DataFrame
Raises:	TypeError: – Must provide a pandas DataFrame representing the data

transparentai.datasets.variable.correlation.compute_pointbiserialr_corr(df, cat_feats=None, num_feats=None)[source]¶

Computes Point Biserial correlation for a dataframe.

Point Biserial Wikipedia definition :

The point biserial correlation coefficient (rpb) is a correlation coefficient used when one variable (e.g. Y) is dichotomous; Y can either be “naturally” dichotomous, like whether a coin lands heads or tails, or an artificially dichotomized variable. In most situations it is not advisable to dichotomize variables artificially[citation needed]. When a new variable is artificially dichotomized the new dichotomous variable may be conceptualized as having an underlying continuity. If this is the case, a biserial correlation would be the more appropriate calculation.

Parameters:	df (pd.DataFrame) – pandas Dataframe with values to compute Point Biserial correlation
Returns:	Correlation matrix computed for Point Biserial coeff
Return type:	pd.DataFrame
Raises:	TypeError: – Must provide a pandas DataFrame representing the data ValueError: – cat_feats and num_feats must be set or be both None TypeError: – cat_feats must be a list TypeError: – num_feats must be a list

transparentai.datasets.variable.correlation.cramers_v(x, y)[source]¶

Returns the Cramer V value of two categorical variables using chi square. This correlation metric is between 0 and 1.

Code source found in this article : https://towardsdatascience.com/the-search-for-categorical-correlation-a1cf7f1888c9

Parameters:	x (array like) – first categorical variable y (array like) – second categorical variable
Returns:	Cramer V value
Return type:	float

transparentai.datasets.variable.correlation.merge_corr_df(df_list)[source]¶

Merges correlation matrix from compute_correlation() function to one. Needs 3 dataframe : pearson_corr, cramers_v_corr and pbs_corr.

This matrix has a default : the cramers_v_corr is scale from 0 to 1, but the others are from to -1 to 1. Be sure to understand this.

Parameters:	df_list (list) – List of correlation matrices
Returns:	Merged dataframe of correlation matrices
Return type:	pd.DataFrame

Preformated datasets¶

transparentai.datasets.datasets.load_adult()[source]¶: Load Adult dataset. Source : https://archive.ics.uci.edu/ml/datasets/Adult

transparentai.datasets.datasets.load_boston()[source]¶: Load boston dataset Source : https://archive.ics.uci.edu/ml/machine-learning-databases/housing/

transparentai.datasets.datasets.load_iris()[source]¶: Load Iris dataset. Source : http://archive.ics.uci.edu/ml/datasets/Iris/

`transparentai.fairness`¶

Fairness module¶

transparentai.fairness.fairness.create_privilieged_df(df, privileged_group)[source]¶

Returns a formated dataframe with protected attribute columns and whether the row is privileged (1) or not (0).

example of a privileged_group dictionnary :

>>> privileged_group = {
     'gender':['Male'],                # privileged group is man for gender attribute
     'age': lambda x: x > 30 & x < 55  # privileged group aged between 30 and 55 years old
 }

Parameters:	df (pd.DataFrame) – Dataframe to extract privilieged group from. privileged_group (dict) – Dictionnary with protected attribute as key (e.g. age or gender) and a list of favorable value (like [‘Male’]) or a function returning a boolean corresponding to a privileged group
Returns:	DataFrame with protected attribute columns and whether the row is privileged (1) or not (0)
Return type:	pd.DataFrame
Raises:	TypeError: – df is not a pandas.DataFrame TypeError: – privileged_group is not a dictionnary ValueError: – privileged_group has not valid keys (in df columns)

transparentai.fairness.fairness.compute_fairness_metrics(y_true, y_pred, df, privileged_group, metrics=None, pos_label=1, regr_split=None)[source]¶

Computes the fairness metrics for one attribute

metrics can have str or function. If it’s a string then it has to be a key from FAIRNESS_METRICS global variable dict. By default it uses the 5 fairness function :

statistical_parity_difference
disparate_impact
equal_opportunity_difference
average_odds_difference
theil_index

You can also use it for a regression problem. You can set a value in the regr_split argument so it converts it to a binary classification problem. To use the mean use ‘mean’. If the favorable label is more than the split value set pos_label argument to 1 else to 0.

Example

>>> from transparentai.datasets import load_boston
>>> from sklearn.linear_model import LinearRegression

>>> data = load_boston()
>>> X, y = data.drop(columns='MEDV'), data['MEDV']
>>> regr = LinearRegression().fit(X, y)

>>> privileged_group = {
    'AGE': lambda x: (x > 30) & (x < 55)
}
>>> y_true, y_pred = y, regr.predict(X)
>>> compute_fairness_metrics(y_true, y_pred, data,
                             privileged_group, regr_split='mean')
{'AGE': {'statistical_parity_difference': -0.2041836536594836,
  'disparate_impact': 0.674582301980198,
  'equal_opportunity_difference': 0.018181818181818188,
  'average_odds_difference': -0.0884835589941973,
  'theil_index': 0.06976073748626294}}

Returns a dictionnary with protected attributes name’s as key containing a dictionnary with metric’s name as key and metric function’s result as value

Parameters:	y_true (array like) – True labels y_pred (array like) – Predicted labels df (pd.DataFrame) – Dataframe to extract privilieged group from. privileged_group (dict) – Dictionnary with protected attribute as key (e.g. age or gender) and a list of favorable value (like [‘Male’]) or a function returning a boolean corresponding to a privileged group metrics (list (default None)) – List of metrics to compute, if None then it uses the 5 default Fairness function pos_label (number) – The label of the positive class. regr_split ('mean' or number (default None)) – If it’s a regression problem then you can convert result to a binary classification using ‘mean’ or a choosen number. both y_true and y_pred become 0 and 1 : 0 if it’s equal or less than the split value (the average if ‘mean’) and 1 if more. If the favorable label is more than the split value set pos_label=1 else pos_label=0
Returns:	Dictionnary with protected attributes name’s as key containing a dictionnary with metric’s name as key and metric function’s result as value
Return type:	dict
Raises:	ValueError: – y_true and y_pred must have the same length ValueError: – y_true and df must have the same length TypeError: – metrics must be a list

transparentai.fairness.fairness.model_bias(y_true, y_pred, df, privileged_group, pos_label=1, regr_split=None, returns_text=False)[source]¶

Computes the fairness metrics for protected attributes refered in the privileged_group argument.

It uses the 4 fairness function :

statistical_parity_difference
disparate_impact
equal_opportunity_difference
average_odds_difference

You can also use it for a regression problem. You can set a value in the regr_split argument so it converts it to a binary classification problem. To use the mean use ‘mean’. If the favorable label is more than the split value set pos_label argument to 1 else to 0.

This function is using the fairness.compute_metrics function. So if returns_text is False then it’s the same output.

Example

>>> from transparentai.datasets import load_boston
>>> from sklearn.linear_model import LinearRegression

>>> data = load_boston()
>>> X, y = data.drop(columns='MEDV'), data['MEDV']
>>> regr = LinearRegression().fit(X, y)

>>> privileged_group = {
    'AGE': lambda x: (x > 30) & (x < 55)
}
>>> y_true, y_pred = y, regr.predict(X)
>>> model_bias(y_true, y_pred, data,
               privileged_group, regr_split='mean')
{'AGE': {'statistical_parity_difference': -0.2041836536594836,
  'disparate_impact': 0.674582301980198,
  'equal_opportunity_difference': 0.018181818181818188,
  'average_odds_difference': -0.0884835589941973,
  'theil_index': 0.06976073748626294}}

>>> bias_txt = model_bias(y_true, y_pred, data,
                          privileged_group, regr_split='mean',
                          returns_text=True)
>>> print(bias_txt['AGE'])
The privileged group is predicted with the positive output 20.42% more often than the unprivileged group. This is considered to be not fair.
The privileged group is predicted with the positive output 1.48 times more often than the unprivileged group. This is considered to be not fair.
For a person in the unprivileged group, the model predict a correct positive output 1.82% more often than a person in the privileged group. This is considered to be fair.
For a person in the privileged group, the model predict a correct positive output or a correct negative output 8.85% more often than a person in the unprivileged group. This is considered to be fair.
The model has 2 fair metrics over 4 (50%).

Parameters:	y_true (array like) – True labels y_pred (array like) – Predicted labels df (pd.DataFrame) – Dataframe to extract privilieged group from. privileged_group (dict) – Dictionnary with protected attribute as key (e.g. age or gender) and a list of favorable value (like [‘Male’]) or a function returning a boolean corresponding to a privileged group pos_label (number) – The label of the positive class. regr_split ('mean' or number (default None)) – If it’s a regression problem then you can convert result to a binary classification using ‘mean’ or a choosen number. both y_true and y_pred become 0 and 1 : 0 if it’s equal or less than the split value (the average if ‘mean’) and 1 if more. If the favorable label is more than the split value set pos_label=1 else pos_label=0 returns_text (bool (default False)) – Whether it return computed metrics score or a text explaination for the computed bias.
Returns:	Dictionnary with metric’s name as key and metric function’s result as value if returns_text is False else it returns a text explaining the model fairness over the 4 metrics.
Return type:	dict

transparentai.fairness.fairness.find_correlated_feature(df, privileged_group, corr_threshold=0.4)[source]¶

Finds correlated feature with protected attribute set in the privileged_group argument.

This function is a helper to find out if protected attribute can be found in other features.

Returns a dictionnary with protected attributes name’s as key containing a dictionnary with metric’s name as key and metric function’s result as value.

Example

>>> from transparentai.datasets import load_adult
>>> from transparentai import fairness

>>> data = load_adult()

>>> privileged_group = {
        'gender':['Male'],
        'marital-status': lambda x: 'Married' in x,
        'race':['White']
    }

>>> fairness.find_correlated_feature(data, privileged_group,
                                     corr_threshold=0.4)
{'gender': {'marital-status': 0.4593,
  'occupation': 0.4239,
  'relationship': 0.6465},
 'marital-status': {'relationship': 0.4881,
  'gender': 0.4593,
  'income': 0.4482},
 'race': {'native-country': 0.4006}}

Parameters:	df (pd.DataFrame) – Dataframe to extract privilieged group from. privileged_group (dict) – Dictionnary with protected attribute as key (e.g. age or gender) and a list of favorable value (like [‘Male’]) or a function returning a boolean corresponding to a privileged group corr_threshold (float (default 0.4)) – Threshold for which features are considered to be correlated
Returns:	Dictionnary with protected attributes name’s as key containing a dictionnary with correlated features as key and correlation coeff as value
Return type:	dict

Fairness metrics¶

Statistical parity difference¶

transparentai.fairness.metrics.statistical_parity_difference(y, prot_attr, pos_label=1)[source]¶

Computes the statistical parity difference for a protected attribute and a specified label

Computed as the difference of the rate of favorable outcomes received by the unprivileged group to the privileged group.

The ideal value of this metric is 0 A value < 0 implies higher benefit for the privileged group and a value > 0 implies a higher benefit for the unprivileged group.

Fairness for this metric is between -0.1 and 0.1

\[Pr(\hat{Y} = v | D = \text{unprivileged}) - Pr(\hat{Y} = v | D = \text{privileged})\]

code source inspired from aif360 statistical_parity_difference

Parameters:	y (array like) – list of predicted labels prot_attr (array like) – Array of 0 and 1 same length as y which indicates if the row is member of a privileged group or not pos_label (int (default 1)) – number of the positive label
Returns:	Statistical parity difference bias metric
Return type:	float
Raises:	ValueError: – y and prot_attr must have the same length

Disparate Impact¶

transparentai.fairness.metrics.disparate_impact(y, prot_attr, pos_label=1)[source]¶

Computes the Disparate impact for a protected attribute and a specified label

Computed as the ratio of rate of favorable outcome for the unprivileged group to that of the privileged group.

The ideal value of this metric is 1.0 A value < 1 implies higher benefit for the privileged group and a value > 1 implies a higher benefit for the unprivileged group.

Fairness for this metric is between 0.8 and 1.2

\[\frac{Pr(\hat{Y} = v | D = \text{unprivileged})} {Pr(\hat{Y} = v | D = \text{privileged})}\]

code source inspired from aif360 disparate_impact

Parameters:	y (array like) – list of predicted labels prot_attr (array like) – Array of 0 and 1 same length as y which indicates if the row is member of a privileged group or not pos_label (int (default 1)) – number of the positive label
Returns:	Disparate impact bias metric
Return type:	float
Raises:	ValueError: – y and prot_attr must have the same length

Equal opportunity difference¶

transparentai.fairness.metrics.equal_opportunity_difference(y_true, y_pred, prot_attr, pos_label=1)[source]¶

Computes the equal opportunity difference for a protected attribute and a specified label

This metric is computed as the difference of true positive rates between the unprivileged and the privileged groups. The true positive rate is the ratio of true positives to the total number of actual positives for a given group.

The ideal value is 0. A value of < 0 implies higher benefit for the privileged group and a value > 0 implies higher benefit for the unprivileged group.

Fairness for this metric is between -0.1 and 0.1

\(TPR_{D = \text{unprivileged}} - TPR_{D = \text{privileged}}\)

code source from aif360 equal_opportunity_difference

Parameters:	y_true (array like) – True labels y_pred (array like) – Predicted labels prot_attr (array like) – Array of 0 and 1 same length as y which indicates if the row is member of a privileged group or not pos_label (int (default 1)) – number of the positive label
Returns:	Equal opportunity difference bias metric
Return type:	float
Raises:	ValueError: – y_true and y_pred must have the same length ValueError: – y_true and prot_attr must have the same length

Average odds difference¶

transparentai.fairness.metrics.average_odds_difference(y_true, y_pred, prot_attr, pos_label=1)[source]¶

Computes the average odds difference for a protected attribute and a specified label

Computed as average difference of false positive rate (false positives / negatives) and true positive rate (true positives / positives) between unprivileged and privileged groups.

The ideal value of this metric is 0. A value of < 0 implies higher benefit for the privileged group and a value > 0 implies higher benefit for the unprivileged group.

Fairness for this metric is between -0.1 and 0.1

\[\frac{1}{2}\left[|FPR_{D = \text{unprivileged}} - FPR_{D = \text{privileged}}| + |TPR_{D = \text{unprivileged}} - TPR_{D = \text{privileged}}|\right]\]

A value of 0 indicates equality of odds.

code source from aif360 average_odds_difference

Parameters:	y_true (array like) – True labels y_pred (array like) – Predicted labels prot_attr (array like) – Array of 0 and 1 same length as y which indicates if the row is member of a privileged group or not pos_label (int (default 1)) – number of the positive label
Returns:	Average of absolute difference bias metric
Return type:	float
Raises:	ValueError: – y_true and y_pred must have the same length ValueError: – y_true and prot_attr must have the same length

Theil Index¶

transparentai.fairness.metrics.theil_index(y_true, y_pred, prot_attr, pos_label=1)[source]¶

Computes the theil index for a protected attribute and a specified label

Computed as the generalized entropy of benefit for all individuals in the dataset, with alpha = 1. It measures the inequality in benefit allocation for individuals.

A value of 0 implies perfect fairness.

Fairness is indicated by lower scores, higher scores are problematic

With \(b_i = \hat{y}_i - y_i + 1\):

\[\frac{1}{n}\sum_{i=1}^n\frac{b_{i}}{\mu}\ln\frac{b_{i}}{\mu}\]

code source from aif360 theil_index

Parameters:	y_true (array like) – True labels y_pred (array like) – Predicted labels prot_attr (array like) – Array of 0 and 1 same length as y which indicates if the row is member of a privileged group or not pos_label (int (default 1)) – number of the positive label
Returns:	Theil index bias metric
Return type:	float
Raises:	ValueError: – y_true and y_pred must have the same length ValueError: – y_true and prot_attr must have the same length

`transparentai.models`¶

Evaluation submodule¶

transparentai.models.evaluation.evaluation.compute_metrics(y_true, y_pred, metrics, classification=True)[source]¶

Computes the inputed metrics.

metrics can have str or function. If it’s a string then it has to be a key from METRICS global variable dict.

Returns a dictionnary with metric’s name as key and metric function’s result as value

Parameters:	y_true (array like) – True labels y_pred (array like) – Predicted labels metrics (list) – List of metrics to compute classification (bool (default True)) – Whether the ML task is a classification or not
Returns:	Dictionnary with metric’s name as key and metric function’s result as value
Return type:	dict
Raises:	TypeError: – metrics must be a list

Classification metrics¶

transparentai.models.evaluation.classification.accuracy(y_true, y_pred, **args)[source]¶

Accuracy score based on the sklearn.metrics.accuracy_score function.

More details here : Accuracy score

transparentai.models.evaluation.classification.average_precision(y_true, y_prob, **args)[source]¶

Average prevision score based on the sklearn.metrics.average_precision_score function.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.balanced_accuracy(y_true, y_pred, **args)[source]¶

Balanced accuracy score based on the sklearn.metrics.balanced_accuracy_score function.

More details here : Balanced accuracy score

transparentai.models.evaluation.classification.brier_score(y_true, y_prob, **args)[source]¶

Brier score based on the sklearn.metrics.brier_score_loss function.

More details here : Probability calibration

transparentai.models.evaluation.classification.confusion_matrix(y_true, y_pred, **args)[source]¶

Confusion matrix based on the sklearn.metrics.confusion_matrix function.

More details here : Confusion matrix

transparentai.models.evaluation.classification.f1(y_true, y_pred, **args)[source]¶

F1 score based on the sklearn.metrics.f1_score function.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.f1_macro(y_true, y_pred, **args)[source]¶

F1 score based on the sklearn.metrics.f1_score function.

Average argument set to ‘macro’.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.f1_micro(y_true, y_pred, **args)[source]¶

F1 score based on the sklearn.metrics.f1_score function.

Average argument set to ‘micro’.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.f1_samples(y_true, y_pred, **args)[source]¶

F1 score based on the sklearn.metrics.f1_score function.

Average argument set to ‘samples’.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.f1_weighted(y_true, y_pred, **args)[source]¶

F1 score based on the sklearn.metrics.f1_score function.

Average argument set to ‘weighted’.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.false_negatives(y_true, y_pred, pos_label=1)[source]¶

Returns the number of false negatives given a class number.

\[FN = \sum_{i}^n (y_i = 1) \& (\hat{y}_i \ne 1)\]

Parameters:	y_true (array like) – True labels y_pred (array like) – Predicted labels pos_label (int (default 1)) – Label class number (if binary classification then it’s 1)
Returns:	Number of false negatives
Return type:	int

transparentai.models.evaluation.classification.false_positive_rate(y_true, y_pred, pos_label=1)[source]¶

transparentai.models.evaluation.classification.false_positives(y_true, y_pred, pos_label=1)[source]¶

Returns the number of false positives given a class number.

\[FP = \sum_{i}^{n} (y_i \ne 1) \& (\hat{y}_i = 1)\]

Parameters:	y_true (array like) – True labels y_pred (array like) – Predicted labels pos_label (int (default 1)) – Label class number (if binary classification then it’s 1)
Returns:	Number of false positives
Return type:	int

transparentai.models.evaluation.classification.jaccard(y_true, y_pred, **args)[source]¶

Jaccard score based on the sklearn.metrics.jaccard_score function.

More details here : Jaccard similarity coefficient score

transparentai.models.evaluation.classification.log_loss(y_true, y_prob, **args)[source]¶

Log loss based on the sklearn.metrics.log_loss function.

More details here : Log loss

transparentai.models.evaluation.classification.matthews_corrcoef(y_true, y_pred, **args)[source]¶

Matthews correlation coefficient based on the sklearn.metrics.matthews_corrcoef function.

More details here : Matthews correlation coefficient

transparentai.models.evaluation.classification.precision(y_true, y_pred, **args)[source]¶

Precision score based on the sklearn.metrics.precision_score function.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.precision_micro(y_true, y_pred, **args)[source]¶

Precision score based on the sklearn.metrics.precision_score function.

Average argument set to ‘micro’.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.recall(y_true, y_pred, **args)[source]¶

Recall score based on the sklearn.metrics.recall_score function.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.recall_micro(y_true, y_pred, **args)[source]¶

Recall score based on the sklearn.metrics.recall_score function.

Average argument set to ‘micro’.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.roc_auc(y_true, y_prob, **args)[source]¶