transparentai.models

Evaluation submodule

transparentai.models.evaluation.evaluation.compute_metrics(y_true, y_pred, metrics, classification=True)[source]

Computes the inputed metrics.

metrics can have str or function. If it’s a string then it has to be a key from METRICS global variable dict.

Returns a dictionnary with metric’s name as key and metric function’s result as value

Parameters:
  • y_true (array like) – True labels
  • y_pred (array like) – Predicted labels
  • metrics (list) – List of metrics to compute
  • classification (bool (default True)) – Whether the ML task is a classification or not
Returns:

Dictionnary with metric’s name as key and metric function’s result as value
Return type:

dict
Raises:

TypeError: – metrics must be a list

Classification metrics

transparentai.models.evaluation.classification.accuracy(y_true, y_pred, **args)[source]

Accuracy score based on the sklearn.metrics.accuracy_score function.

More details here : Accuracy score

transparentai.models.evaluation.classification.average_precision(y_true, y_prob, **args)[source]

Average prevision score based on the sklearn.metrics.average_precision_score function.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.balanced_accuracy(y_true, y_pred, **args)[source]

Balanced accuracy score based on the sklearn.metrics.balanced_accuracy_score function.

More details here : Balanced accuracy score

transparentai.models.evaluation.classification.brier_score(y_true, y_prob, **args)[source]

Brier score based on the sklearn.metrics.brier_score_loss function.

More details here : Probability calibration

transparentai.models.evaluation.classification.confusion_matrix(y_true, y_pred, **args)[source]

Confusion matrix based on the sklearn.metrics.confusion_matrix function.

More details here : Confusion matrix

transparentai.models.evaluation.classification.f1(y_true, y_pred, **args)[source]

F1 score based on the sklearn.metrics.f1_score function.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.f1_macro(y_true, y_pred, **args)[source]

F1 score based on the sklearn.metrics.f1_score function.

Average argument set to ‘macro’.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.f1_micro(y_true, y_pred, **args)[source]

F1 score based on the sklearn.metrics.f1_score function.

Average argument set to ‘micro’.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.f1_samples(y_true, y_pred, **args)[source]

F1 score based on the sklearn.metrics.f1_score function.

Average argument set to ‘samples’.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.f1_weighted(y_true, y_pred, **args)[source]

F1 score based on the sklearn.metrics.f1_score function.

Average argument set to ‘weighted’.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.false_negatives(y_true, y_pred, pos_label=1)[source]

Returns the number of false negatives given a class number.

\[FN = \sum_{i}^n (y_i = 1) \& (\hat{y}_i \ne 1)\]
Parameters:
  • y_true (array like) – True labels
  • y_pred (array like) – Predicted labels
  • pos_label (int (default 1)) – Label class number (if binary classification then it’s 1)
Returns:

Number of false negatives
Return type:

int
transparentai.models.evaluation.classification.false_positive_rate(y_true, y_pred, pos_label=1)[source]
transparentai.models.evaluation.classification.false_positives(y_true, y_pred, pos_label=1)[source]

Returns the number of false positives given a class number.

\[FP = \sum_{i}^{n} (y_i \ne 1) \& (\hat{y}_i = 1)\]
Parameters:
  • y_true (array like) – True labels
  • y_pred (array like) – Predicted labels
  • pos_label (int (default 1)) – Label class number (if binary classification then it’s 1)
Returns:

Number of false positives
Return type:

int
transparentai.models.evaluation.classification.jaccard(y_true, y_pred, **args)[source]

Jaccard score based on the sklearn.metrics.jaccard_score function.

More details here : Jaccard similarity coefficient score

transparentai.models.evaluation.classification.log_loss(y_true, y_prob, **args)[source]

Log loss based on the sklearn.metrics.log_loss function.

More details here : Log loss

transparentai.models.evaluation.classification.matthews_corrcoef(y_true, y_pred, **args)[source]

Matthews correlation coefficient based on the sklearn.metrics.matthews_corrcoef function.

More details here : Matthews correlation coefficient

transparentai.models.evaluation.classification.precision(y_true, y_pred, **args)[source]

Precision score based on the sklearn.metrics.precision_score function.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.precision_micro(y_true, y_pred, **args)[source]

Precision score based on the sklearn.metrics.precision_score function.

Average argument set to ‘micro’.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.recall(y_true, y_pred, **args)[source]

Recall score based on the sklearn.metrics.recall_score function.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.recall_micro(y_true, y_pred, **args)[source]

Recall score based on the sklearn.metrics.recall_score function.

Average argument set to ‘micro’.

More details here : Precision, recall and F-measures

transparentai.models.evaluation.classification.roc_auc(y_true, y_prob, **args)[source]

Area Under the Receiver Operating Characteristic Curve (ROC AUC) score based on the sklearn.metrics.roc_auc_score function.

More details here : Receiver operating characteristic (ROC)

transparentai.models.evaluation.classification.roc_auc_ovo(y_true, y_prob, **args)[source]

Area Under the Receiver Operating Characteristic Curve (ROC AUC) score based on the sklearn.metrics.roc_auc_score function.

multi_class argument is set to ‘ovo’.

More details here : Receiver operating characteristic (ROC)

transparentai.models.evaluation.classification.roc_auc_ovo_weighted(y_true, y_prob, **args)[source]

Area Under the Receiver Operating Characteristic Curve (ROC AUC) score based on the sklearn.metrics.roc_auc_score function.

Average argument set to ‘weighted’ and multi_class to ‘ovo’.

More details here : Receiver operating characteristic (ROC)

transparentai.models.evaluation.classification.roc_auc_ovr(y_true, y_prob, **args)[source]

Area Under the Receiver Operating Characteristic Curve (ROC AUC) score based on the sklearn.metrics.roc_auc_score function.

multi_class argument is set to ‘ovr’.

More details here : Receiver operating characteristic (ROC)

transparentai.models.evaluation.classification.roc_auc_ovr_weighted(y_true, y_prob, **args)[source]

Area Under the Receiver Operating Characteristic Curve (ROC AUC) score based on the sklearn.metrics.roc_auc_score function.

Average argument set to ‘weighted’ and multi_class to ‘ovr’.

More details here : Receiver operating characteristic (ROC)

transparentai.models.evaluation.classification.roc_curve(y_true, y_prob, **args)[source]

Area Under the Receiver Operating Characteristic Curve (ROC AUC) score based on the sklearn.metrics.roc_auc_score function.

More details here : Receiver operating characteristic (ROC)

transparentai.models.evaluation.classification.true_negatives(y_true, y_pred, pos_label=1)[source]

Returns the number of true negatives given a class number.

\[TN = \sum_{i}^{n} (y_i \ne 1) \& (\hat{y}_i \ne 1)\]
Parameters:
  • y_true (array like) – True labels
  • y_pred (array like) – Predicted labels
  • pos_label (int (default 1)) – Label class number (if binary classification then it’s 1)
Returns:

Number of true negatives
Return type:

int
transparentai.models.evaluation.classification.true_positive_rate(y_true, y_pred, pos_label=1)[source]
transparentai.models.evaluation.classification.true_positives(y_true, y_pred, pos_label=1)[source]

Returns the number of true positives given a class number.

\[TP = \sum_{i}^{n} (y_i = 1) \& (\hat{y}_i = 1)\]
Parameters:
  • y_true (array like) – True labels
  • y_pred (array like) – Predicted labels
  • pos_label (int (default 1)) – Label class number (if binary classification then it’s 1)
Returns:

Number of true positives
Return type:

int

Regression metrics

transparentai.models.evaluation.regression.explained_variance(y_true, y_pred, **args)[source]

Explained variance score based on the sklearn.metrics.explained_variance_score function.

More details here : Explained variance score

transparentai.models.evaluation.regression.max_error(y_true, y_pred, **args)[source]

Max error based on the sklearn.metrics.max_error function.

More details here : Max error

transparentai.models.evaluation.regression.mean_absolute_error(y_true, y_pred, **args)[source]

Mean absolute error based on the sklearn.metrics.mean_absolute_error function.

More details here : Mean absolute error

transparentai.models.evaluation.regression.mean_gamma_deviance(y_true, y_pred, **args)[source]

Mean Gamma deviance based on the sklearn.metrics.mean_gamma_deviance function.

More details here : Mean Poisson, Gamma, and Tweedie deviances

transparentai.models.evaluation.regression.mean_poisson_deviance(y_true, y_pred, **args)[source]

Mean Poisson deviances based on the sklearn.metrics.mean_poisson_deviance function.

More details here : Mean Poisson, Gamma, and Tweedie deviances

transparentai.models.evaluation.regression.mean_squared_error(y_true, y_pred, **args)[source]

Mean squared error based on the sklearn.metrics.mean_squared_error function.

More details here : Mean squared error

transparentai.models.evaluation.regression.mean_squared_log_error(y_true, y_pred, **args)[source]

Mean squared logarithmic error based on the sklearn.metrics.mean_squared_log_error function.

More details here : Mean squared logarithmic error

transparentai.models.evaluation.regression.median_absolute_error(y_true, y_pred, **args)[source]

Median absolute error based on the sklearn.metrics.median_absolute_error function.

More details here : Median absolute error

transparentai.models.evaluation.regression.r2(y_true, y_pred, **args)[source]

R² score, the coefficient of determination based on the sklearn.metrics.r2_score function.

More details here : R² score, the coefficient of determination

transparentai.models.evaluation.regression.root_mean_squared_error(y_true, y_pred, **args)[source]

Root mean squared error based on the sklearn.metrics.mean_squared_error function.

squared argument is set to False.

More details here : Mean squared error

Model Explainer

class transparentai.models.explainers.ModelExplainer(model, X, model_type=None, feature_names=None, multi_label=False)[source]

Class that allows to understand a local prediction or model behavior using shap package. For the moment this class will work only for model that TreeExplainer, LinearExplainer or KernelExplainer of shap package can handle.

Example

For binary classification (adult dataset):

>>> from transparentai.datasets import load_adult
>>> from sklearn.ensemble import RandomForestClassifier

>>> data = load_adult()
>>> X, Y = data.drop(columns='income'), data['income'].replace({'>50K':1, '<=50K':0})
>>> X = X.select_dtypes('number')
>>> clf = RandomForestClassifier().fit(X,Y)

>>> explainer = ModelExplainer(clf, X, model_type='tree')
>>> explainer.explain_global_influence(X, nsamples=1000)
 99%|===================| 1988/2000 [01:01<00:00]
{'age': 0.08232147427281439,
 'fnlwgt': 0.051546309804410356,
 'education-num': 0.07579739409175655,
 'capital-gain': 0.07904473020037411,
 'capital-loss': 0.02746167321242212,
 'hours-per-week': 0.060904331971380544}

>>> explainer.explain_local_influence(X.iloc[0])
{'age = 25': -0.07041555656760465,
 'fnlwgt = 226802': -0.025452222766471095,
 'education-num = 7': -0.07771055672375951,
 'capital-gain = 0': -0.08661166294186842,
 'capital-loss = 0': 0.005169999992358498,
 'hours-per-week = 40': -0.02528000040561892}

For multi label classification (iris dataset):

>>> from transparentai.datasets import load_iris
>>> from sklearn.ensemble import RandomForestClassifier

>>> data = load_iris()
>>> X, Y = data.drop(columns='iris plant'), data['iris plant']
>>> Y = Y.replace({'setosa':0, 'versicolor':1, 'virginica':2})
>>> clf = RandomForestClassifier().fit(X,Y)

>>> explainer = ModelExplainer(clf, X, model_type='tree', multi_label=True)
>>> explainer.explain_global_influence(X)
{0: {'sepal length (cm)': 0.01175688849131886,
  'sepal width (cm)': 0.005942666575467832,
  'petal length (cm)': 0.22338177293802924,
  'petal width (cm)': 0.16601288524931274},
 1: {'sepal length (cm)': 0.02572877729050815,
  'sepal width (cm)': 0.008901222137936085,
  'petal length (cm)': 0.2281212172475334,
  'petal width (cm)': 0.19257521807807496},
 2: {'sepal length (cm)': 0.02847011091114645,
  'sepal width (cm)': 0.011024999958494059,
  'petal length (cm)': 0.23041677331694704,
  'petal width (cm)': 0.20166499567956975}}

>>> explainer.explain_local_influence(X.iloc[0])
{0: {'sepal length (cm) = 5.1': 0.021333332546055316,
  'sepal width (cm) = 3.5': 0.011599999857135118,
  'petal length (cm) = 1.4': 0.42903332408517597,
  'petal width (cm) = 0.2': 0.31883332636673},
 1: {'sepal length (cm) = 5.1': 0.012099999799393118,
  'sepal width (cm) = 3.5': 0.0018000002391636372,
  'petal length (cm) = 1.4': -0.21319999573752285,
  'petal width (cm) = 0.2': -0.15029999669175595},
 2: {'sepal length (cm) = 5.1': -0.03343333344208076,
  'sepal width (cm) = 3.5': -0.013400000038091093,
  'petal length (cm) = 1.4': -0.21583332964917645,
  'petal width (cm) = 0.2': -0.16853333076229318}}

For regression (boston dataset):

>>> from transparentai.datasets import load_boston
>>> from sklearn.linear_model import LinearRegression

>>> data = load_boston()
>>> X, Y = data.drop(columns='MEDV'), data['MEDV']
>>> regr = LinearRegression().fit(X, Y)

>>> explainer = ModelExplainer(regr, X, model_type='linear')
>>> explainer.explain_global_influence(X)
{'CRIM': 0.5167422492788898,
 'ZN': 0.7756203068845728,
 'INDUS': 0.12750516344183324,
 'CHAS': 0.3459732772883547,
 'NOX': 1.7001686711898643,
 'RM': 1.9555806154096416,
 'AGE': 0.017036261147963947,
 'DIS': 2.537086257135257,
 'RAD': 2.3074416123109764,
 'TAX': 1.7711676384532529,
 'PTRATIO': 1.7028349208723197,
 'B': 0.5086851450326836,
 'LSTAT': 2.9991432546436037}

>>> explainer.explain_local_influence(X.iloc[0])
{'CRIM = 0.0063': 0.3896189542190243,
 'ZN = 18.0': 0.308063041889274,
 'INDUS = 2.31': -0.18146644441613213,
 'CHAS = 0.0': -0.18584127208907195,
 'NOX = 0.538': 0.29661462781287745,
 'RM = 6.575': 1.1062538448823005,
 'AGE = 65.2': -0.002336189759535761,
 'DIS = 4.09': -0.4352292308278341,
 'RAD = 1.0': -2.616541593062981,
 'TAX = 296.0': 1.3843997187946957,
 'PTRATIO = 15.3': 3.006425898946704,
 'B = 396.9': 0.37457147693105614,
 'LSTAT = 4.98': 4.026504219585754}
model_type

Type of model to inspect, it can only be ‘tree’, ‘linear’ or None

model

model to inspect

multi_label

Whether there is more than 2 classes in the label column (only for classification)

Type:bool
explainer

explainer object that has expected values and can compute shap values

Type:shap.TreeExplainer, shap.LinearExplainer or shap.KernelExplainer
feature_names

list of feature names (length == length of X columns)

Type:np.array
global_explain

dictionnary with feature names as keys and global feature importance as values

Type:dict
Parameters:
  • model – model to inspect
  • X (array like) – data (possibly training) to start the explainer
  • model_type (str (default None)) – Type of model to inspect, it can only be ‘tree’, ‘linear’ or None
  • feature_names (np.array or list) – list of feature names (length == length of X columns)
  • multi_label (bool) – Whether there is more than 2 classes in the label column (only for classification)
Raises:
  • TypeError: – X must be an array like. Valids dtypes are pandas.DataFrame, pandas.Series, numpy.ndarray and list
  • ValueError: – model_type must be ‘tree’, ‘linear’ or None
  • AttributeError: – model has neither a predict() function or a predict_proba() function
compute_shap_values(X)[source]

Computes the shap values using explainer attribute.

Parameters:X (array like) – A matrix of samples (# samples x # features) on which to explain the model’s output.
Returns:For models with a single output this returns a matrix of SHAP values (# samples x # features). Each row sums to the difference between the model output for that sample and the expected value of the model output (which is stored as expected_value attribute of the explainer). For models with vector outputs this returns a list of such matrices, one for each output.
Return type:np.ndarray
explain_global_influence(X, nsamples=None)[source]

Global explaination for a model based on a sample X If there are a lot of data this function could last a while.

Parameters:
  • X (array like) – Data to explain
  • nsamples (None or int or float (default None)) – If not None reduce the data to a sample of nsamples else if <= 1. reduce to len(df) * nsamples
Returns:

dictionnary with feature names as keys and feature importance as values
Return type:

dict
Raises:
  • TypeError: – X must be an array like. Valids dtypes are pandas.DataFrame, pandas.Series, numpy.ndarray and list
  • ValueError: – X must have the same number of feature than the X used in the class initialization
explain_local_influence(X, feature_classes=None)[source]

Explain a local prediction : only one row required.

Parameters:
  • X (array like) – Local row to explain
  • feature_classes (dict) – This dictionnary provides new values for categorical feature so that the feature can be more interpretable. dictionnary with features names as keys and for value a dictionnary with key, value pair representing current value and value to display.
Returns:

dictionnary with feature names as keys and feature importance as values
Return type:

dict
Raises:
  • TypeError: – X must be an array like. Valids dtypes are pandas.DataFrame, pandas.Series, numpy.ndarray and list
  • ValueError: – X must be one row
  • ValueError: – X must have the same number of feature than the X used in the class initialization
format_feature_importance(feat_importance, top=None)[source]

Format feature importance with a top value so that it returns only the features that have the biggest influence

Parameters:
  • feat_importance (pd.Series or dict) – current feature importance
  • top (int) – number of value to get
Returns:

Feature importance formated
Return type:

pd.Series
init_explainer(X)[source]

Initialize the explainer.

If model_type is None then use shap.KernelExplainer class.

Else if it’s ‘tree’ then shap.TreeExplainer.

Else use shap.LinearExplainer.

Parameters:X (array like) – data (possibly training) to start the explainer
Returns:explainer initialized
Return type:shap.KernelExplainer, shap.TreeExplainer, shap.LinearExplainer
plot_global_explain(X=None, nsamples=None, top=None, color='#3498db', **kwargs)[source]

Display a plot for model global explanation based on a sample X.

Parameters:
  • X (pd.DataFrame or np.array) – Data to explain
  • nsamples (None or int or float (default None)) – If not None reduce the data to a sample of nsamples else if <= 1. reduce to len(df) * nsamples
  • top (int) – top n feature to display (in case there are too much features)
  • str (default '#3498db') (color) – Color of the bar plot
Raises:

AttributeError: – If X parameter is None then you have to add X in explain_global function first or directly in this function if you prefer to plot directly.
plot_local_explain(X, feature_classes=None, top=None, num_class=None, **kwargs)[source]

Display a plot for a local prediction based on X set.

Parameters:
  • X (array like) – Local row to explain
  • feature_classes (dict) – This dictionnary provides new values for categorical feature so that the feature can be more interpretable. dictionnary with features names as keys and for value a dictionnary with key, value pair representing current value and value to display.
  • num_class (int (default None)) – Class number for which we want to see the explanation if it’s a binary classification then the value is 1 if None and it’s a multi_label classifier then plots for each class
plot_local_explain_interact(X, feature_classes=None, visible_feat=None, top=None, num_class=None, **kwargs)[source]

Display a plot for a local prediction based on X set.

Parameters:
  • X (array like) – Local row to explain
  • feature_classes (dict) – This dictionnary provides new values for categorical feature so that the feature can be more interpretable. dictionnary with features names as keys and for value a dictionnary with key, value pair representing current value and value to display.
  • visible_feat (list (default None)) – List of feature to interact with
  • num_class (int (default None)) – Class number for which we want to see the explanation if it’s a binary classification then the value is 1 if None and it’s a multi_label classifier then plots for each class