BSAN 750 Lab 4: Logistic Regression for Credit-Card Default

Foundations of Machine Learning and AI

A bank wants to know which cardholders are likely to miss next month’s payment. What should it do with that prediction?

In this lab we build a logistic regression model on 30,000 credit-card clients. We go from raw data to a business decision rule: predicted probabilities, then classifications, then a cutoff chosen by the cost of each type of mistake.

Learning objectives

  1. Explain why a binary response needs a probability model.
  2. Fit a logistic regression and interpret coefficients as log odds and odds ratios.
  3. Convert predicted probabilities into classes with a cutoff, and read a confusion matrix.
  4. Evaluate ranking quality with ROC/AUC and precision–recall.
  5. Choose a cutoff that reflects business costs, using validation data.
  6. Report honest performance on an untouched test set.

1. Set up Python

We use pandas for tables, statsmodels to fit and summarize the logistic model, and scikit-learn for data splitting and classification metrics.

If a package is missing, run this once in a separate cell:

%pip install numpy pandas matplotlib scipy statsmodels scikit-learn
# RUN: Import the tools used in this lab.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import statsmodels.api as sm
import statsmodels.formula.api as smf
from scipy.stats import chi2_contingency
from sklearn.model_selection import train_test_split, StratifiedKFold
from sklearn.metrics import (
    confusion_matrix, ConfusionMatrixDisplay, roc_curve, roc_auc_score,
    precision_recall_curve, average_precision_score, precision_score,
    recall_score, f1_score,
)
from IPython.display import display

pd.set_option("display.precision", 3)
plt.rcParams.update({"figure.dpi": 110, "font.size": 11})
print("Setup complete.")

2. Load and understand the data

Before coding: What does one row represent? What would a bank lose from each kind of prediction error?

The data describe 30,000 credit-card clients of a bank in Taiwan in 2005. Amounts are in New Taiwan (NT) dollars. The response records whether the client defaulted on the next month’s payment.

Source: Yeh (2009), Default of Credit Card Clients, UCI Machine Learning Repository, CC BY 4.0.

This is a teaching example, not a lending decision system.

# RUN: Read the data and give the response a short name.
url = "https://www.dropbox.com/s/tnoo06n8m842uit/credit_card_default.csv?dl=1"
credit = pd.read_csv(url)
credit = credit.rename(columns={"default payment next month": "default"})

print("Rows and columns:", credit.shape)
print("Missing entries:", credit.isna().sum().sum())
display(credit.head())

How common is default?

Before coding: Suppose a model predicts “no default” for every client. What is its accuracy? Would the bank find it useful?

# RUN: Class balance of the response.
display(credit["default"].value_counts().rename("Clients").to_frame()
        .assign(Share=lambda t: t["Clients"] / t["Clients"].sum()))

When one class is rare, a model can reach high accuracy by always predicting the common class. It would still miss every default. For imbalanced problems, look at which errors a model makes, not only how many.

Variables used in this lab

In this lab we use a subset of predictors.

Variable Meaning
default Defaulted next month: 1 = yes, 0 = no (the response)
LIMIT_BAL Credit limit (NT dollars)
SEX 1 = male, 2 = female
EDUCATION 1 = graduate school, 2 = university, 3 = high school, 4 = other
MARRIAGE 1 = married, 2 = single, 3 = other
PAY_0 Most recent repayment status: −1 = paid on time; 1–9 = months of payment delay
BILL_AMT1 Most recent bill statement (NT dollars)
PAY_AMT1 Most recent payment made (NT dollars) 
credit1 = credit.copy()[["default", "LIMIT_BAL", "SEX", "EDUCATION", "MARRIAGE", "PAY_0", "BILL_AMT1", "PAY_AMT1"]]

Are variables in the right type?

for column in ["SEX", "EDUCATION", "MARRIAGE", "PAY_0"]:
    print(f"Data type of {column}: {credit1[column].dtype}")
    display(credit1[column].value_counts().sort_index().rename(f"{column} count").to_frame().T)

3. Explore relationships with the response

Before coding: What is the null hypothesis of a chi-square test of independence? If the test rejects, does education cause default?

A two-way table (a simple pivot table) counts clients in each combination of two categorical variables. Row-wise default rates are often easier to read than raw counts.

In a real project, explore only the training data so the test set stays untouched. Here we look at the full data to learn the tools.

# RUN: Counts, default rates by education, and a chi-square test.
education_counts = pd.crosstab(credit1["EDUCATION"], credit1["default"])
display(education_counts)

education_rates = credit1.groupby("EDUCATION")["default"].mean()

fig, ax = plt.subplots(figsize=(6, 3.5))
education_rates.plot.bar(ax=ax, rot=0, color="blue")
ax.set(xlabel="Education (1 = graduate, 2 = university, 3 = high school, 4 = other)",
       ylabel="Default rate", title="Default rate by education")
plt.tight_layout()
plt.show()
# Chi-square test
chi2, p_value, dof, expected = chi2_contingency(education_counts)
print(f"Chi-square = {chi2:.2f}; df = {dof}; p-value = {p_value:.3g}")
print(f"Smallest expected count = {expected.min():.1f}")

Your turn

  1. State your conclusion from the chi-square test in plain words.
  2. Build the same table of default rates for MARRIAGE. Which variable looks more useful for prediction, EDUCATION or MARRIAGE? Why?
# PRACTICE: Default rate by MARRIAGEs.
# Hint: data.groupby("MARRIAGE", observed=True)["default"].mean()

# YOUR CODE HERE

4. Split the data: three samples, three jobs

Before coding: We will choose a cutoff later. Which sample should we use to choose it? What goes wrong if we choose it using the test sample?

Sample Share Job
Training 60% Estimate the coefficients
Validation 20% Choose the cutoff (a decision rule)
Test 20% Report final performance, once 
# RUN: Reserve 20% for testing, then divide the remaining 80% into 60% / 20%.
development, test = train_test_split(
    credit1, test_size=0.20, random_state=750)
train, valid = train_test_split(
    development, test_size=0.25, random_state=751)

display(pd.DataFrame({
    "Sample": ["Training", "Validation", "Test"],
    "Rows": [len(train), len(valid), len(test)],
    "Default rate": [train["default"].mean(), valid["default"].mean(), test["default"].mean()],
}).set_index("Sample"))

5. Fit and interpret logistic regression

Logistic regression models the log odds of default as a linear function of the predictors:

\[\log\left(\frac{p}{1-p}\right)=\beta_0+\beta_1x_1+\cdots+\beta_kx_k=\eta.\]

Solving for the probability gives the S-shaped logistic curve, which always stays between 0 and 1:

\[p=\frac{1}{1+e^{-\eta}}.\]

In statsmodels, smf.glm(..., family=sm.families.Binomial()) fits this model. The binomial family uses the logit link by default. C(...) turns a variable into indicator (dummy) variables.

# RUN: Fit the model on the training sample only.
formula = (
    "default ~ LIMIT_BAL + BILL_AMT1 + PAY_AMT1 + PAY_0"
    " + C(SEX) + C(EDUCATION) + C(MARRIAGE)"
)
model = smf.glm(formula, data=train, family=sm.families.Binomial()).fit()
print(model.summary())

How to interpret the coefficients \(\beta\)?

A coefficient \(\beta\) is the change in log odds for a one-unit increase in that predictor, holding the others fixed. Exponentiating gives an odds ratio (why?):

  • \(e^{\beta} > 1\): the odds of default are multiplied up.
  • \(e^{\beta} < 1\): the odds of default are multiplied down.
  • \(e^{\beta} = 1\): no association.

Thus, \((e^{\beta}-1)*100\%\) can be interpreted as the % change of odds.

For a categorical predictor, the odds ratio compares a category with the reference category.

# RUN: Exponentiate coefficients and their confidence limits.
ci = model.conf_int()
odds_ratios = pd.DataFrame({
    "Coefficient": model.params,
    "Odds ratio": np.exp(model.params),
    "OR lower 95%": np.exp(ci[0]),
    "OR upper 95%": np.exp(ci[1]),
    "Odds (% change)": round(100 * (np.exp(model.params) - 1), 2),
    "p-value": model.pvalues,
})
display(odds_ratios)

Your turn

  1. Interpret the coefficient for PAY_AMT1 in one sentence.
  2. Interpret the coefficient for EDUCATION(3) in one sentence.
  3. Which predictors appear to matter most? Is a small p-value the same as a large effect?
  4. The model includes SEX and MARRIAGE. In many jurisdictions (for example, under the U.S. Equal Credit Opportunity Act), lenders may not use sex or marital status in credit decisions. What would you do with these predictors if this model were used for real decisions? Does removing them guarantee the model is fair?

Model fit

All three measures below are built from the log-likelihood of the fitted model:

\[\ell(\hat\beta)=\sum_{i=1}^{n}\Big[y_i\log\hat p_i+(1-y_i)\log(1-\hat p_i)\Big].\]

It is never positive. The closer it is to 0, the better the predicted probabilities match the observed 0/1 outcomes.

  • Deviance measures lack of fit (smaller is better). It plays a role like SSE in linear regression. For a 0/1 response: \[D=-2\,\ell(\hat\beta).\]
  • AIC balances fit against the number of coefficients \(k\) (including the intercept). Smaller is better: \[AIC=-2\,\ell(\hat\beta)+2p.\]
  • There is no ordinary \(R^2\) for logistic regression. But there is a pseudo-\(R^2\) (McFadden’s), defined as \[R^2_{\text{McF}}=1-\frac{\ell(\hat\beta)}{\ell_0},\] where \(\ell_0\) is the log-likelihood of the intercept-only model, which predicts the overall default rate for everyone. It measures how much the predictors improve on that baseline. Values are usually much lower than OLS \(R^2\). Values of 0.2–0.4 are often considered a good fit.

In Python: model.llf is \(\ell(\hat\beta)\), model.llnull is \(\ell_0\), and model.deviance and model.aic give \(D\) and AIC. Try 1 - model.llf / model.llnull to get the pseudo-\(R^2\).

None of these tell us how the model does on new clients. That is what the validation and test samples are for.

print(f"Deviance: {model.deviance:.2f}")
print(f"AIC: {model.aic:.2f}")
print(f"pseudo Rsq: {1 - model.llf / model.llnull:.2f}")

7. From probabilities to classifications

Before coding: The model gives each client a probability. What extra decision do we need before we can say “this client will default”?

model.predict(new_data) returns predicted probabilities of default. First, look at how those probabilities separate the two groups.

# RUN: Predicted probabilities on the validation sample.
valid_y = valid["default"]
valid_prob = model.predict(valid)

fig, ax = plt.subplots(figsize=(7, 3.8))
bins = np.linspace(0, 1, 41)
ax.hist(valid_prob[valid_y == 0], bins=bins, alpha=0.6, color="green", label="Did not default (0)")
ax.hist(valid_prob[valid_y == 1], bins=bins, alpha=0.6, color="red", label="Defaulted (1)")
ax.set(xlabel="Predicted probability of default", ylabel="Clients",
       title="Validation: predicted probabilities by actual outcome")
ax.legend()
plt.tight_layout()
plt.show()

Questions: Do the two histograms overlap? If you drew a vertical line at 0.5, which clients would fall on the “wrong” side? What happens if you slide that line left?

The confusion matrix

A cutoff (pcut) turns probabilities into classes: predict default when \(\hat p \ge pcut\).

Metric Formula Question it answers
Recall (TPR, sensitivity) TP/(TP+FN) Of the clients who defaulted, what share did we catch?
False negative rate (FNR) FN/(TP+FN) Of the defaulters, what share did we miss?
False positive rate (FPR) FP/(FP+TN) Of the good clients, what share did we wrongly flag?
Precision TP/(TP+FP) Of the clients we flagged, what share really defaulted?
F1 2PR/(P+R) A single balance of precision (P) and recall (R)
Misclassification rate (MR) (FP+FN)/n What share of all predictions were wrong?

The helper below computes all useful metrics for a given cutoff. We reuse it for the rest of the lab.

# RUN: A reusable helper for classification metrics at one cutoff.
def classification_summary(y, prob, cutoff, w_fp=1, w_fn=5):
    predicted = (prob >= cutoff).astype(int)
    tp, fn, fp, tn = confusion_matrix(y, predicted, labels=[1, 0]).ravel()
    return {
        "Cutoff": cutoff, "TP": tp, "FN": fn, "FP": fp, "TN": tn,
        "Recall": tp / (tp + fn),
        "FNR": fn / (tp + fn),
        "FPR": fp / (fp + tn),
        "Precision": precision_score(y, predicted, zero_division=0),
        "F1": f1_score(y, predicted, zero_division=0),
        "MR": (fp + fn) / len(y),
        "Cost per client": (w_fp * fp + w_fn * fn) / len(y),
    }
# get confusion matrix and classification metrics for pcut=0.5
#  on the validation sample
cutoff = 0.5
valid_class = (valid_prob >= cutoff).astype(int)
ConfusionMatrixDisplay.from_predictions(
    valid_y, valid_class, labels=[1, 0], cmap="Blues", colorbar=False)
plt.title("Validation confusion matrix: cutoff 0.50")
plt.tight_layout()
plt.show()

display(pd.DataFrame([classification_summary(valid_y, valid_prob, cutoff)]))

Your turn

  1. At cutoff 0.5, what share of defaulters does the model catch? Compare its MR with the “predict nobody defaults” rule from Section 2.
  2. Predict first: if the cutoff drops to 0.2, what happens to recall, FPR, and precision? Then check with the cell below.
  3. A bank could use this model to flag risky accounts for review. Which metric would the review team care about? Which would a risk manager care about?
# PRACTICE: Plot the validation confusion matrix at cutoff 0.2 and summarize its metrics.
# YOUR CODE HERE

8. ROC and precision–recall: judging the scores before choosing a cutoff

Before coding: Why must these functions receive predicted probabilities rather than predicted 0/1 classes?

Each cutoff gives one confusion matrix. Sweep the cutoff from 1 down to 0 and plot the results:

  • ROC curve: TPR (recall) against FPR. The diagonal is random guessing.
  • Precision–recall curve: precision against recall. The baseline is the default rate, since a random flag is correct at that rate.

Key idea — AUC measures ranking, not classification. ROC AUC is the probability that a randomly chosen defaulter gets a higher predicted probability than a randomly chosen non-defaulter. It does not depend on any single cutoff. An AUC of 0.5 is random ranking; 1.0 is perfect.

For the precision–recall curve we report average precision (AP), a weighted average of precision across recall levels. AP is more informative than ROC AUC when positives are rare.

# RUN: ROC and precision–recall curves on the validation sample.
fpr, tpr, _ = roc_curve(valid_y, valid_prob)
precision, recall, _ = precision_recall_curve(valid_y, valid_prob)
valid_auc = roc_auc_score(valid_y, valid_prob)
valid_ap = average_precision_score(valid_y, valid_prob)

fig, axes = plt.subplots(1, 2, figsize=(10, 4))
axes[0].plot(fpr, tpr, label=f"ROC AUC = {valid_auc:.3f}")
axes[0].plot([0, 1], [0, 1], "--", color="gray", label="Random ranking")
axes[0].set(xlabel="False positive rate", ylabel="True positive rate (recall)",
            title="Validation ROC")
axes[1].step(recall, precision, where="post", label=f"AP = {valid_ap:.3f}")
axes[1].axhline(valid_y.mean(), ls="--", color="gray", label="Default rate (baseline)")
axes[1].set(xlabel="Recall", ylabel="Precision", title="Validation precision–recall")
for ax in axes:
    ax.set(xlim=(0, 1), ylim=(0, 1.02))
    ax.legend(loc="lower left", fontsize=9)
plt.tight_layout()
plt.show()

Your turn

  1. Interpret the validation AUC in one sentence, using the “random defaulter vs. random non-defaulter” idea.
  2. Where on the ROC curve is cutoff 0.5? Where is cutoff 0.2? (Use your metrics from Section 7.)
  3. Changing the cutoff moves us along the curve. What would it take to move the whole curve up?

9. Choose a cutoff that reflects business costs

Before coding: Missing a defaulter usually costs a bank more than reviewing a good client. Should the cutoff go up or down from 0.5?

Assume, for illustration, that a false positive costs 1 unit and a false negative costs 5 units. These weights are teaching assumptions. In practice they come from the business.

\[\text{Cost per client}=\frac{1\cdot FP+5\cdot FN}{n}.\]

We search a grid of cutoffs on the validation sample and keep the one with the lowest cost. We also record the cutoffs that maximize F1 and minimize MR, to show that “best” depends on the goal.

# RUN: Grid search over cutoffs on validation data. The test sample is not used.
w_fp, w_fn = 1, 5
grid = np.linspace(0, 1, 101)
cutoff_results = pd.DataFrame(
    [classification_summary(valid_y, valid_prob, c, w_fp, w_fn) for c in grid])

# idxmin / idxmax return the first (smallest) cutoff when there is a tie.
best_cutoff = cutoff_results.loc[cutoff_results["Cost per client"].idxmin(), "Cutoff"]
best_f1_cutoff = cutoff_results.loc[cutoff_results["F1"].idxmax(), "Cutoff"]
best_mr_cutoff = cutoff_results.loc[cutoff_results["MR"].idxmin(), "Cutoff"]
print(f"Cost-ratio rule: {w_fp / (w_fp + w_fn):.3f}")
print(f"Grid-search minimum-cost cutoff: {best_cutoff:.2f}")

fig, axes = plt.subplots(1, 2, figsize=(10, 3.8))
axes[0].plot(cutoff_results["Cutoff"], cutoff_results["Cost per client"], color="#2563EB")
axes[0].axvline(best_cutoff, color="#C8102E", ls="--", label=f"Minimum cost: {best_cutoff:.2f}")
axes[0].set(xlabel="Cutoff", ylabel="Cost per client", title="Validation cost (FP = 1, FN = 5)")
axes[1].plot(cutoff_results["Cutoff"], cutoff_results["MR"], label="MR")
axes[1].plot(cutoff_results["Cutoff"], cutoff_results["F1"], label="F1")
axes[1].set(xlabel="Cutoff", ylabel="Value", title="Validation MR and F1")
for ax in axes:
    ax.legend()
plt.tight_layout()
plt.show()
# RUN: Compare the three objectives side by side on validation data.
objective_table = pd.DataFrame([
    {"Objective": "Minimum cost (FP = 1, FN = 5)", **classification_summary(valid_y, valid_prob, best_cutoff)},
    {"Objective": "Maximum F1", **classification_summary(valid_y, valid_prob, best_f1_cutoff)},
    {"Objective": "Minimum MR", **classification_summary(valid_y, valid_prob, best_mr_cutoff)},
]).set_index("Objective")
display(objective_table)

Your turn

  1. Why does minimizing MR tend to pick a higher cutoff than minimizing cost? What does MR assume about the two error types?
  2. By construction, the minimum-cost rule cannot have a lower MR than the minimum-MR rule. Does that make it a worse decision rule? Explain.
  3. How will the cutoff, MR, and cost change if shifting the false-negative cost to 10. Rerun the search to check.

Decide your cost weights and decision rule before you look at the test results.

# PRACTICE: Repeat the validation search with w_fn = 10.
# Hint: classification_summary(valid_y, valid_prob, c, w_fp=1, w_fn=10) for each c in grid.

# YOUR CODE HERE

10. Final assessment on the test sample

Before coding: Why must we freeze the model and the cutoff before looking at the test sample?

The model was fit on training data. The cutoff was chosen on validation data. The test sample now plays the role of new clients. We compare three rules:

  • the validation-selected minimum-cost cutoff,
  • cutoff 0.5 as a fixed reference, and
  • predicting that nobody defaults (shown as cutoff = inf: nobody is ever flagged).

Do not pick a new rule based on this table.

# RUN: Test data enter the analysis only now.
test_y = test["default"]
test_prob = model.predict(test)

test_comparison = pd.DataFrame([
    {"Rule": "Validation minimum cost", **classification_summary(test_y, test_prob, best_cutoff, w_fp, w_fn)},
    {"Rule": "Fixed cutoff 0.50", **classification_summary(test_y, test_prob, 0.5, w_fp, w_fn)},
    {"Rule": "Predict nobody defaults", **classification_summary(test_y, test_prob, np.inf, w_fp, w_fn)},
]).set_index("Rule")
display(test_comparison)
print(f"Test ROC AUC: {roc_auc_score(test_y, test_prob):.3f}")
print(f"Test average precision: {average_precision_score(test_y, test_prob):.3f}")

ConfusionMatrixDisplay.from_predictions(
    test_y, (test_prob >= best_cutoff).astype(int), labels=[1, 0],
    cmap="Blues", colorbar=False)
plt.title(f"Test confusion matrix: cutoff {best_cutoff:.2f}")
plt.tight_layout()
plt.show()

Your turn

  1. Why does changing the cutoff leave ROC AUC unchanged?
  2. Does the lowest-cost rule also have the highest accuracy? What does this tell you about using accuracy to judge a credit model?
  3. Plot the test ROC curve and report its AUC. This is reporting, not another chance to tune.
  4. If you have a different model, what sample should you use for the final comparison?
  5. Write a 4–6 sentence summary for a bank manager. Cover how well the model ranks clients, the tradeoff between missed defaults and false alarms, and the cost assumption behind your cutoff.
# PRACTICE: Plot the test ROC curve and report its AUC.
# Hint: follow the ROC code in Section 8 with test_y and test_prob.

# YOUR CODE HERE

Lab 4 checklist

Restart the kernel and run the notebook from top to bottom. Then make sure you can explain:

  1. Why accuracy can mislead when one class is rare.
  2. Why coded variables such as EDUCATION and PAY_STATUS enter as categories, and what the reference category is.
  3. How to read a coefficient as a change in log odds and as an odds ratio, and why an odds ratio is not a change in probability.
  4. What a cutoff does, and how recall, FPR, precision, and MR change as it moves.
  5. What ROC AUC measures, and why it does not depend on the cutoff.
  6. How error costs determine a sensible cutoff, and why it is chosen on validation data.
  7. Why the test sample is used once, at the end.

Python reference

Task Python
Two-way table pd.crosstab(df["x"], df["y"])
Chi-square test chi2_contingency(table)
Stratified split train_test_split(df, test_size=..., stratify=df["y"])
Fit logistic regression smf.glm(formula, data=df, family=sm.families.Binomial()).fit()
Categorical predictor C(x) in the formula
Coefficients and odds ratios model.params, np.exp(model.params)
Predicted probabilities model.predict(new_df)
Confusion matrix (class 1 first) confusion_matrix(y, predicted, labels=[1, 0])
ROC curve and AUC roc_curve(y, prob), roc_auc_score(y, prob)
Precision–recall and AP precision_recall_curve(y, prob), average_precision_score(y, prob)
Stratified k-fold StratifiedKFold(n_splits=5, shuffle=True)

References: statsmodels GLM, scikit-learn classification metrics.