9  Perturbation Stability: Comparing LIME and SHAP for Logistic Regression and BERT

9.1 Objectice

Synthesize findings from applying LIME and SHAP to explain two models: - TF-IDF + Logistic Regression - Fine-tuned BERT

9.1.1 Evaluation Criteria:

• Interpretability: Are the word-level explanations intuitive?
  
• Stability: Do explanations remain consistent after perturbations? (Jaccard similarity)
  
• Visualization Effectiveness: Are the insights clearly communicated?

9.1.2 Logistic Regression + LIME

• Top Words: Literal, generic terms like "SpeakerPelosi", "Montage", "kid", "AWESOME".
• Interpretability: ✅ High — linear model yields transparent explanations.
• Stability: Jaccard = 0.662
• Insights:
  • Words map clearly to model predictions.
  • Explanation quality is stable enough for reliable human inspection.

9.1.3 Logistic Regression + SHAP

• Top Words: Similar to LIME — lexical, interpretable tokens with strong polarity or topic focus.
• Interpretability: ✅✅ Very High — SHAP’s additive structure mirrors linear coefficients.
• Stability: Jaccard = 0.910
• Insights:
  • Most robust and faithful explanations among all configurations.
  • Ideal for explainable AI deployments in real-world settings.

9.1.4 BERT + LIME

• Top Words: Emotionally or contextually rich words (e.g., "inadequate", "LOVE", "smartrobotics").
• Interpretability: ⚠️ Moderate — sometimes insightful, but inconsistent.
• Stability: Jaccard = 0.475
• Insights:
  • BERT’s contextual embeddings are difficult to fully capture with LIME’s local linear approximations.
  • Visuals (bar plots, word clouds) help, but the sensitivity to input changes undermines reliability.

9.1.5 BERT + SHAP

• Top Words: Named entities, sentiment tokens, and syntactic markers (e.g., "Ranked", "RAVENS", "sad").
• Interpretability: ✅ High — SHAP can extract meaningful attributions from a complex black-box.
• Stability: Jaccard = 0.885
• Insights:
  • Despite BERT’s complexity, SHAP shows strong robustness.
  • Offers a balance between deep model capacity and explanation reliability.

9.2 Comparative Table

Method	Model	Interpretability	Jaccard Similarity	Explanation Stability
LIME	Logistic Regression	High	0.662	Medium-High
LIME	BERT	Moderate	0.475	Low
SHAP	Logistic Regression	Very High	0.910	Excellent
SHAP	BERT	High	0.885	Very Good

9.3 Takeaways

• SHAP + Logistic Regression provides the best trade-off between transparency and robustness — recommended for trustworthy NLP systems.
• LIME + BERT produces intuitive but unstable explanations — useful for local debugging, but not for auditing.
• SHAP outperforms LIME on both models in stability.
• Even for complex models like BERT, SHAP maintains explanation consistency, making it suitable for high-stakes applications.

Recommendation: For interpretable and dependable NLP systems, favor SHAP with simpler models or combine SHAP+LIME to gain both global and local insights.