Business Context

ShopLens, an e-commerce analytics company, is building a sentiment classifier for product reviews. The team wants to know whether stop-word removal should be part of the preprocessing pipeline, and under what conditions it may reduce model quality.

Data

You are given 420,000 English product reviews collected over 18 months from electronics, home, and beauty categories. Reviews range from 3 to 250 words (median: 38 words). Labels are positive (62%), negative (24%), and neutral (14%). The text contains contractions, negations, emojis, misspellings, and short phrases such as "not worth it", "never again", and "I do recommend it".

Success Criteria

A good solution should clearly explain stop-word removal, implement at least two preprocessing variants, and show when removing stop words helps or hurts performance. Target macro-F1 ≥ 0.84 on the held-out test set, with special attention to errors involving negation and short reviews.

Constraints

• Training must run on a single CPU or one small GPU
• Inference latency should stay below 50 ms per review for the production model
• The solution must compare a sparse baseline and a transformer-based approach

Requirements

1. Define stop-word removal and explain its purpose in NLP pipelines.
2. Build a TF-IDF + Logistic Regression baseline with and without stop-word removal.
3. Fine-tune a lightweight transformer (for example, DistilBERT) and explain why stop-word removal is usually not applied to transformer tokenizers.
4. Identify cases where stop-word removal hurts performance, especially around negation, emphasis, and short texts.
5. Report metrics, compare error patterns, and recommend a production preprocessing policy.
6. Provide modern Python code for preprocessing, training, and evaluation.