Collaborative CIS 5190 final project · University of Pennsylvania

Headline-Only News Source Classifier

A collaborative applied machine-learning project testing whether publisher identity can be inferred from headline-like URL slug text alone. The public repository documents URL collection, preprocessing into pseudo-headlines, TF-IDF/logistic/SVC experiments, RoBERTa-base fine-tuning, and accuracy evaluations without using domain names, article bodies, or explicit source metadata as model input.

PythonPyTorchRoBERTaNLPText Classificationscikit-learn
GitHub RepositoryAll Projects

Headline-only source classification

The model sees cleaned headline-like text, not domains, article bodies, or source metadata.

01
URL slugs
FOX / NBC article paths
02
Pseudo-headlines
domain and metadata removed
03
RoBERTa-base
binary sequence classifier
04
Evaluation
85.34% hidden-test accuracy
14,205 self-collected examples
PyTorch + Transformers
URL-slug preprocessing

Overview

Project Strengths

  • Constrained NLP problem: classify FOX vs NBC using only cleaned headline-like text derived from URL slugs.
  • Self-collected URL-label dataset with 14,205 examples: 5,394 FOX and 8,811 NBC.
  • Pipeline covers URL normalization, pseudo-headline extraction, label normalization, model training, inference compatibility, and accuracy evaluation.
  • Modeling work compares classical TF-IDF approaches with a final RoBERTa-base sequence classifier implemented with PyTorch and Transformers.

My Contribution

  • Collaborative CIS 5190 Applied Machine Learning final project by Yunzhe Deng and Fusang Leng.
  • Verified attribution from the final report lists Yunzhe Deng for model training and project report writing, and Fusang Leng for web scraping and model training.
  • The site describes the work as collaborative and does not imply sole ownership of the project.

System / Architecture

Architecture

  1. 01Research question: can the source of a news headline be predicted using only headline-like text, without domains, article bodies, or source metadata?
  2. 02Dataset: the final self-collected corpus contains 14,205 URL-label examples, split into 12,785 training examples and 1,420 validation examples.
  3. 03Preprocessing: `prepare_data()` extracts the final URL path segment, removes source-identifying URL context, cleans it into lowercase pseudo-headline text, and normalizes labels to `fox` or `nbc`.
  4. 04Modeling: earlier TF-IDF/logistic and Linear SVC experiments are retained for comparison; the final model uses `roberta-base` with a binary sequence-classification head.
  5. 05Evaluation: accuracy is reported on a self-collected validation split, an instructor-provided evaluation set, and a hidden leaderboard/test set.

Implementation

Implementation

  • Fine-tuned RoBERTa-base with max sequence length 128, AdamW, cross-entropy loss with inverse-frequency class weights, batch size 8, gradient accumulation 4, learning rate 2e-5, and two epochs.
  • Kept Project B-compatible top-level `model.py` and `preprocess.py` wrappers so course evaluators can call `Model.predict()` and `prepare_data()`.
  • Implemented explicit `fox`/`nbc` label mappings to avoid label-index mismatches during checkpoint loading and inference.
  • Retained scripts for URL-source merging, RoBERTa training, and evaluation while excluding the original trained checkpoint from the public portfolio repository.

Headline-only source classification

The model sees cleaned headline-like text, not domains, article bodies, or source metadata.

01
URL slugs
FOX / NBC article paths
02
Pseudo-headlines
domain and metadata removed
03
RoBERTa-base
binary sequence classifier
04
Evaluation
85.34% hidden-test accuracy
14,205 self-collected examples
PyTorch + Transformers
URL-slug preprocessing

Results / Validation

  • Course TF-IDF/logistic regression baseline: 66.49% accuracy.
  • Self-collected validation split: 93.24% accuracy at epoch 2.
  • Instructor-provided evaluation set: 73.2457% accuracy on 3,805 examples with 17.229 ms average inference time.
  • Hidden leaderboard/test set: 85.34% accuracy for submission alias `46 test2`, with 6.47 ms average inference time.

Challenges / Decisions

  • Random validation splits can overstate generalization when repeated URL templates, near-duplicates, timing effects, or collection artifacts appear in both train and validation data.
  • Some retained URLs are not clean article pages, including live-blog, video, watch, and person paths, which can produce weak pseudo-headlines.
  • URL-slug text is a proxy for headlines and may differ from the displayed article headline.
  • Final RoBERTa precision, recall, F1, and leaderboard rank were not verified in the available source materials.
  • The repository license notice states that viewing the portfolio repository does not grant permission to copy, modify, redistribute, or reuse the materials.