← Back to Course

Assignment 1: Text Processing & Tokenization

First Name *

Last Name *

Student ID *

QU Email *

Team Size

Upload all four required files below. Each file must follow the exact naming format specified.

Simple Tokenizer * simple_tokenizer.py

↑

Drop your file here or click to browse

simple_tokenizer.py • Max 5MB

✓

Phase 1: Word-level tokenizer implementation (25 points)

Regex Tokenizer * regex_tokenizer.py

↑

Drop your file here or click to browse

regex_tokenizer.py • Max 5MB

✓

Phase 2: Pattern-based tokenizer implementation (25 points)

BPE Tokenizer * bpe_tokenizer.py

↑

Drop your file here or click to browse

bpe_tokenizer.py • Max 5MB

✓

Phase 3: Byte-Pair Encoding implementation (25 points)

Performance Analyzer * performance_analyzer.py

↑

Drop your file here or click to browse

performance_analyzer.py • Max 5MB

✓

Phase 4: Performance analysis and comparison (25 points)

File Naming Requirements: