All articles

Engineering Case Study: Turning Raw College PDFs into a Full-Scale PWA Ecosystem with Forum & Push Notifications

pwacase-studyfastapireactarchitecturereverse-engineeringparsing

Engineering Case Study: Turning Raw College PDFs into a Full-Scale PWA Ecosystem

In higher education, student user experience often encounters legacy IT infrastructure bottlenecks. At the Higher College of Informatics (VKI NSU), the primary source of operational academic data — the class timetable — had been distributed for years via unstructured, manually formatted multi-page PDF documents.

This case study provides a technical breakdown of developing an autonomous student platform: from solving complex Data Extraction and Reverse Engineering challenges to architecting a reactive PWA client, a scalable FastAPI backend, and an integrated student community hub.

1. Problem Statement & Architectural Context

From a product engineering perspective, the legacy data delivery model presented critical UX friction points:

  • Absence of Machine-Readable APIs: The official institutional portal provided no public endpoints or data feeds for integration.
  • Excessive Interaction Friction: Students were required to download large PDF files daily, manually pan and zoom tabular grids, and search for their group identifiers among hundreds of dense cells.
  • Zero Event Reactivity: Whenever emergency timetable adjustments occurred (instructor substitutions, classroom relocations), students received updates post-factum.

The project objective was clearly defined: build an autonomous, high-throughput Fullstack service that automatically ingests unstructured raw files, transforms them into strictly typed API payloads, and delivers real-time updates across mobile and desktop clients.

2. Data Engineering: Reverse Engineering & Custom PDF Parsing Pipeline

Extracting tabular structures from PDFs is notoriously difficult because the PDF format inherently lacks semantic concepts of rows, columns, or tables — representing solely coordinate-based vector draw commands and text glyphs.

The Breakdown of Legacy Open-Source Solutions

During initial prototyping, third-party implementations were evaluated, including the VkiHub bot by DedMaxTech. However, subsequent college schedule layout changes caused critical pipeline breakdowns:

  1. Unstable Bounding Boxes: Shifting column coordinate baselines led to cross-column data contamination.
  2. Dynamic Merged Cells: Multi-span merged cells for sub-groups, combined lecture streams, and alternating week schedules broke deterministic grid calculations.
  3. Heterogeneous Entity Formatting: Highly irregular instructor naming patterns (Ivanov I.I., Ivanov I. I, Ivanov-Petrov A.), non-standard classroom labels, and auxiliary markers.

Relying on external solutions without direct control over the logic was unsustainable: minor upstream visual adjustments triggered cascade failures throughout the entire data ingestion layer.

Engineering a Custom Core Parser from Scratch

In the re-architected backend (services/parser.py), legacy dependencies were completely superseded by a custom heuristic pipeline:

[ College Portal PDF ] 
       │
       ▼ (Camelot Lattice/Stream Extraction)
[ Raw Tabular Matrix ] 
       │
       ▼ (find_header_row & is_group Regex)
[ Dynamic Header Normalization ] 
       │
       ▼ (Heuristic Cell Parsing & Entity Extraction)
[ Strongly-Typed JSON Payload ] 
       │
       ▼ (Inverted Index Construction)
[ Entity Indexes: Groups / Teachers / Classrooms ]

Key engineering decisions implemented in the core parser:

  • Dynamic Header Resolution: The find_header_row algorithm scans matrix rows using strict regex anchors (^[А-ЯA-Z]\d{4}[а-яa-z]{1,2}\d$), automatically detecting dynamic boundaries between metadata headers and the schedule grid.
  • Heuristic Cell Deconstruction (parse_cell): A granular text parser extracts and normalizes entities:
    • Instructor name isolation and initial standardization: extract_teacher.
    • Room number and campus venue extraction: extract_room.
    • Academic session classification (isLecture, isPractice, isExam, isConsultation, isCanceled).
    • Lexical de-duplication and noise reduction (remove_duplicate_words).
def parse_cell(self, text: str):
    if not text or not text.strip():
        return None

    text = self.clean_text(text)
    teacher = self.extract_teacher(text)
    classroom = self.extract_room(text)
    isLecture = 'лекци' in text.lower() or 'диста' in text.lower()
    lesson_text = self.extract_subject(text)
    lesson = self.normalize_lesson(self.remove_duplicate_words(lesson_text))

    if not lesson:
        return None

    formatted_lesson = (lesson[0].upper() + lesson[1:]) if lesson.lower() == lesson else lesson
    if classroom:
        formatted_lesson = formatted_lesson.replace(classroom, '')

    isIssusing = ((not teacher or not classroom) and (not isLecture or (isLecture and 'нет' in text.lower()))) or 'выставле' in text.lower()

    return {
        'line': self.remove_duplicate_words(text) if isIssusing else text,
        'lesson': formatted_lesson,
        'teacher': teacher,
        'classroom': classroom,
        'isLecture': isLecture,
        'isPractice': 'практик' in text.lower(),
        'isExam': 'экзам' in text.lower(),
        'isIssusing': isIssusing,
        'isConsultation': 'консул' in text.lower(),
        'isCanceled': 'отмен' in text.lower()
    }
  • Inverted Index Construction: Secondary projections are computed on the fly — instructor timetables (/schedule/teachers) and room occupancy (/schedule/classrooms), enabling O(1) room availability lookups for any given time slot.

3. System Architecture & Tech Stack

The architecture follows strict separation of concerns, optimized for minimal Time-to-Interactive (TTI) and low memory footprint.

System Layer Tech Stack Architectural Role
Backend Core Python 3.12, FastAPI, Tortoise ORM Async request handling, scheduled polling workers, transactional consistency
Database PostgreSQL Persistent storage for user profiles, forum threads, push notification subscriptions
Frontend App React 19, TypeScript, Vite Single Page Application, client routing, optimistic UI state updates
PWA & Offline Service Workers, Web App Manifest Background asset caching, home-screen installation, offline runtime
Styling & UI Modular SCSS, Custom Design System Lightweight custom component library avoiding third-party UI framework bloat
Push Gateway Web Push Protocol, VAPID, pywebpush OS-level background notifications for schedule changes

4. Product Expansion: The Student Forum

While initial versions fulfilled basic schedule queries, collaboration with the project UI designer revealed strong community potential: converting daily utility traffic into an active communication hub.

UGC Subsystem Architecture

The forum subsystem was engineered with focus on low time-to-value:

  • Security & Sessions: Session-token authentication, role-based access control, and moderation workflows.
  • Hierarchical Content Engine: Topic categories, tagged posts, nested comment trees, and live reaction states.
  • Mobile-First UX: Responsive post authoring dialogs, optimistic UI transitions on comment submission, and smooth micro-animations.

5. Core Platform Features

Today, the platform serves as a unified digital companion for students and faculty:

  1. Real-Time Classroom Availability: Powered by inverted room indexing, users can find empty classrooms for self-study and project collaboration with a single tap.

  2. Event-Driven Push Notifications: A background scheduling worker periodically polls official sources. When changes for a tracked group or instructor are detected, Web Push alerts are delivered immediately.

  3. Offline-First PWA Capabilities: Full offline functionality during spotty connectivity in basement lecture halls via progressive Service Worker caching.

  4. Integrated Student Community: A centralized space for study group coordination, course discussions, academic resource sharing, and peer support.

6. Engineering Takeaways & Conclusion

Operating the platform highlighted key technical lessons:

  • Source Resilience: Handling official portal outages required robust fallback caching of the last known valid schedule snapshots.
  • Ownership of Core Logic: Eliminating fragile external parser dependencies and engineering custom heuristics brought parsing error rates down to near zero.

Conclusion

This project demonstrated how engineering rigor, architectural autonomy, and a relentless focus on user pain points can transform a clumsy PDF file into a reliable, high-adoption student ecosystem.