IPEDS_StudentToFacultyRatio: Adding provisional data mapping, updated download/preprocess scripts, and golden validation

[smarthg-gi]

Import: IPEDS_StudentToFacultyRatio

Summary of Changes

This PR updates the IPEDS_StudentToFacultyRatio import pipeline to support provisional data releases, handle NCES URL structure changes, add validation configs and golden data verification.

Key changes

1. Download Script (download.py)

• Updated download script with provisional data download logic where revised (_rv) datasets are downloaded first and downloads available provisional dataset if no revised file exists.

2. Preprocessing Logic (preprocess.py)

• Added provisional dataset logic to append measurementMethod : "NCES_ProvisionalEstimate" for provisional files while omitting it for revised releases.

3. Schema & Mapping

• Added mapping for provisional data in the pvmap
• Included measurementMethod in output_columns in metadata.csv.

4. Import Manifest (manifest.json)

• Configured manifest with fortnightly cron schedule and vallidation_config.json

5. Validation & Goldens (validation_config.json & golden_data/)

• Created validation_config.json with deletion percentage threshold checks and golden data verification rules.
• Added golden_data/golden_observations.csv and updated golden_data/golden_summary_report.csv.

[gemini-code-assist]

Code Review

This pull request updates the IPEDS student-to-faculty ratio import to support downloading and preprocessing both revised and provisional datasets, adding a measurementMethod column to handle provisional estimates. Feedback on these changes suggests simplifying the download logic in download.py to prevent redundant zip file downloads by trying candidate URLs in order of preference and extracting files in a single pass. Additionally, the reviewer notes that dropping the measurementMethod column from revised files in preprocess.py would cause schema inconsistencies across years, recommending instead to keep the column and populate it with empty strings for revised datasets.

[gemini-code-assist]

The current implementation of process_and_filter_zip and download_for_year splits the download process into two separate phases (Phase 1 for revised data, Phase 2 for provisional data). This causes the script to download the same large zip files multiple times (up to 8 times per year if only provisional data is available), which is highly inefficient and risks rate-limiting or IP bans from the NCES servers.

We can simplify this significantly by trying the candidate URLs in order of preference (with BASE_URL_RV first) and extracting whatever matches either the revised or provisional pattern. Since preprocess.py already handles selecting the revised file and deleting redundant files, we only need to download the first working zip file per year.

def process_and_filter_zip(zip_path: str, output_dir: str) -> bool:
    """
    Unzips and filters contents of zip_path.
    Extracts files matching RV_PATTERN or PROVISIONAL_PATTERN.
    Returns True if matching file(s) were found and extracted, False otherwise.
    """
    zip_filename = os.path.basename(zip_path)

    try:
        with zipfile.ZipFile(zip_path, 'r') as zip_ref:
            all_files = zip_ref.namelist()
            files_to_extract = [
                f for f in all_files if RV_PATTERN.search(os.path.basename(f))
            ]
            if not files_to_extract:
                files_to_extract = [
                    f for f in all_files if PROVISIONAL_PATTERN.search(os.path.basename(f))
                ]

            if not files_to_extract:
                return False

            for file_name in files_to_extract:
                zip_ref.extract(file_name, output_dir)
                logging.info("    Extracted: %s", file_name)

            return True

    except zipfile.BadZipFile:
        logging.fatal("  FATAL ERROR: %s is a corrupted or empty zip file. Cannot proceed.", zip_filename)
        raise RuntimeError(f"Corrupted or empty zip file encountered: {zip_filename}")
    except Exception as e:
        logging.fatal("  FATAL ERROR: An unexpected error occurred during unzipping/extraction of %s: %s", zip_filename, e)
        raise RuntimeError(f"Extraction failed for {zip_filename}: {e}")
    finally:
        if os.path.exists(zip_path):
            try:
                os.remove(zip_path)
            except OSError:
                pass

References

1. When downloading and processing ZIP files, ensure that exception handling covers both network-related errors (e.g., requests.exceptions.RequestException) and ZIP-specific errors (e.g., zipfile.BadZipFile) to prevent the script from crashing on corrupted or invalid downloads.

[gemini-code-assist]

Simplify download_for_year to try candidate URLs in order of preference, downloading and extracting the first successful match. This completely eliminates the redundant downloads and the complex two-phase logic. Also, configure a higher number of retries (e.g., tries=10) to ensure download success from unstable external sources, and handle network-related exceptions.

def download_for_year(year: int) -> bool:
    """
    Downloads data for a given year by trying candidate URLs in order of preference.
    """
    logging.info("\nProcessing year %d...", year)

    urls_to_try = [
        BASE_URL_RV,
        BASE_URL_LEGACY,
        BASE_URL_COMPLETE,
        BASE_URL_PROV,
    ]

    for url_template in urls_to_try:
        if "{year}" in url_template:
            url = url_template.format(year=year)
        else:
            url = url_template.format(year)

        zip_filename = f"EF{year}D.zip"
        download_path = os.path.join(DOWNLOAD_DIR, zip_filename)

        try:
            download_success = download_file(
                url=url,
                output_folder=DOWNLOAD_DIR,
                unzip=False, # <-- CRITICAL: Do not let utility unzip file
                tries=10,
                delay=1,
                backoff=1
            )

            if download_success and os.path.exists(download_path) and zipfile.is_zipfile(download_path):
                if process_and_filter_zip(download_path, DOWNLOAD_DIR):
                    logging.info("  Successfully fetched dataset for year %d.", year)
                    return True
        except requests.exceptions.RequestException as e:
            logging.info("  Network error for candidate URL %s: %s", url, e)
        except Exception as e:
            logging.info("  Candidate URL %s failed: %s", url, e)

    logging.info("Warning: No dataset found for year %d across candidate URLs.", year)
    return False

References

1. When downloading data from unstable or unreliable external sources, configure a higher number of retries (e.g., tries=10) to ensure download success, even if it results in a long cumulative wait time.
2. When downloading and processing ZIP files, ensure that exception handling covers both network-related errors (e.g., requests.exceptions.RequestException) and ZIP-specific errors (e.g., zipfile.BadZipFile) to prevent the script from crashing on corrupted or invalid downloads.

[gemini-code-assist]

Dropping the measurementMethod column from revised files will cause the CSV schema to be inconsistent across different years. Since all CSV files in this import share the same student_faculty_ratio_metadata.csv and student_faculty_ratio.tmcf, they must all contain the exact same columns. If a revised CSV is missing the measurementMethod column, the import tool will fail with a 'column not found' error.

Instead of dropping the column, keep it in the revised CSV but set its values to an empty string.

                # Check if the source dataset is provisional (lacks '_rv' suffix)
                is_provisional = "_rv" not in selected_filename.lower()
                if is_provisional:
                    df["measurementMethod"] = "NCES_ProvisionalEstimate"
                    logging.info(f"Added 'measurementMethod' = 'NCES_ProvisionalEstimate' to provisional file {new_filename}")
                else:
                    df["measurementMethod"] = ""
                    logging.info(f"Set empty 'measurementMethod' for revised file {new_filename}")