Putting INSIGHT on Hold and Focusing on Form B Data Consolidation

This week, I placed the INSIGHT project on hold and shifted my focus to cleaning and consolidating Form B data from multiple years. Instead of continuing system-side uploads, I worked on preparing the datasets externally through Python scripting, which proved to be a more practical and reliable approach at this stage.

My main task was to clean the raw Excel and CSV files and create a consolidated view of the Form B records. Since the forms came from different academic years and followed varying formats, I used Python scripts to standardize the structure, clean inconsistent values, and generate usable consolidated outputs. This helped make the data more organized and easier to review before any future integration into a system.

The INSIGHT system was temporarily paused because uploading all of the raw files directly into the platform would require rigorous mapping for each form type. Given the volume of files and the differences in layout across years, forcing everything into the system immediately would have taken more time and introduced more errors. For the meantime, Python-based data processing was the better solution because it allowed faster cleanup, more controlled transformations, and clearer validation of the outputs.

Through this process, I was able to improve the readiness of the Form B datasets while also identifying the kinds of structural differences that would need to be considered later if INSIGHT resumes. Rather than rushing incomplete uploads, I focused on producing cleaner and more dependable consolidated data first.

Python, VSCode, Excel

A major challenge this week was handling large amounts of Form B files coming from different years, each with slight differences in structure, labels, and formatting. I addressed this by using Python scripts to automate repetitive cleaning steps and by building a consolidated view that made cross-year data easier to compare and validate.

Another challenge was deciding whether to keep pushing the INSIGHT upload workflow despite the mapping complexity. I resolved this by putting the system work on hold for now and prioritizing data preparation outside the platform. This made the work more manageable and reduced the risk of uploading inconsistent or incorrectly mapped records.