Putting INSIGHT on Hold and Focusing on Form B Data Consolidation
This week, I placed the INSIGHT project on hold and shifted my focus to cleaning and consolidating Form B data from multiple years. Instead of continuing system-side uploads, I worked on preparing the datasets externally through Python scripting, which proved to be a more practical and reliable approach at this stage.
My main task was to clean the raw Excel and CSV files and create a consolidated view of the Form B records. Since the forms came from different academic years and followed varying formats, I used Python scripts to standardize the structure, clean inconsistent values, and generate usable consolidated outputs. This helped make the data more organized and easier to review before any future integration into a system.
The INSIGHT system was temporarily paused because uploading all of the raw files directly into the platform would require rigorous mapping for each form type. Given the volume of files and the differences in layout across years, forcing everything into the system immediately would have taken more time and introduced more errors. For the meantime, Python-based data processing was the better solution because it allowed faster cleanup, more controlled transformations, and clearer validation of the outputs.
Through this process, I was able to improve the readiness of the Form B datasets while also identifying the kinds of structural differences that would need to be considered later if INSIGHT resumes. Rather than rushing incomplete uploads, I focused on producing cleaner and more dependable consolidated data first.
