Update on my failed attempt to catalog every .gov subdomain

I spent three days building a scraper to grab every .gov subdomain from a public dataset, but on day four the whole thing crashed because one weird site returned a 50MB XML file. Now I have 14,000 partial records and no clean way to merge them back together. Should I start over from scratch or try to salvage the broken dataset with some regex hackery?

2 comments

2 Comments

grayt4611d agoTop Commenter

A 500MB CSV of potholes?! That's wild. I wonder if someone actually ran analytics on it or just dumped the whole thing as a flex.

barnes.morgan11d ago

Read a story somewhere about a guy who lost six months of crawler data because one odd PDF ate his whole sqlite database. His fix was to write a script that just skipped any file over 10MB. Worked well enough that he kept the rest of the data. Probably easier to add a filter for file size and keep your 14,000 records than to start over. Those .gov sites have some real weird edge cases, like the one that serves a 500MB CSV of every pothole in a city. Just cut the big files and move on.