Actually, color is famously unreliable for mineral ID, even as a first guess. Something like quartz can be clear, purple, pink, or smoky grey. Hardness is way more useful from the start.
Okay but honestly color catches my eye first every time. Sure hardness tells you more, but I'm gonna notice if something's bright green before I start scratching it.