Sounds like the guy was just bored and wanted to mess with you. Watching someone prune for 10 minutes is not exactly must see TV, so whatever he said was probably just him killing time. People say stuff like that on job sites all day long and nobody remembers it by lunch. Getting called out by some random dude does not mean you did anything wrong or that it actually matters. Unless he was your boss or the client, his opinion carries about as much weight as a fart in the wind. I would not lose one minute of sleep over it.