I found a red scotch pad(more agressive than green) works really well.
i cut a circle out of a red scotchpad and mounted it on a rubber grinder backing pad with a bolt through and used it on a 3000 rpm mains drill
it cleans the glue off and only lightly buffs the hypalon, after which you can slop some toluene or cellulose thinners or evostick thinners over it and blend it into the surrounding area.
theres areas on my boat where i pulled the rope bags off and they are invisible now its weathered too.