You need to do several runs at wide open throttle with your boat normally laden.
Note the boat's speed and what the engine revs are at WOT (the WOT revs figure being the more important if you haven't got a speedo or GPS)
Find the engine manufacturers recommended WOT range for your engine & see where the figure you achieved is.
Once you've got that you can select the best prop for your usage without a lot of trial & error.
Have a look at this thread & you'll get the idea:
http://www.rib.net/forum/f36/prop-op...5hp-71277.html