Middlemen give there time to help others conduct trades safely, and they are being trusted with the amount of gold to get that feedback
Also you would need a level of feedback in order to be able to middleman anyway since who would trust a newcomer
Think there would be more scams without this really, than those that could potentially come from the feedback itself. Edit, more scams without the availability of free trustable middlemen, some might not want to pay a fee and risk it etc, obviously trusted can still do for free without feedback, however less incentive to do so without some form of payment due to the time taken in doing the middle manning