"It was testimony to the romantic speculation he inspired that there were whispers about him from those who had found little that it was necessary to whisper about in this world." Nick meant that the people at the party were rather wild and daring in their casual acceptance of all kinds of behavior, so their being in awe of Gatsby was remarkable.