It is believed that Hamlet speaks rudely to Ophelia because of his pent up rage towards his mother. He has cursed all women and is likely rude to Ophelia because she falls under the category of frail women and he is also hoping that his mother might hear his rant. It could also be because he once loved Ophelia and hopes to protect her by pushing her away from him with rudeness. Or he could be being rude because he hopes she never enters into the "filthy" act of sex if she is in a nunnery and can never "breed" with sinners or breed more sinners like him.