They were categorized as jewish id they were not the "perfect human" by Hitlers standard of the "perfect person" was someone with blond, blue eyes and fair skin. If they did not have these features Hitler would categorize them as jews and would send them to concentration camps. Also he had used the star of david to determine whether they are jewish or not and they are to wear it at all times. This was one way Hitler could tell if the person was a German or a Jew.
