I think you mix the two concepts of "the singularity is not physical" (which is true, as it is only a mathematical construct) with "black holes are not physical". We use different mathematical constructs to approximate physics effects. Using the notion of a singularity works if you are beyond the event horizon, but if you are close this would be a different and more complicated affair. Nevertheless, a black hole is real, wether or not we can fully describe it in every regime of space-time.