A black hole has a mass and deforms spacetime, a chair has a mass and deforms spacetime. The chair might have a more complex internal structure (we are not sure, because we can't check for black holes), but why would that be relevant to call it a physical object? A black hole is obviously there in space, so it's a physical object.