The first (and best thing) to understand is; there's always some risk. The key is to minimise it.
I would say stick with the simulator until you can get somewhere that you actually feel comfortable flying - if you were comfortable doing it in the communal garden, you wouldn't feel the need to ask.
Would it be safe? Probably. Even moreso if you agree with the housing association (or other body that monitors the garden) ahead of time and notify residents. They may flat-out refuse you, however, as is their right.