Thanks everyone for being so helpful. I've ended up applying Igor's
But I've dropped this line as I wasn't sure what to do with it:
I am guessing it can be used if I want to match more then just google's
user agent. But in any case what I did worked very nice.
I tested it using Firefox user agent and I got forbidden page, then
tried adding my ip to geo bit and I was allowed.
And there is lot less "google spiders" on our site now, and all of them
have ips I added to allowed list.
Thanks again for fast and great advices.
Since I am a noob when it comes to Nginx and now that I know I can get
some help here I am sure I'll have more questions :)