Modify RubyGems install location on Windows

I am making a RubyPortable (PAF) for Windows and I want to have Rubygems not install to the user directory (C:/Users/username/.gem/ruby/some_version_number) or even have that in the GEM_PATH so that when using it on other machines and installing gems they are not left on that machine; that is to say the only location I want them installed to is GEM_HOME.

I have searched all over but am still unclear how to do this. Ideally, I would want to make it so that it is a permanent modification that is always enabled whenever I start RubyPortable but a ‘temporary’ fix that enables it for that session only would work as well.

Can anyone point me in the right direction?

Thank you.