I thought the public key was suppose to be the one you kept and the private key would stay on the server
In a scenario where you use key based authentication to log into an SSH server (which in general is remote, foreign, untrusted), your private key is your secret and belongs to your account on your machine, i.e. the SSH client.
Normally you generate a key pair on the (future) SSH client, then distribute the public key to a server (or many servers). The server uses authorized_keys to store your public key(s). This way your most secret private key (my_keyname in your case) never leaves your own machine.
It's not crystal clear but I think you generated the key pair on the machine you planned to use as a server.
and the private key would stay on the server?
It should stay where it was generated, but it should have been generated on the client side in the first place. PuTTY is an SSH client. There's a companion tool named PuTTYgen to generate keys.
Answering follow-up questions (from the OP's comment):
I assumed when I created the keys using the command mentioned in OP that it would have created both the public and private key
Right.
… I would take the public key and access the server henceforth.
Your public key is meant to be… well, public. You can distribute it to servers all over the world, so they will be able to tell you are you. For this to work you need to authenticate using the corresponding secret private key.
Though back to my initial question, am I using the keys correctly? Or did I grab my private key and essentially using it as my access key?
You do use your private key as your access key. So yes, you're using the keys correctly now, but you generated them on the "wrong" end of the client-server setup, so you needed to "grab" the private one from one machine to the other. Normally a private key doesn't need to be transferred because an SSH-savvy user generates a key pair where the private key should stay, and transfers the public key to other machines.
There is was a comment (now deleted) that I find very misleading. It's not from the OP and it may easily confuse the OP (or any other user), therefore I'm addressing it here.
should have created the public and private key on the server, and then used the public key on the client
A key pair should be generated where its private key will be used. It's technically possible to generate keys elsewhere and just copy each to where it should finally be (they are just files after all), but the point is your private key should be private, i.e. known to a single machine/account. Copying a private key between machines is against this rule. In this case the private key is to authenticate the client (PuTTY), so it should exist on the client machine from the very beginning.
This allows the server which has the private key (which is used to decrypt the data) to communicate with the client which is encrypting the data (with the public key).
No. These keys are used for authentication, i.e proving that you are who you claim you are. Encryption is a separate thing and it's performed by symmetrical encryption key, session-based.
To authenticate itself, a client needs its private key whose public counterpart is known to the server beforehand. Creating a key pair on the client machine and distributing the private key is what should be done here.
But the server has its own private key(s) (usually in /etc/ssh/, system-wide) and when any client connects for the first time, the corresponding public key is pushed to the client and stored in known_hosts. This way the client can tell in the future this particular server is the very same server as usual. If the client connects again and the server fails to authenticate with the very same key, the (client) user is warned, this may indicate MITM attack.
private key (which is used to decrypt the data) […] encrypting the data (with the public key)
This asymetrical encryption is not the case here. Rather in PGP/GPG: you encrypt with the public key, so only the owner of the private key can decrypt (oversimplified anyway, yet this is the concept). In this scenario anyone can encrypt because the public key is known to the world. It may be the other way around: you encrypt with your own private key and anyone can decrypt and be sure the message comes exactly from you.
But not in SSH. Here the key pairs are used to authenticate relevant parties, then symmetrical encryption comes into play.
