It requires quite an amount of hand tuning and heavily depends on the tolerance of the monitor how far you can go. It depends on how your monitor is connected too.
If your monitor is connected via DisplayPort or via DVI you are often limited to a range between 50 Hz and 60 Hz monitor refresh and therefore limit the highest possible resolution. And not all monitors accept 50 Hz as refresh anyway.
If your monitor is connected via HDMI you have a wider range since a refresh rates of 30 Hz and 24 Hz are allowed.
I am lucky that my monitor even works with 12 Hz which I needed when my settings were not as tweaked as they are now.
All screen shots are in German here, since I am.... guess.
First you have to enable down-sampling.
Enter the Nvidia Control Panel. Hit the right mouse button on an empty space on the desktop and click where I pointed in the screen shot.
Within the panel you select the option "Desktop size and position", for scaling select "No scaling", change the "Scaling with" to "GPU", mark the "override scaling mode from games and programs" check-box. Hit apply. No seriously, many forget to hit apply before they continue.
Next you switch to "Change resolution", hit "Adjust".
Now we are at the interesting part. First you need to click on "Allow resolutions which are not supported by your display" or whatever is written there in English.
These are just my current resolutions, and not all of them are active (see the check-box). I had a much longer list of in-between resolutions.
Now for the first example, 2880x1800 aka "Retina". Be aware, this is the 16:10 setting for a 1920x1200 monitor.
This one is actually quite simple not much to adjust.
First switch the timing from "Auto" to "Manual".
Secondly you enter the resolution you want, here 2880 and 1800.
In most cases you will have to tine the vertical total amount of pixels to 1209.
Before you hit the Test you should end all programs which run with "always on top", like Fraps, task manager, anything! Even if the program is minimized there is a chance that this program will get the focus upon testing, and if you cannot see on your screen your keyboard input goes to the wrong program.
Then hit Test. Usually you should see your desktop at a smaller scale than usual.
A box like this should appear, I propose to hit OK. If you can't see a thing either wait fifteen seconds or hit the "n" key.
In some cases you will get the message "Your device does not support this resolution".
If you can see it you are lucky.
If you are out of luck you get the message-box without being able to see it. Hit "n" or "Return", and hope the keyboard input actually goes to the message box, else you will have to reset your computer.
Now for the second example, 2880x1620, "Retina" as well, but 16:9, fitting nicely on a 1920x1080 screen.
Since I am a nice guy I provide you with a selection of timings which work on most monitors (50 Hz or higher refresh). Most of them are 16:10 though, but you will find it easy to adjust them. Just take the number of columns (for example, 3328), divide it by 16, and multiply it with 9 to get the number of works (example result: 1872).
If you run a 16:9 screen you should always use "Active vertical pixels" with 1080, "total amount of pixels" with 1089. They should work, but your hardware is different, if something blow up, not my problem.
Now we come to the Beyond Ultra HD section. We are leaving the world of 50 Hz or 60 Hz. From here on HDMI is a REQUIREMENT, simply because neither DisplayPort or DVI have 30 Hz or 24 Hz written in their specifications. If you use DP or DVI and still get a picture, tell me.
This is also the region where you may get driver crashes. The list is somewhat incomplete, I spare you many between resolutions and variations with 12Hz, 16Hz, 22Hz, 23Hz, 24Hz or 27Hz when there is a 30 Hz version of the same resolution.