Try shutting down the VM, then in the manager (not the VM), go to File|Preferences|Display and set the Maximum Guest Screen Size to None.

Source: virtualbox.org • View topic – Win 7 resolution max 1024×768