Creo que ya sé lo que te pasa: ¿Has incluido el flag DM_DISPLAYFREQENCY en dmFields?
   Cita:  · lpDevMode
Pointer to a DEVMODE structure that describes the graphics mode to switch to. The dmSize member must be initialized to the size, in bytes, of the DEVMODE structure. The following fields in the DEVMODE structure are used:
dmBitsPerPel
 Bits per pixel
dmPelsWidth
 Pixel width
dmPelsHeight
 Pixel height
dmDisplayFlags
 Mode flags
dmDisplayFrequency
 Mode frequency
 
In addition to setting a value in one or more of the preceding DEVMODE members, you must also set the appropriate flags in the dmFields member. The flags indicate which members of the DEVMODE structure are used for the display settings change. If the appropriate bit is not set in dmFields, the display setting will not be changed. Set one or more of the following flags:
DM_BITSPERPEL
 Use the dmBitsPerPel value.
DM_PELSWIDTH
 Use the dmPelsWidth value.
DM_PELSHEIGHT
 Use the dmPelsHeight value.
DM_DISPLAYFLAGS
 Use the dmDisplayFlags value.
DM_DISPLAYFREQENCY
 Use the dmDisplayFrequency value.