glutEstablishOverlay establishes an overlay (if possible) for the current window.
glutEstablishOverlay establishes an overlay (if possible) for the current window. The requested display mode for the overlay is determined by the initial display mode. glutLayerGet(GLUT_OVERLAY_POSSIBLE) can be called to determine if an overlay is possible for the current window with the current initial display mode. Do not attempt to establish an overlay when one is not possible; GLUT will terminate the program.
If glutEstablishOverlay is called when an overlay already exists, the existing overlay is first removed, and then a new overlay is established. The state of the old overlay's OpenGL context is discarded.
The initial display state of an overlay is shown, however the overlay is only actually shown if the overlay's window is shown.
Implicitly, the window's layer in use changes to the overlay immediately after the overlay is established.
X Implementation Notes
GLUT for X uses the SERVER_OVERLAY_VISUALS convention  is used to determine if overlay visuals are available. While the convention allows for opaque overlays (no transparency) and overlays with the transparency specified as a bitmask, GLUT overlay management only provides access to transparent pixel overlays.
Until RGBA overlays are better understood, GLUT only supports color index overlays.