dacc.log

Problem

We’re working on a project at Rieke Lab that has some staunch video timing and reliability requirements. We need to deliver frames without any underruns, and to know the exact moment they’re actually displayed on the screen. This requires that our rendering code runs strictly between refreshes, and that the back buffer is swapped in at the right moment.

Cocoa, OpenGL, and CoreVideo are the tools we picked, and it’s been tricky setting them up and understanding their roles. Hopefully this post will save someone the effort of getting them working together.

There is a CoreVideoOpenGLView subclass of NSOpenGLView attached to this post that I believe achieves our aims while drawing full-screen 2D animations.

Read on for details…

Solution

Refresh Sync

OpenGL has an option that coordinates the buffer swaps with display refreshes. Since we’re using Cocoa, we set this on an NSOpenGLContext:

if ( useOpenGLSync ) {
  const GLint swapInterval = 60;
  [context setValues: &swapInterval
        forParameter: NSOpenGLCPSwapInterval];
}

Once this is set, we know that partially drawn buffers won’t go out. i.e. We won’t see that characteristic flicker when part of the buffer is the previous frame and the other part the latest. However, we have no way of detecting an underrun, or knowing exactly when a frame will actually be displayed.

Enter CoreVideo.

Timing and Underruns

The CoreVideo DisplayLink invokes a callback between display refreshes. It provides a timestamp indicating when the frame will actually appear on the display. If this timestamp is in the past, you know you’ve underrun.

First create the callback:

CVReturn drawCallback(
  CVDisplayLinkRef displayLink,
  const CVTimeStamp *now,
  const CVTimeStamp *outputTime,
  CVOptionFlags flagsIn,
  CVOptionFlags *flagsOut,
  void *context
)
{
  NSAutoreleasePool *pool = [[NSAutoreleasePool alloc] init];
  [(CoreVideoOpenGLView *) context renderFrame];
  [pool release];
 
  return kCVReturnSuccess;
}

.. then setup the DisplayLink:

if ( useCoreVideo ) {
  CVDisplayLinkCreateWithCGDisplay(displayId, &displayLink);
  CVDisplayLinkSetOutputCallback(displayLink, drawCallback, self);
 
  CVDisplayLinkStart(displayLink);
}

Full Screen Mode

Going full-screen from an NSOpenGLView turned out to be more difficult than you’d expect. Lots of voodoo here, but the attached class works — not always for reasons I understand.

One critical thing appears to be managing the full-screen NSOpenGLContext outside of the NSOpenGLView:

if ( fullScreenMode ) {
  NSOpenGLPixelFormatAttribute attributes [] = {
    NSOpenGLPFADoubleBuffer,
    NSOpenGLPFAAccelerated,
    NSOpenGLPFAColorSize, 16,
    NSOpenGLPFADepthSize, 16,
    NSOpenGLPFAFullScreen,
    NSOpenGLPFAScreenMask,
    CGDisplayIDToOpenGLDisplayMask(displayId),
    0
  };
 
  NSOpenGLPixelFormat* pixelFormat = [[NSOpenGLPixelFormat alloc]
    initWithAttributes: attributes];
 
  context = [[NSOpenGLContext alloc] initWithFormat: pixelFormat
                                       shareContext: nil];
}
else {
  context = [self openGLContext];
}
 
if ( fullScreenMode ) {
  [context setFullScreen];
}

2D Drawing

For our purposes we want to draw only in 2D. This is mostly a matter of setting a 2D orthographic projection.

glViewport(0, 0, width, height);
 
glShadeModel(GL_FLAT);
 
glMatrixMode(GL_PROJECTION);
glLoadIdentity();
gluOrtho2D(0.0, [widthNumber doubleValue], 0.0, [heightNumber doubleValue]);
 
glMatrixMode(GL_MODELVIEW);
glLoadIdentity();
 
glPixelStorei(GL_UNPACK_ALIGNMENT, 1);

I believe the last line is related to using glDrawPixels — can’t remember precisely.

Caveat Emptor

This solution is still pending rigorous testing. We have some electronics set up that allow us to digitize actual flashes of light produced by a small OLED display. By counting electrical pulses and correlating their timestamps with CoreVideo, we should get a good empirical verification that frames are going out when we think they are.

Example Code

The attached example project contains an app that lets you turn on OpenGL refresh syncs, CoreVideo, and full-screen mode in turn. You can only Command-Q after going into full-screen. =) The frame rendering is done from Python.