Camera and iPhone

It’s been a while since i wrote last post but it is time to cover another topic. I start seeing more and more Instagram like apps and as i have to put some camera preview in one of the last apps i was developing. I come to idea that is it a good thing to talk about. Basically if you want to use camera in you app you are using UIImagePickerController and in most cases is good enought. But what if you really need some nice custom animations and most of all custom size?

Yeah you can say that if it comes to this we can use AVCaptureVideoPreviewLayer and this will cover all edge cases with animation and additional decorations , size etc. Ok you are right in this case your code will looks like this and we are done.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
- (void)setupCamera
{
  //setup session
    self.session = [[AVCaptureSession alloc] init];
    self.session.sessionPreset = AVCaptureSessionPresetPhoto;
    self.videoDevice = [self frontCamera];
    self.videoInput = [AVCaptureDeviceInput deviceInputWithDevice:self.videoDevice error:nil];
    [self.session addInput:self.videoInput];

    self.captureVideoPreviewLayer = [[AVCaptureVideoPreviewLayer alloc] initWithSession:self.session];
    CALayer *viewLayer = self.cameraButton.layer;

    [viewLayer setMasksToBounds:YES];

    self.captureVideoPreviewLayer.videoGravity = AVLayerVideoGravityResizeAspectFill;
    [[self.captureVideoPreviewLayer connection] setVideoOrientation: AVCaptureVideoOrientationLandscapeRight];
    self.captureVideoPreviewLayer.frame = [viewLayer bounds];
    [viewLayer addSublayer:self.captureVideoPreviewLayer];

    // Start Running the Session
    [self.session startRunning];

    //fade in 
    CABasicAnimation *anim = [CABasicAnimation animationWithKeyPath:@"opacity"];
    anim.fromValue = @(0);
    anim.toValue = @(1);
    anim.duration = 0.5f;
    [self.captureVideoPreviewLayer addAnimation:anim forKey:nil];
}

But we don’t want it to be easy right? well let’s say we need some effect like in instagram to make it simple let’s use saturation:

I’m using here OpenGL GLKitViewController attached to my rootViewController with some custom shader.

Advantages:

-Fully customizable
-Whatever size i want
-Can apply CAAnimation, UIView aniamtion etc
-It’s bloody fast

Of course i’m not telling this is the best option possible i saw some benchmarks revealing that Accelerate framework from Apple can win i some cases. This is a good moment to talk about something i saw that many developers that are not involved in gamedev thought that learning OpenGL is wast of time, which is not true as knowing how GPU works and how to program it gives you almost endless possibilities and in many cases the best performance possible. Trust me i’m an engineer :)

Ok enought talking show me some code: (I’m assuming that you know the basics of OpenGL ES, if not there are some good tutorials that explaing it)

Let’s start with setting up our scene, we need main controller with some GLKitViewController attached to it by embed segue using container.

Now we need to handle the segue in our main controller

1
2
3
4
5
6
7
8
9
10
11
12
13
14
- (void)prepareForSegue:(UIStoryboardSegue *)segue sender:(id)sender
{
    if ([segue.identifier isEqualToString:MDPSegueIdentifiers.customCamera]) {

        MPDCustomCameraViewController *cameraController = segue.destinationViewController;

        self.cameraController = cameraController;

        [self decorateCameraController:cameraController];
        [self animateCameraController:cameraController];

        [cameraController startCameraCapture];
    }
}

Both function are not important right now. One of it simply animate the camera view and second one mask it to the shape of the circle. Now let’s talk about our heart of app.

To render Camera output in OpenGL we need to capture it texture somehow. Gladly Apple AVCaptureSession allow us to bind camera output directly to textures using YCbCr color space.

First we need to setup OpenGL view:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
- (void)viewDidLoad {

    [super viewDidLoad];

    self.context = [[EAGLContext alloc] initWithAPI:kEAGLRenderingAPIOpenGLES2];

    if (!self.context) {
        NSLog(@"Failed to create ES context");
    }

    //create Opengl view we are processing view update on demand so pause the view
    self.glView = (GLKView *)self.view;
    self.glView.context = self.context;
    self.glView.drawableDepthFormat = GLKViewDrawableDepthFormat24;
    self.preferredFramesPerSecond = 60;

    self.glView.delegate = self;

    self.view.backgroundColor = [UIColor whiteColor];

    self.retinaScaleFactor = [[UIScreen mainScreen] nativeScale];

    self.view.contentScaleFactor = self.retinaScaleFactor;

    _sessionPreset = AVCaptureSessionPreset640x480;

    [self setupGL];
    [self setupAVCapture];
}

Now lets setup AVSession to preapre data capture into 2 textures one containing chroma and second one luma as described in YCbCr spec.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
 CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL, _context, NULL, &_videoTextureCache);
    if (err)
    {
        NSLog(@"Error at CVOpenGLESTextureCacheCreate %d", err);
        return;
    }

    //-- Setup Capture Session.
    self.session = [[AVCaptureSession alloc] init];
    [self.session beginConfiguration];

    //-- Set preset session size.
    [self.session setSessionPreset:_sessionPreset];

    //-- Creata a video device and input from that Device.  Add the input to the capture session.
    NSArray *devices = [AVCaptureDevice devicesWithMediaType:AVMediaTypeVideo];

    AVCaptureDevice * videoDevice = nil;

    for (AVCaptureDevice *device in devices) {

        if ([device hasMediaType:AVMediaTypeVideo]) {

            if ([device position] == AVCaptureDevicePositionFront) {
                videoDevice = device;
                break;
            }
        }
    }

    if(videoDevice == nil)
        assert(0);

    //-- Add the device to the session.
    NSError *error;
    AVCaptureDeviceInput *input = [AVCaptureDeviceInput deviceInputWithDevice:videoDevice error:&error];
    if(error)
        assert(0);

    [self.session addInput:input];

    //-- Create the output for the capture session.
    AVCaptureVideoDataOutput * dataOutput = [[AVCaptureVideoDataOutput alloc] init];
    [dataOutput setAlwaysDiscardsLateVideoFrames:YES]; // Probably want to set this to NO when recording

    //-- Set to YUV420.
    [dataOutput setVideoSettings:[NSDictionary dictionaryWithObject:[NSNumber numberWithInt:kCVPixelFormatType_420YpCbCr8BiPlanarFullRange]
                                                             forKey:(id)kCVPixelBufferPixelFormatTypeKey]]; // Necessary for manual preview

    // Set dispatch to be on the main thread so OpenGL can do things with the data
    [dataOutput setSampleBufferDelegate:self queue:dispatch_get_main_queue()];

    [self.session addOutput:dataOutput];
    [self.session commitConfiguration];

This code simply setup camera output to render in YCbCr mode which will return in delegate data for 2 textues. Also we setup device to front camera if you want to use Back camera modify this code:

1
2
3
4
5
6
7
8
9
10
11
12
AVCaptureDevice * videoDevice = nil;

    for (AVCaptureDevice *device in devices) {

        if ([device hasMediaType:AVMediaTypeVideo]) {

            if ([device position] == AVCaptureDevicePositionFront) {
                videoDevice = device;
                break;
            }
        }
    }

And replace AVCaptureDevicePositionFront with AVCaptureDevicePositionBack.

Now most important function this one will capture our output and generate new textues for shader every frame.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
- (void)captureOutput:(AVCaptureOutput *)captureOutput
didOutputSampleBuffer:(CMSampleBufferRef)sampleBuffer
       fromConnection:(AVCaptureConnection *)connection
{
    CVReturn err;
    CVImageBufferRef pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer);
    size_t width = CVPixelBufferGetWidth(pixelBuffer);
    size_t height = CVPixelBufferGetHeight(pixelBuffer);

    if (!_videoTextureCache)
    {
        NSLog(@"No video texture cache");
        return;
    }

    [self cleanUpTextures];

    // CVOpenGLESTextureCacheCreateTextureFromImage will create GLES texture
    // optimally from CVImageBufferRef.

    // Y-plane
    glActiveTexture(GL_TEXTURE0);
    err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                       _videoTextureCache,
                                                       pixelBuffer,
                                                       NULL,
                                                       GL_TEXTURE_2D,
                                                       GL_RED_EXT,
                                                       (int)width,
                                                       (int)height,
                                                       GL_RED_EXT,
                                                       GL_UNSIGNED_BYTE,
                                                       0,
                                                       &_lumaTexture);
    if (err)
    {
        NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
    }

    glBindTexture(CVOpenGLESTextureGetTarget(_lumaTexture), CVOpenGLESTextureGetName(_lumaTexture));
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);

    // UV-plane
    glActiveTexture(GL_TEXTURE1);
    err = CVOpenGLESTextureCacheCreateTextureFromImage(kCFAllocatorDefault,
                                                       _videoTextureCache,
                                                       pixelBuffer,
                                                       NULL,
                                                       GL_TEXTURE_2D,
                                                       GL_RG_EXT,
                                                       (int)(width * 0.5),
                                                       (int)(height * 0.5),
                                                       GL_RG_EXT,
                                                       GL_UNSIGNED_BYTE,
                                                       1,
                                                       &_chromaTexture);
    if (err)
    {
        NSLog(@"Error at CVOpenGLESTextureCacheCreateTextureFromImage %d", err);
    }

    glBindTexture(CVOpenGLESTextureGetTarget(_chromaTexture), CVOpenGLESTextureGetName(_chromaTexture));
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_S, GL_CLAMP_TO_EDGE);
    glTexParameterf(GL_TEXTURE_2D, GL_TEXTURE_WRAP_T, GL_CLAMP_TO_EDGE);
}

Rendering is quiet simple:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
- (void)glkView:(GLKView *)view drawInRect:(CGRect)rect
{
    glViewport(0, 0,
               rect.size.width*self.retinaScaleFactor,
               rect.size.height*self.retinaScaleFactor);

    glClearColor(0.5f, 0.5f, 0.5f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT | GL_DEPTH_BUFFER_BIT);

    if(_lumaTexture != 0 && _chromaTexture != 0){

        glUseProgram(_shaderProgram);

        glActiveTexture(GL_TEXTURE0);
        glBindTexture(CVOpenGLESTextureGetTarget(_lumaTexture), CVOpenGLESTextureGetName(_lumaTexture));
        glUniform1i(_tex1Uniform,0);
        glActiveTexture(GL_TEXTURE1);
        glBindTexture(CVOpenGLESTextureGetTarget(_chromaTexture), CVOpenGLESTextureGetName(_chromaTexture));
        glUniform1i(_tex2Uniform,1);
        glUniformMatrix4fv(_matrixUniform, 1 ,false ,GLKMatrix4MakeRotation(0, 0, 0, 1).m);
        glUniform1f(_saturationUniform, _saturation);
        glDisable(GL_DEPTH_TEST);

        glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, squareVertices);
        glEnableVertexAttribArray(ATTRIB_VERTEX);
        glVertexAttribPointer(ATTRIB_COORDS, 2, GL_FLOAT, 0, 0, textureVertices);
        glEnableVertexAttribArray(ATTRIB_COORDS);

        glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
    }
}

Simply set the viewport to whole view size and fireup shader that combines both textures into final color.

1
2
3
4
5
6
7
8
9
mediump vec3 yuv;
mediump vec3 rgb;

yuv.x  = texture2D(SamplerY, texCoordVarying).r;
yuv.yz = texture2D(SamplerUV, texCoordVarying).rg - vec2(0.5, 0.5);

rgb = mat3( 1.0, 1.0, 1.0,
            0.0, -.18732, 1.8556,
            1.57481, -.46813, 0.0) * yuv;

One more thing thanks to shaders and realtime computing i was eg able to do relatime saturation filter, filter itself is quiet simple. You just take a output color as vector3 and get as result dot product of the color and some precomuted constant value vector

1
2
mediump vec3 gray = vec3(dot(rgb, vec3(0.2126, 0.7152, 0.0722)));
mediump vec3 outColor = mix(rgb, gray, saturationFactor);

See that we use uniform to mix between gray and output color.

And that’s all i’v got for today as always full source code available on my github. As you can see now you can apply multiple effects on you camera simply by writing some additional shaders or extending current one, and all works in realtime.

P.S. Remember that you need to run this on physical device to use camera as it is not available on simulator.