视频采集渲染流程分析

在增加滤镜功能之前，需要对 WebRTC 视频采集的流程有一定了解。

WebRTC 中定义了 VideoCapture 接口类，其中定义了相机的初始化，预览，停止预览销毁等操作。

实现类是 CameraCapture，并且封装了Camera1Capture、Camera2Capture 两个子类，甚至还有屏幕共享。

WebRTC 中开始视频采集非常的简单：

```kotlinval videoCapture = createVideoCapture()videoSource = videoCapture.isScreencast.let { factory.createVideoSource(it) }videoCapture.initialize(surfaceTextureHelper,applicationContext,videoSource?.capturerObserver)videoCapture.startCapture(480, 640, 30)```

这里主要看一下 VideoSource类和capturerObserver。

VideoSource 中有以下方法

```java@Override    public void onFrameCaptured(VideoFrame frame) {      final VideoProcessor.FrameAdaptationParameters parameters =          nativeAndroidVideoTrackSource.adaptFrame(frame);      synchronized (videoProcessorLock) {        if (videoProcessor != null) {          videoProcessor.onFrameCaptured(frame, parameters);          return;        }      }      VideoFrame adaptedFrame = VideoProcessor.applyFrameAdaptationParameters(frame, parameters);      if (adaptedFrame != null) {        nativeAndroidVideoTrackSource.onFrameCaptured(adaptedFrame);        adaptedFrame.release();      }    }```

采集到的视频帧数据会回调给 onFrameCaptured，在这里会做一下对视频的裁切缩放处理，并通过nativeAndroidVideoTrackSource传递给 Native层。

重点是 VideoProcessor 对象，据查是在2019年2月新增的。VideoSource里面有 setVideoProcessor 方法用于设置VideoProcessor，在上面方法中可知，如果设置了VideoProcessor，视频帧则走VideoProcessor的onFrameCaptured，否则的话直接传入 Native。

用 VideoProcessor 来实现处理发送前的视频帧非常方便，我们先来看下VideoProcessor类。

```javapublic interface VideoProcessor extends CapturerObserver {  public static class FrameAdaptationParameters {   ...    public FrameAdaptationParameters(int cropX, int cropY, int cropWidth, int cropHeight,        int scaleWidth, int scaleHeight, long timestampNs, boolean drop) {      ...    }  }  default void onFrameCaptured(VideoFrame frame, FrameAdaptationParameters parameters) {    VideoFrame adaptedFrame = applyFrameAdaptationParameters(frame, parameters);    if (adaptedFrame != null) {      onFrameCaptured(adaptedFrame);      adaptedFrame.release();    }  }.... }```

VideoSource中调用的 onFrameCaptured(frame, parameters) 并非CapturerObserver的onFrameCaptured，也就是暂时不会传入Native增，它在这个方法中也做了对ViewFrame的裁切缩放，之后再传入底层。

所以我们可以在这里实现对视频帧的美颜滤镜处理。

```kotlin class FilterProcessor : VideoProcessor{   	   			private var videoSink:VideoSink               override fun onCapturerStarted(success: Boolean) {        }        override fun onCapturerStopped() {        }        override fun onFrameCaptured(frame: VideoFrame?) {           val newFrame = // TODO: 在这对VideoFrame进行视频滤镜美颜处理           sink.onFrame(newFrame)        }        override fun setSink(sink: VideoSink?) {            //设置视频接收器 用来渲染并将frame传入Native          videoSink = sink        }    }val videoCapture = createVideoCapture()videoSource = videoCapture.isScreencast.let { factory.createVideoSource(it) }videoSource.setVideoProcessor(FilterProcessor())//设置处理器videoCapture.initialize(surfaceTextureHelper,applicationContext,videoSource?.capturerObserver)videoCapture.startCapture(480, 640, 30)```

美颜的话可以用 GPUImage，也可以用商用SDK。

以上是在应用层的实现，利用 WebRTC自带的类就行。如果是NDK开发，道理也是一样的。

创建一个代理类 CapturerObserverProxy 实现 CapturerObserver，并将真正的 nativeCapturerObserver传进来，Native会回调视频帧数据给 CapturerObserverProxy的 onFrameCaptured，然后在 onFrameCaptured 中对视频进行美颜滤镜处理，再将处理好的 VideoFrame 用 nativeCapturerObserver 传给底层编码传输。

```javapublic class CapturerObserverProxy implements CapturerObserver {    public static final String TAG = CapturerObserverProxy.class.getSimpleName();    private CapturerObserver originalObserver;    private RTCVideoEffector videoEffector;    public CapturerObserverProxy(final SurfaceTextureHelper surfaceTextureHelper,                                 CapturerObserver observer,                                 RTCVideoEffector effector) {        this.originalObserver = observer;        this.videoEffector = effector;        final Handler handler = surfaceTextureHelper.getHandler();        ThreadUtils.invokeAtFrontUninterruptibly(handler, () ->                videoEffector.init(surfaceTextureHelper)        );    }    @Override    public void onCapturerStarted(boolean success) {        this.originalObserver.onCapturerStarted(success);    }    @Override    public void onCapturerStopped() {        this.originalObserver.onCapturerStopped();    }    @Override    public void onFrameCaptured(VideoFrame frame) {        if (this.videoEffector.needToProcessFrame()) {            VideoFrame.I420Buffer originalI420Buffer = frame.getBuffer().toI420();            VideoFrame.I420Buffer effectedI420Buffer =                    this.videoEffector.processByteBufferFrame(                            originalI420Buffer, frame.getRotation(), frame.getTimestampNs());            VideoFrame effectedVideoFrame = new VideoFrame(                    effectedI420Buffer, frame.getRotation(), frame.getTimestampNs());            originalI420Buffer.release();            this.originalObserver.onFrameCaptured(effectedVideoFrame);        } else {            this.originalObserver.onFrameCaptured(frame);        }    }} videoCapturer.initialize(videoCapturerSurfaceTextureHelper, context, observerProxy);```

以上就是给 WebRTC 增加美颜功能的实现～