FFmpeg H.264 RTSP Server Using Live555 [c++][Code on Git]

Hello again everyone! Today's post is related to video streaming/serving and transcoding.
I was working on an RTSP server to transcode a camera feed on the fly. I needed the server to serve only one client (unicast). I will explain bits of the code here and you will find the full code on my repository.

The Server is composed of 5 components:

1. Transcoder (using FFmpeg):
• Decoder 
• Encoder
2. Live Framed Source (Using Live555)
3. Media Sub-session  (Using Live555)
4. RTSP Server (Using Live555)
5. Main file

Transcoder

Our transcoder here is transcoding any video codec to H.264.
This is done over 2 steps.

Transcoder: Decoder

The decoder role here is to provide a frame for the encoder to be sent later on over network.

The output of the decoder is a uint8_t * buffer of the RGB data. This means it doesn't have to be an FFmpeg decoder, we can use for instance OpenCV and get the buffer data from Mat.data which also a uint8_t *.

The decoder has a constructor that takes a path to the media file to be played. An initialize function to set the FFmpeg structs and get the file info (width, height, GOP*, bitrate and frameRate). A playMedia function which starts to play the video file. An onFrame callback function used to report the arrival of a new frame, so that we can send it later on to the encoder then to the server to be served.The onFrame function is set using the setOnframeCallbackFunction at the beginning of the run. The decoder design is oriented for file processing only. Incase of streaming video,  the decoder need to threaded(streaming packets and deciding).

*GOP: is the interval of I frames, How often do we get I frames.

Transcoder: Encoder

As stated before we want to transcode the video to H.264, the encoder role here is to encode the video frames and fill the RTSP server buffer with them. That's why the encoder has 2 queues one for receiving the input RGB frames and the other one is to output encoded packets to the server buffer.

The sendNewFrame function is simply enqueue a new RGB frame. The encoder main loop is responsible of dequeuing the input queue and call the writeFrame function to encode the frame and enqueue the encoded packet in the output queue. The GetFrame function is called to extract a new packet from the output queue asynchronously from the server side. In case of there is no new frame to write to the server buffer the call of GetFrame is ignored waiting for the encoder resignal the server on a new input.

Live Framed Source

The encoder is interacting with the server and this is done through an interface defined by the Live555 (FramedSource). After implementing this interface we got the FFmpegH264Source class. 

We are using a FramedSource interface because we only can get out of FFmpeg encoded single frames in form of packets. The api offered by the FramedSource interface is basically a way to start delivering frames by using doGetNextFrame, a way to stop getting them by using doStopGettingFrames and an extra call back function for the encoder to use when no new frames are in the queue and we started to get some to resignal the calls of doGetNextFrame.

Media Sub-session

The media sub-session is mainly describing the input and the output of the server when client connect to the server, In our case because we are using an unicast approach, we have only one sub-session that takes as source H264VideoStreamDiscreteFramer defined in createNewStreamSource. 

H264VideoStreamDiscreteFramer means that we will provide the RTSP server with discrete single frames in each call of GetFrame invoked by doGetNextFrame. Because FFmpeg by default needs to output the packets to be saved in a file, we creates a dummy video file at the beginning to fool FFmpeg then we ignore the writing part and direct the packets to the server buffer. The only issue here is that FFmpeg is adding extra information to the packet related to the location of the frame in the video file. If we take a packet extracted by FFmpeg we can see that is composed of a NALU. Each NALU has a start code of 4 bytes (in the case of x264 used by FFmpeg) marking the beginning of new frame in a video file. Because we are using H264VideoStreamDiscreteFramer it assumes the packets without the start code. So basically what we do in the encoder, we remove the first 4 bytes from the packet and then we enqueue it in the output queue before reaching the server buffer. The other aspect described by the Sub-session is the RTPSink that it will use to serve the client. We are using a H264VideoRTPSink which holds the all the encoded packets extracted from the encoder before being served to the client. Here you may need to resize the buffer of the sink before starting the server incase of 4k frames.

RTSP Server

This component is mainly initializing and linking any Live555 components together. It has a main loop that is called and get blocked at the end of it waiting for any clients. Here you may want to name your session or give an authentication credentials to it or even set the port you want to stream on if you don't want Live555 to choose them for you. And at the end of the server initialization we define the input source and sub-session we want to use. Here also you can change the Unicast to Multicast to server more clients for the same source. If everything is OK the server will print for you the URL you can stream on and then enter the eventLoop and block.

Main file

Here an instance of the decoder, encoder and server are created and initialized each instance is running on a separate thread. The decoder is running on the main thread. The playMedia function is called at the end of the main function and it will start by getting the frames from a file and send it to the encoder one by one. Then, the encoder will encode them and send them to server and there they will be ready for a client to stream them.

If there is any bugs please notify me I am still working on enhancements especially on performance

Ok that's it, if you have any questions feel free to ask !!

and here it is the git repo of the project:

https://github.com/MESAI/FFmpegRTSPServer