Design and Analysis of Multi-picture Software in Video Conference System
Design and Analysis of Multi-picture Software in Video Conference System
0 Introduction The development of video conferencing has gone through a process from scratch to immaturity to basic maturity. In this development process, various remote conference systems such as video conferences, desktop conferences, and multimedia conferences have emerged one after another. One of the key technologies for the development of video conference systems is video coding. Using the low bit rate video codec scheme H.263 recommends that the video image can be compressed to at least about 20 kbps, so that it can transmit audio signals through the 28.8 kbps V.34Modem on the ordinary telephone line. In fact, while the audio and video coding protocol is continuously improved and developed, the protocol of the video conference itself has also been updated, and H.323 recommendations based on packet switching networks have emerged. Another key technology for the development of video conferencing towards high performance, multi-function, collaboration, intelligence and integration is multi-screen synthesis technology.
The traditional method of storing data streams in RGB format for multi-picture synthesis is a waste of resources. Therefore, in order to maximize the utilization rate of resources under the condition of limited frequency band, this paper proposes a new solution to solve the problem of resource utilization without affecting the picture quality. In fact. Video conference multi-picture software has been applied to many fields of life. And it will be more widely used.
1 Data flow capture
The structure of capturing data packets under Windows is shown in Figure 1. The NDIS (Network Driver Interface SpecificaTIon) of NDIS is a network driver specification jointly developed by Microsoft and 3Com. This specification provides a large number of operating functions. It can provide services for protocol drivers at the upper layer, thereby shielding the differences of various network cards at the lower layer. NDIS upward can support multiple network protocols, such as TCP / IP, NWLink IPX / SPX, NETBEUI, etc., downward can support multiple network cards produced by different manufacturers. 
The design steps of using winpcap to capture and filter network packets are as follows:
(1) Open the network card and set the network card to promiscuous mode;
(2) The callback function Network Tap collects data packets from the network device driver after obtaining the monitoring command, and transmits the monitored data packets to the filtering program;
(3) When Packet filter detects that a data packet arrives, the NDIS intermediate driver calls the packet driver, which can pass the data to each packet filtering program participating in the process;
(4) The Packet filter filter program decides to discard or receive data packets, and copies the received data to the corresponding program;
(5) After passing the packet filter, submit the unfiltered data packet to the core buffer, and when the system buffer is full, copy the data packet to the user buffer. The listener can read the captured data packets directly from the user buffer;
(6) Turn off the network card.
2 The video data stream captured by the decoding of the video data stream. Usually H.263 video coding recommendations are adopted in network transmission, and the image format is CIF format. The data stream in H.263-CIF format is a compression-encoded image format. This format can save frequency band resources during network transmission and maximize resource utilization under limited bandwidth. However, the CIF format must be processed before it can be applied to multi-picture synthesis. Only after decoding to make the captured video data stream into YUV (RGB) format, this image format can be directly used for multi-screen synthesis, and the image synthesis in this format is also more convenient and fast. In order to change the image from H.263 format to YUV (RGB) format, H.263 decoder should be used for decoding. The schematic diagram of its codec algorithm is shown in Figure 2. 
3 After decoding the multi-picture composite video data stream, it can be converted into YUV (RGB) format. The next link is the synthesis of video data streams (multi-picture synthesis), which is also the key technology of this article. In a video conference system, there may be more than two or three participants, and all participants may be distributed throughout the country or even around the world. In order for each participant to have direct face-to-face communication, it is necessary to see all participants at the same time. Therefore, multi-picture synthesis technology is essential. Through multi-picture synthesis, N different video pictures can be displayed on the same monitor in real time, so that each participant can use the software to see other members.
The main sampling formats of YUV are YCbCr4: 2: O, YCbCr4: 2: 2, YCbCr4: 1: 1 and YCbCr4: 4. : 4. Among them, YCbCr4: 1: 1 is more commonly used, which means that each point saves an 8-bit brightness value (that is, Y value), and every 2x2 points saves a Cr and Cb value, so that the image will not be in human vision Changes too much. The 4 points of the RGB (R, G, B are all 8 bit unsigned) models require 8 × 3 = 24 bits, but now only 8+ (8ï¼4) + (8ï¼4) = 12 bits are required, average each Dots occupy 12 bits. This compresses the image data to half. YUV411 extracts the Y component for each pixel, and the UV component is sampled every 4 pixels in the horizontal direction. YUV420 does not sample the V component to 0, but only doubles the color difference sampling frequency in the horizontal direction compared to YUV411, and reduces the color difference sampling by half in the U / V interval in the vertical direction.
The data stream in YUV format can be obtained after decoding. The number of brightness sampling pixels and the number of brightness sampling lines of H.263-CIF image format are 352x288. The image is composed of many pixels, that is, an H.263-CIF image Occupies 352 × 288 pixels, after decoding. These values ​​will not change, that is, the number of brightness sampling pixels and the number of brightness sampling lines in the YUV format are also 352 × 288, and also occupy 352 × 288 pixels. Now assume that the video data is in YUV420 format. Its sampling format is 4: 2: 0. When storing video data in YUV420 format, you can store all Y first, then U, and then V according to the storage format shown in Figure 3, and each pixel occupies one Y, averaging four pixels A U and V. Each pixel occupies one byte, then the number of bytes occupied by an image will be 352 × 288 × (1 + 1, 4 + 1/4). 
The following uses four pictures as an example to introduce the multi-picture synthesis process of the software. After the captured four video data streams are decoded. All data of each image is continuously stored in the storage unit. Here I, II, III, and IV are used to represent these four images, and their storage status in the storage unit is shown in FIG. 4. Each image is stored separately. The actual four-in-one image needed to be obtained is shown in Figure 5. 
When the image is displayed, all pixels are arranged in order. For example, a 352 × 288 image, when storing pixels, first store the 352 pixels of the Y signal line l, followed by the second line ... Until the 288th line stores the Y signal, then stores all the pixels of the U signal from the 1st to 288th line, and finally stores all the pixels of the V signal. But after the four images are combined, the size of the image will become twice that of an image, that is, 704 × 576, so the storage format needs to be changed. That is to readjust the pixel storage order of the four images, the specific process is as follows:
(1) Store the 352 pixels of the first line of the Y signal of the first image, and then store the 352 pixels of the second line of the Y signal of the second image. A total of 704 pixels:
(2) Store lines 2 to 288 of the Y signal of the new image in this order;
(3) Line 289 of the new image consists of 352 pixels in line 1 of the Y signal of the third image plus 352 pixels in line 1 of the Y signal of the fourth image;
(4) The Y signals of lines 290 to 576 of the new image are also stored in this order until all the Y signals of the multi-screen composite image are stored;
(5) From step (1) to step (4) again to complete the storage of U and V signals.
After the above five steps, the new multi-screen image is stored according to the designed format. The display of the multi-screen can be achieved by using Microsoft's DirectShow, so that the multi-screen software system design of the video conference can be completed.
4 Conclusion In the analysis and design process of the multi-picture software of the video conference system, the video data stream can be captured first and used for multi-picture synthesis after decoding. Multi-picture synthesis is a key part of the video conference system. Data capture and decoding are the basis for this process. This article uses four-picture synthesis as an example to introduce the multi-picture synthesis process.
In video conferences, the use of multi-picture software can better conduct real-time dialogues and improve conference efficiency. The multi-picture software introduced in this article can be applied not only to video conferences, but also to other occasions. For example, in the screen synthesis, the software can be used to change the storage format of the data slightly to obtain the picture-in-picture effect of the TV screen; using the multi-screen synthesis technology in the common monitoring system can enable multiple monitors to capture Images are displayed on one screen at the same time. Video conference multi-screen software can not only be widely used in conferences, but also play an active role in complex command and control occasions such as military, transportation, and public security.
splicing screen, large display,large lcd display, LCD screen
Shenzhen Risingstar Outdoor High Light LCD Co.Ltd , https://www.risingstarlcd.com