Dr. Shan Liu, Tencent Media Lab: The 5G Era is Coming, and Multimedia is Evolving Rapidly

From December 19th to 20th, 2020, Tencent hosted the 2020 Techo Park Developers Conference held in Beijing Fashion Design Plaza. At the main forum of the conference, Dr. Shan Liu, a distinguished scientist at Tencent and General Manager of Tencent Media Lab, gave a presentation on the theme "From Video Codec to Interactive Immersive Media", which focused on the evolution of video codec technology and Tencent's exploration in new and interactive immersive media.

Liu said in her speech that the worldwide outbreak has shifted the activities in many areas from offline to online, which placed a higher demand to maintain high-quality services while reducing the pressure on network data bandwidth due to increased use of multimedia technology. As an Internet technology company, Tencent has many of its businesses tightly integrated with multimedia technologies which makes video encoding and decoding expertise an important core competency for Tencent.

In multimedia technology, Tencent is continuously driving innovation breakthroughs. Since participating in the development of the H.266/VVC standard in 2018, Tencent Media Lab has contributed more than 100 technologies which have been adopted into the standard, ranking first in the world in number of adopted technologies. In November 2019, to actively promote the commercial use of AV1 standard, Tencent Cloud become the first cloud service provider in China to support AV1 transcoding. In October 2020, Tencent took the lead in releasing real-time H.266/VVC HD and Ultra HD players in China within 3 months of the H.266 standard completion.

In terms of new media and immersive interactive media, Liu believes that with the advent of the 5G era, the vast application scenarios and the commercialization of deep immersive media will be fully activated. The combination of immersive media technology with real-time media transmission technology can generate more immersive applications. At present, Tencent has launched a number of immersive media solutions to meet the development needs of different businesses.

At the end of the speech, Liu said Tencent will continue to actively invest in multimedia technology research and development, to embrace the construction of open-source ecology, and to provide effective services and support to developers and partners.

In the exhibition area, Tencent Media Lab's immersive media solutions were unveiled, attracting many audience members to experience VR360, point cloud and other 3DoF to 6DoF immersive media technologies and products. Off-site audiences can even log in to the official Techo applet to experience the real-time dynamics of the venue through 5G+VR360 technology and experience the exhibition site at any angle of 360 degrees.

The following is a transcript of Dr. Shan Liu's speech:

Hello, leaders, colleagues, distinguished guests. I'm Shan Liu, from Tencent Media Lab. I'm sorry I can't be there today, so I will be sharing with you in the form of video. The topic being shared today is "From Video Codec to Interactive Immersive Media".

At last year's Techo event, I spoke on the topic "Video Codec Technology and Application". So, the first major thing to share today, I'll share with you a brief introduction to some of the evolutions and iterations of video codec technology during the year. After that, I'll introduce Tencent Media Lab and Tencent Cloud's exploration and experiments in the direction of new and interactive immersive media.

This year has been a special and challenging year. The worldwide pandemic has brought great changes to our lives. Many activities in education, office, entertainment, sales and other fields have changed from offline to online due to the epidemic, and multimedia technology provides indispensable support for those cloud services. Since March, global video traffic has surged, and network bandwidth has been under great pressure. Regional disconnections caused by excessive network pressure has repeatedly occurred around the world. The European Commission has had to interview media service providers such as Netflix, demanding that the picture quality be reduced during peak periods and that valuable data bandwidth be reserved for home office and study. According to the survey results, the streaming media software market has accelerated by 19%, and reducing the pressure of network data bandwidth while maintaining high-quality services also puts forth higher requirements for multimedia technologies.

At the same time, people's demand for the quality of the video viewing experience continues to rise. Higher resolutions, higher brightness and color dynamic range, higher frame rates and other technical indicators, in conjunction with VR360, Freeview and other new media methods put forward higher requirements for data bandwidth. Internet traffic data show that in 2017, SD and HD video content accounted for about half each of the video bandwidth. By 2019, SD content share fell to about 1/3, HD content became the mainstream, and Ultra HD content share is climbing. It is expected that by 2022, the proportion of Ultra HD content will further increase to about 1/4 of the video bandwidth. At the same time, China's VR content market has continued to grow at a rate of 2-3 times per year since 2016, according to the China Industry Information Network. All of this makes video codec, or video compression, a technology field which has been around for decades, as important and relevant even today.

Tencent as an Internet technology company, with many of its businesses closely related to video as a media service, such as Tencent Video, Microscope, WeChat, QQ, Education, Pan-Entertainment as well as products such as Tencent Conference and Tencent Classroom, all of which have made great contributions to online education and telecommuting in this outbreak. Therefore, having a leading and efficient video codec technology is an important core capability that Tencent must have.

In a brief review, over the past three decades, many companies and research institutions around the world have invested enormous resources in the development of numerous video codec technologies and the formation of generations of video codec standards. Among the mainstream standards are international standards developed by ISO/IEC and ITU, such as the MPEG-2, H.264/AVC, H.265/HEVC and H.266/VVC. VVC, which was finalized in July of this year. Tencent began participating in the development of H.266/VVC standards since early 2018 and has submitted hundreds of technical proposals to the standards organization over a period of more than two years, about 100 technologies have been adopted by the standards, ranking Tencent first in the world in number.

At the same time, Tencent and other AOMedia member companies have been actively expanding the open-source ecosystem and promoting the commercialization of AV1 since Tencent officially joined AOMedia last year as the first Chinese company to join AOMedia's board of directors, and in November last year, Tencent Cloud became the first cloud service provider in China to support AV1 transcoding. In the formulation of the next generation of open-source video codec standard, namely AV2, Tencent contributed a large number of important proposals, including common test conditions, AV2 requirements documentation, and has contributed a large number of technologies covering block division, in-frame prediction, transformation, quantification, loop filtering and many other core technical areas, and the cumulative contribution to compression ratio puts Tencent to the forefront of the world. Tencent experts co-chair the Technology Incubation Group with Google experts in the AOMedia organization and the Software Implementation Working Group with Facebook experts.

We also invest heavily in the construction, promotion, and application of the national standard AVS. At present, Tencent Cloud supports AVS2 and AVS3 HD/Ultra HD real-time transcoding, and we have also made AVS privatization deployment for TV stations and other units.

In October this year, just three months after the first edition of the H.266/VVC standard was finalized, Tencent was the first in China to release an H.266 HD/Ultra HD real-time player. This player features Tencent self-developed H.266 software decoder, which can support real-time decoding of in HD, Ultra HD and screen content sharing and is an international leader in decoding efficiency.

What you're seeing now is playing the standard test sequence with our published H.266 player.

Each set of video codec standards requires a standard test sequence to test the performance of the proposed technology to help determine whether it should be adopted into the standard.

Tencent's iconic sequence “Honor of Kings”, which is also representative content of our business, is included in the VVC standard test sequence set.

This player is currently open source for developers.

Tencent Cloud actively participates in the construction of open-source communities, while also serving developers in various industries through Tencent Cloud's advanced technical capabilities. Tencent Cloud is designed not only for the domestic market, but also has a full layout and consideration for overseas business. For the overseas OTT market, Tencent Cloud has created a series of media service products to provide sufficient technical product support on an international scale to overseas developers and service providers. Media service series products can provide 8 common overseas streaming media protocols such as RTP/HLS/DASH, stable live broadcast service 24 hours a day, 7 days, supporting local deployment in more than 60 countries and regions.

With the rapid development of science and technology, people are no longer satisfied with just watching traditional two-dimensional video but are eager for a more authentic and immersive experience. Immersive media, through the integration of the physical and virtual worlds, is considered to be one of the disruptive trends that will change the way we live and work in the future. The immersive media content market represented by VR has grown significantly in recent years, with the rapid expansion of the application industry chain and an increasingly wide range of applications reaching industries such as travel, education, entertainment, medical care and manufacturing. It is estimated that the scale of the immersive media market will reach $161.1 billion USD by 2025. Therefore, more companies and manufacturers are also increasing their investment in immersive media technology R&D and production.

A deeper immersive experience is mainly reflected in the 6DOF of a real scene, clearer and smoother content display, multi-channel interaction, etc. This relies on VR, AR, point cloud, Freeview and other core technologies with traditional media formats such as pictures, videos, text, and sound combined with compression, transmission, display, interaction and other links are finally presented through different devices such as mobile phones, computers, head-mounted, large screens and more. From the perspective of immersive application scenarios, this widens the scope from a consumer market for personal entertainment to a vertical industry application scenario for enterprise market. In the future, with the large-scale adoption of 5G, the breakthroughs in immersive media hardware and technology, the decline in production costs and the development of more high-quality content will promote the adoption of immersive media products and services for mainstream groups and the application and business scenarios of deep immersive media will be fully realized.

A high-quality and efficient interactive immersive system includes a wide range of technical modules, from acquisition, processing, compression, transmission, to decompression, post-processing, rendering and interaction. It contains technologies such as projection, acquisition stitching, FOV, adaptive transmission, and more along with the transmission protocols such as HLS, DASH, RTC, etc. Because the amount of data in immersive media content is larger than traditional HD/Ultra HD video, such as the VR360 concert shown here, and the freeview basketball game, effectively combining all these technical modules and optimizing them is even more important and critical to providing a high-quality end-to-end experience.

Combining immersive media technology with real-time media transmission technology can produce more immersive applications. For example, traditional video conferencing can only use a single fixed lens, vision and interactivity have certain limitations. By incorporating immersive technology, you can create audio-visual effects of three and six degrees of freedom combined with virtual meeting room settings to provide participants with more comprehensive meeting information and a richer meeting experience.

Point cloud is another representative technology in the field of immersive media, and it has also received more attention in the past two years. Point cloud-to-end system includes point cloud data processing, compression, model reconstruction and rendering, interaction and other technical modules. Tencent's self-developed point cloud system can be rebuilt 3D objects and space through video, pictures, depth and other information, which can be used for business scenarios such as exhibitions and real estate. Because point cloud uses three-dimensional spatial lattice to express real objects and scenes, it can be imagined that the amount of data required to build a high-precision point cloud model is very huge. Therefore, point cloud data compression is also an indispensable link in point cloud system. Tencent multimedia experts actively participate in the formulation of international standards for point cloud compression and have technical proposals adopted by international standards. At the same time, they serve as the co-chair of the AVS point cloud thematic group.

Tencent Cloud has now launched a number of immersive media solutions, including VR video solutions, Tencent immersive solutions, etc., to meet the development needs of different businesses. Tencent's immersive solution can provide complete spatial modeling capabilities, and support the full platform display and sharing of H5 pages, Android, iOS mobile terminals and applets.

We have gradually entered the 5G era. The 5G network provides us with superior bandwidth and ultra low latency, making more applications within reach and making the interconnection of everything possible. Under the influence of 5G, the production, acquisition and dissemination of media content are all changing. Whether it is 4K/8K, or VR/AR/MR/point cloud, these applications that were limited by network bandwidth in the past may ushered in with breakthroughs driven by 5G. Tencent will continue to actively invest in multimedia technology research and development, embrace the construction of an open-source ecosystem, and provide effective services and support for developers and partners.

Thank you.