Artificial Intelligence and Machine Learning in the Media Sector
Fraunhofer FOKUS’ business unit Future Applications and Media (FAME) develops advanced media tech-related solutions. In order to further improve these technologies, concepts of Data Science, Artificial Intelligence and Machine Learning are being used more and more. The researchers and developers of FOKUS look into the different areas of these concepts and examine the applicability of the different media-related applications and technologies.
Table of contents:
- Usage Analytics & Needs Analysis
- Content Creation & Adding Meta-Data
- Content Encoding & Storage
- Offering & Discovery
- Content Delivery & Playback
3. A typical Data Science Project
The Media Life Cycle
The scientists of Fraunhofer FOKUS have focused on the use of artificial intelligence in the entire life cycle of media - from its creation to the evaluation of its use. In each of the steps, large amounts of data are produced which, if processed correctly, can massively optimize the quality or costs of the media.
The life cycle begins before the content is created with automated demand analyses and the identification of success factors in order to precisely meet the needs of the end consumer. In the actual creation phase, intelligent tools support production by recommending and suggesting improvements to the medium and its content. The necessary meta data can be generated both manually and automatically.
An important aspect also concerns the encoding settings, which, if chosen correctly, can save large amounts of data rate and thus also financial expenses. Once media have been created, they should of course be found. Semantic searches lead to a better findability and recommendation systems can support if the users do not know what they are looking for.
Then the technical delivery of the media takes place via adaptive technologies and the presentation of the media content on the end devices of the users. To ensure the best possible quality of experience, extensive metrics are collected and prepared for business stakeholders. The life cycle closes when this data is evaluated to improve existing and create new media.
Usage Analytics & Needs Analytics
An important aspect of AI for Media is to analyze media usage. Broadcasters require to know about the real audiences in order to select appropriate advertisements or to optimize the program curation. Operators want to understand where their services work well and if it shows undesired behavior under specific conditions. Website providers are evaluating user flows in order to optimize the appearance and design of the service. And educational staff is analyzing the learning activities of students in order to adapt lectures and media offerings to the learners’ needs.
The business unit Future Applications and Media researches and develops sophisticated business analytics tools based on Machine Learning for different stakeholders from the media industry: for publishers, content providers, broadcasters, metadata providers and even educational institutions.
Smart Audience Measurements
Especially for smaller TV channels and in audience weak slots (daytime, night) it is challenging to derive precise audience data from traditional household panels, which are often too small in their size. Based on the increasing market penetration of HbbTV enabled devices our solution provides precise data even for smaller TV channels.
The standard-based broadcast measurement solution satisfies the demand for a digital and real-time solution to measure linear broadcast on an international level to gather and compare key performance indicators with those of the Over-the-Top (OTT) and digital distribution. All tracking data is available in real-time and accessible via the web-based HbbTV Research Toolkit. Additionally, a HbbTV-based overlay app can be used to monitor the live performance of the channel within the broadcaster’s linear TV program. The Research Toolkit offers customizable visualizations for live and historical data, report templates, automatized reports, interfaces for editors and non-research specialists to easily review the performance of their program as well as an extensive research environment for professional TV and media researchers.
360° View Path Analysis and Prediction
Offering 360° videos follows completely different rules than conventional video solutions. Content creators still need to learn these new rules. Thus, they need to know how their videos are consumed in order to improve their products. The same holds true for marketing and advertising companies. Placing ad banners in the middle of the screen would surely ruin the VR experience.
The Smart View Path Prediction Solution shows how to identify the presumably most interesting fields of view in a 360° video at a given time. This information is used to optimize the rendering process, as well as to automatically adapt the end user’s field of view to the most important scene of an individual. Providing informed automatic camera movement is the biggest benefit for the viewer.
Social Network Analysis
One of the main challenges in recommendation systems is to solve the so-called Cold Start Problem. When new items or users are introduced, a typical recommender engine does not know anything about this element. Usually, new registered users need to enter manually demographic data, preferences or at least give feedback to some offerings in order to let the engine predict personalized recommendations.
For collaborative filtering, new items need a set of feedback data to be recommended in order to avoid random recommendations or even skipping over this item. Fraunhofer FOKUS introduces an approach that analyzes texts from third party service providers, such as social networks (Facebook, Twitter, LinkedIn, etc.) and review websites, in order to enrich the item and user data of recommender systems and thus allow an adequate personalization. Beyond direct user input from social networks like Facebook where users are able to like, share or rate items, the system analyzes opinions, in terms of sentiments, and extracted keywords as well as the preferences of friends in the social community.
Assisted Content Production
Editors spend a significant amount of time on unimportant tasks, which are a result of too complicated user interfaces or too much choice when searching for media to embed. The Fraunhofer FOKUS Smart Content Production tools enrich common editors with Machine Learning support in order to make their usage more efficient.
Asset databases are still accessible via search tools. However, they actively recommend media assets that are of interest for the currently created publication/article/news. Therefore, the article’s metadata is analyzed and matched asset metadata that worked well in similar situations. Moreover, the Smart Content Production tools also optimize the content of the media – for instance, suggests better working text orders for articles, automatically cuts video scenes with trained cutscene data of directors and even suggests eye-catchy headlines and keywords.
Per-Title/ Scene/ Chunk Encoding
Video streaming content differs in terms of complexity and requires title-specific encoding settings to achieve a certain visual quality. A classic "one-fits-all" encoding ladder ignores this characteristics and applies the same encoding settings to all video files. In the worst case this leads to a waste of bandwidth and storage, quality impairments, and a bad user experience. Our title-based encoding solution has the potential to significantly decrease the storage and delivery costs of video streams while improving the perceptual quality.
The FAMIUM Deep Encode tool leverages different machine learning techniques to avoid the computational heavy test encodes. Examples of algorithms in this area include decision trees (e. g. through Boosted Decision Trees) and Deep Neural Networks (through Recurrent Neural Networks or Deep Convolutional Networks). By constant validation and retraining of our models the accuracy and efficiency of the algorithms are improved with each input and the entire system learns on its own.
Offering & Discovery
Adaptive systems are designed to adapt its presentation layer, the offered content selection or the navigation support to the humans’ needs. This feature is called personalization, as each user interacts with an individual user interface. In order to understand the demands and preferences of users, data must be collected and analyzed that represents the users’ behaviors. Based on this information, offerings can be personalized for individuals.
Currently, most users are not able to overlook all offerings of a media web service at once – nor in a lifetime. So they have to find a considered selection of those items they want to buy or consume. A recommender system helps its users to decide on products or media they might be interested in. Therefore, a set of different approaches can be utilized to get predictions of the users' behaviors and preferences.
Fraunhofer FOKUS’ business unit FAME realized state-of-the-art techniques and developed novel high-quality algorithms, especially based on recommender system techniques, to personalize media web services.
In order to generate the best and most accurate recommendations, the recommendation system “TV Predictor” combines the best fitting algorithms in a hybrid switching, cascading or merging way. The usage of these algorithms depends on the user’s request:
- Find similar TV programs to the selected one by using common content-based filtering algorithms, such as the Euclidean Distance or the Cosine Similarity, or by using unsupervised learning algorithms, such as Association Rules
- Get program highlights for a specific time period based on the favorite programs of similar users (Pearson Correlation Coefficient) and predictions of program ratings (Slope One)
- Calculate a personalized program guide changing the channel automatically by using clustering to pre-select programs best fitting the user’s interests and rating predictions
- Overlay upcoming program recommendations while watching TV, based on recognized behavior patterns (calculated by a Support Vector Machine) to find user interests, such as genres and categories, favored actors, directors, and producers or even the preferred channels, weekdays or times to watch specific content
Multiscreen Recommendations and Advertisements
Based on various standards, the Fraunhofer FOKUS FAMIUM offers implementations for the key multiscreen features, especially those for smart selection of other screens and advertisement delivery. The FAMIUM Multiscreen Advertisement solution addresses the needs of broadcasters and content providers to select, schedule and distribute appropriate media content (on-demand or live) to multiple platforms easily. By supporting different standards, devices from both worlds, broadcast and broadband, can be reached from one Web-based management interface.
Adaptive Learning Technologies
In digital learning environments, analysis of students’ interactions with the learning objects provides important information about the students’ behavior. Thereby, the analysis of the usage data leads to a better understanding of the learning process and, thus, optimizes teaching and learning.
Fraunhofer FOKUS developed a Smart Learning Recommender (SLR) where students can keep track of their personal predicted knowledge level on different learning objects at every point in time and get personalized learning recommendations to overcome individual learning weaknesses. Besides content metadata, such as exam relevance, lecture times and pre-requisites, SLR takes different factors into account for each student and learning object, such as the user’s self-assessments, interactions with the content, performance in exercises, as well as individual forgetting curves and the learning progress of classmates. At the same time, teachers can make use of this data to get an overview of the students’ overall progress and so are aware of potential knowledge gaps.
Broadcast Quality Assurance
Digital service broadcast suffers from misconfigurations at the source and dynamically changing signal propagation conditions. The lack of feedback leads to undisclosed service degradation and decreased experience. Our Broadcast Probing System offers cloud-based continuous near real-time monitoring of broadcast networks by utilizing massively distributed low-cost probes. Controlled either individually or in groups, the probes are securely instructed to execute scheduled jobs like scanning, tuning and transport stream inspection.
Intelligent Video Analytics
SAND metric reporting enables video players (e.g., for MPEG-DASH) to provide streaming performance information like average throughput, buffer level, representation switch events and initial playout delay (QoE metrics defined in ISO/IEC 23009-1). SAND shared resource allocation allows network components to control how much bandwidth a client should use. This is useful in scenarios in which multiple DASH clients share the same network and compete for the available bandwidth (e.g. in a stadium, train or airplane). Artificial Intelligent algorithms support technical staff to analyze the vast amount of mined data, observe video playback patterns and notify about outliers and anomalies of particular clients.
Data is the new Currency
In order to improve web and media services data is being collected from the presented services and products as well as from the users and their way of conduct. The gained information is crucial for the general understanding of the usage, for business analytics, for the improvement and optimization of the offer as well as for the personalization and individualization of the services. Without the knowledge about users and behaviors, current media business would not survive. However, humans are not capable anymore to process and analyze all the gained data manually. Therefore, software components may assist the human stakeholders, give hints and suggestions for improvements and even make decisions (semi-)automatically.
A set of different concepts and terms arose in the last few years: Big Data, Data Science, Artificial Intelligence, Recommender Systems, Machine Learning, Artificial Neuronal Networks and Deep Learning – just to mention a few of them. While the meaning of the term “Big” in Big Data changes with the advance of time and technology, the other terms have, more or less, strict definitions: Data Science is an academic discipline that processes data in order to extract knowledge. Artificial Intelligence, in contrast, describes the concepts and methods of computer systems that perform tasks of humans or to imitate intelligent human behavior. Machine Learning is a sub-class of Artificial Intelligence that improves its operations over time by learning from the data. Artificial Neural Networks and Deep Learning are specialized methods within the Machine Learning discipline.
A typical Data Science Project
The most important aspect of Data Science is the definition of valuable research questions that build the base for further analysis of the data. The Fraunhofer data scientists help to understand the core questions and assist in transferring business requirements into technical approaches. When customers already collected a massive set of data, the next steps would be sighting, integration, cleaning, structuring, selecting and normalizing the given data.
On the prepared data sets, different algorithms can be applied to perform, for example, regression (e.g., adding new data for forecasts) or classification tasks (to group the already collected data points). Therefore, the researchers developed techniques for pattern recognition and prediction tasks through Supervised Learning and Unsupervised Learning – among others, based on decision trees, clustering, content-based and collaborative filtering, association rules, support vector machines, (deep) artificial neural networks and many more. These methods are used to recognize and learn laws in the data and then apply them to unknown situations.
Finally, the numerical results and algorithmic conclusions need to be presented to the various stakeholders. Therefore, the data is firstly reprocessed, simplified and explained and then visualized to present only those results that appropriately answer the research questions and bring benefits for the persons of interest in the given situation.
Fraunhofer FOKUS’ AI for Media research team has lots of experience in realizing data-driven projects for different industry companies and typically realizes end-to-end solutions or focuses on particular data science topics.