CognitiveSkip™

Product and solution overview

CognitiveSkip™ is a Cognitive Mill product that offers you a unique solution for safely skipping unessential parts of your video content like opening and closing credits, ads, etc. It enables smart EPG navigation, its automated correction, and TV ads skipping and replacement in an EPG.

Let’s review one of the CognitiveSkip™ solutions — End Credits Skip Automation — that analyzes video content and marks the so-called Safe skip zone that can be skipped without missing anything essential.

The End Credit Skip Automation solution accurately identifies where the credits start and end and you get just one end credits detection timestamp to make an informed decision.

Benefits:

You get the Safe point, after which you can safely skip the credits.
You won’t miss post- and in-credit scenes, anything important like extra scenes, bloopers, or creatively designed credits that can be a part of the story.
Such credits will be identified as standout credits and won’t be marked for safe skipping so that you don’t miss anything important and decide yourself to skip them or not.
You can interact with the platform either via the UI or via the API.
You paste the video URL and hit the button — the rest is done automatically.
You get a Skip meta JSON file to process it with third-party systems and software.

Output metadata

The JSON file you get after running a skip meta process is essential for End Credits Skip Automation. It contains metadata for internal use and the following data:

Events. Display the key timestamps in the segments.
Segments. Display the start time and end time of the content item and its type.

Terminology

Safe skip is a segment in the video timeline that can be skipped without the risk of missing something important. It is also known as Safe skip zone or just Skip zone.
Safe point is the final timestamp after which nothing essential can take place, the following content, if any, can be safely skipped or even cut. This is the point when you can switch your audience to the next movie, series, TV show, etc.
Closing credits are a long sheet of text with credits only, without anything belonging to the main action. It can be safely skipped without missing the essential.
Closing credits can be included in the Safe skip segment, or the beginning of closing credits can be the Safe point if nothing important follows them.
Standout end credits are the credits that are either creatively designed or include anything important like extra scenes, bloopers, or appear while the action in the background still goes on. We identify this type of credits and mark it. It is never included in the Safe skip so that you don’t miss anything important.
Post-credits are movie companies’ logos usually displayed after closing credits. Though they get into the Safe skip zone, we identify and mark separately for your convenience.

Video timeline structure

Here's an example of a movie timeline structure with standout end credits and closing credits followed by a post-credits scene and see how it is shown in a meta.json file.

Above is the timeline structure of a movie that includes:

Standout end credits. Credits that are either creatively designed or include anything important like extra scenes, bloopers, etc. We identify this type of credits and mark it. It is never included in the Safe skip so that you don’t miss anything important.

Closing credits. The long sheet of text with credits that can be safely skipped without missing anything essential. This is the Safe skip in this example but they do not always coincide.

Post-credits scene. A scene that follows the closing credits and belongs to the action part. The system identifies it as an action so we guarantee that nothing important will be missed.

Safe point. The final timestamp after which nothing essential can take place, and you can switch your audience to the next movie, series, TV show, etc.

Skip meta JSON file

Here's the JSON file that contains metadata of the movie with the above-described pattern.

Type. Indicates the type of meta, for example, skip meta.

Version. Displays the output format version of the JSON file.

Events. Display the key timestamps in the segments.

For Events, the following data is displayed:

Ms and time. The key points in the video timeline for automatic skipping.
Type. The type of events can be the following: closing credits, standout end credits, post credits, and safe point.
Id. The entity name.
Type. The entity format type.

Segments. Display the start time and end time of the content item and its type.

For Segments, the following data is displayed:

End. The end time of the segment. Ms stands for milliseconds, and time shows when the episode appears in the video timeline.
Start. The start time of the segment. Ms stands for milliseconds, and time shows when the episode appears in the video timeline.
Type. The type of segments can be the following: closing credits, standout end credits, post credits, and safe skip.

Id. The entity name.

Type. The entity format type.

Metadata. Contains the information about the modules and versions for internal use.


  {
    "type": "skip meta",
    "version": "1.0.0.0",
    "data": [
    {
        "events": [
        {
            "ms": 5400000,
            "time": "1:30:00",
            "type": "standout end credits"
          },
          {
            "ms": 5450000,
            "time": "1:30:50",
            "type": "closing credits"
          },
          {
            "ms": 5630000,
            "time": "1:34:20",
            "type": "safe point"
          }
        ],
        "id": "credits events",
        "type": "time events"
      },
      {
        "id": "credits segments",
        "segments": [{
            "end": {
              "ms": 5450000,
              "time": "1:30:50"
            },
            "start": {
              "ms": 5400000,
              "time": "1:30:00"
            },
            "type": "standout end credits"
          },
          {
            "end": {
              "ms": 5570000,
              "time": "1:32:50"
            },
            "start": {
              "ms": 5450000,
              "time": "1:30:50"
            },
            "type": "closing credits"
          },
          {
            "end": {
              "ms": 5570000,
              "time": "1:32:50"
            },
            "start": {
              "ms": 5450000,
              "time": "1:30:50"
            },
            "type": "safe skip"
          }
        ],
        "type": "time segments"
      }
    ],
    "metadata": {
      ...
    }
  }

Segments in the example above include end and start time for two content types:

Standout end credits that we identify but leave the final decision to skip them or not to you.
Closing credits are the credits with nothing to miss except for the long sheet of credits text, for which we guarantee that you can skip them.
Safe skip is a part that can be skipped without the risk of missing something important.
In our example, the Closing credits and Safe skip belong to the same segment.

Events in the example above include timestamps for three content types:

Standout end credits. This timestamp marks the start time of the standout end credits.
Closing credits. This timestamp marks the start time of the closing credits, the beginning of the Safe skip zone, in our case.
Safe point. This timestamp marks the end of the post-credits scene, and the following content, if any, can be safely skipped or even cut.

Other cases

1. A Safe point can also be at the beginning of Closing credits if nothing essential comes after them, as shown in the timeline structure below.

After you process a video that has such structure, you get a result.json file similar to the one displayed below.


  {
    "type": "skip meta",
    "version": "1.0.0.0",
    "data": [
      {
        "events": [
          {
            "ms": 527600,
            "time": "0:08:47.600000",
            "type": "closing credits"
          },
          {
            "ms": 527600,
            "time": "0:08:47.600000",
            "type": "safe point"
          }
        ],
        "id": "credits events",
        "type": "time events"
      },
      {
        "id": "credits segments",
        "segments": [
          {
            "end": {
              "ms": 898880,
              "time": "0:14:58.880000"
            },
            "start": {
              "ms": 527600,
              "time": "0:08:47.600000"
            },
            "type": "closing credits"
          }
        ],
        "type": "time segments"
      }
    ],
    "metadata": {...}
    }

2. In some cases, a Safe skip zone may not coincide with Closing credits.
It happens, for example, when Closing credits start, then get interrupted by an in-credits scene, then they continue.

The system cannot break up the Safe skip zone into parts. There can be only one Safe skip per video. The In-credits scene gets identified as an action that cannot be skipped, so the Safe skip starts after the In-credits scene ends.


  {
    "type": "skip meta",
    "version": "1.0.0.0",
    "data": [
    {
        "events": [
        {
          "ms": 5400000,
          "time": "1:30:00",
          "type": "standout end credits"
        },
        {
          "ms": 5450000,
          "time": "1:30:50",
          "type": "closing credits"
        },
        {
          "ms": 5630000,
          "time": "1:34:20",
          "type": "safe point"
        }
      ]
    },
        "id": "credits events",
        "type": "time events"
      },
      {
        "id": "credits segments",
        "segments": [
        {
          "end": {
            "ms": 5450000,
            "time": "1:30:50"
          },
          "start": {
            "ms": 5400000,
            "time": "1:30:00"
          },
          "type": "standout end credits"
        },
        {
          "end": {
            "ms": 5570000,
            "time": "1:32:50"
          },
          "start": {
            "ms": 5450000,
            "time": "1:30:50"
          },
          "type": "closing credits"
        },
        {
          "end": {
            "ms": 5570000,
            "time": "1:32:50"
          },
          "start": {
            "ms": 5500000,
            "time": "1:31:40"
          },
          "type": "safe skip"
        }
      ],
        "type": "time segments"
      }
    ],
    "metadata": {
      ...
    }
  }

The system is smart enough to analyze the context and recognize video content patterns. But to avoid extra information, the generated result.json file contains only the core data essential for solving your business tasks — the Safe skip timeframe and the Safe point timestamp.

Demo case

This guide shows you how to run the skip meta process to get a result.json file via the UI.

Before running a skip meta process:

1. Sign in to your account at run.cognitivemill.com or register if it’s your first visit.

2. Check that you have the skip meta quota to run the process. If you don't have it, contact us at support@aihunters.com to get it.

You are all set and now you can run the process.

1. Click the Run Process button on the top navigation bar.

The Run a New Process page opens.

2. Select skip meta old from the Process type drop-down list.
Note that only the types of processes you have quotas for are available here.

3. In the Title field, enter a name for your process.

4. In the File source field, either paste a link to the video you want to process — the default File link option — or click the field > select Upload file > click Select file > pick the file from your device to upload.

5. (Optionally) Enter the Start time and End time (in milliseconds) to specify the time interval that the system should analyze.

6. Hit Run Process.
Your newly created process appears at the top of the Process List in the current status.

When the status changes to Completed, you can download metadata.

7. On the Process List page, click the three vertical dots icon next to the process.

8. In the pop-up menu that appears, click Download result.
The result.json file is downloaded to your device.

You can also view how it works in the Cognitive Mill visualizer.
On the Processes page, click the title of the process to open the video in the visualizer.

Here you can see the Safe point, after which the closing credits start and safe skipping is guaranteed.

Current challenges

One Safe skip per video.
Only one Safe skip zone can be identified in a video now. If Closing credits are broken up into two or more segments by mid-credits scenes and followed by a post-credits scene, only the last part of such split Closing credits will be included in the Safe skip zone. For a more detailed example, view Case 2.
Standout end credits.
Standout end credits are sometimes hard to identify because of the variety of styles and forms they can take. In complicated cases, when Standout end credits cannot be accurately recognized, the system doesn’t include them in the skip meta JSON file. It doesn’t affect the End credits skip automation but helps to prevent skipping something important.
Animated logos.
Movie studio animated logos that sometimes appear at the end of closing credits may also be confusing for the system and mistakenly recognized as the main content. In this case, the system will put the Safe point marker after the animated logo, and the Safe skip zone will be before the logo, covering the closing credits.
Indistinct credits.
In some cases, when the action still goes on, or the movie scenes change, the appearing credits are indistinct and not clearly seen. The system may ignore such credits and not include them in the meta.json file.

We are working hard on the most efficient ways how to overcome the current challenges so that in the following versions of CognitiveSkip™ we won’t face them anymore.