Welcome on Planet VideoLAN. This page gathers the blogs and feeds of VideoLAN's developers and contributors. As such, it doesn't necessarly represent the opinion of all the developers, the VideoLAN project, ...
If you follow this blog, you should know everything about AV1.
AV1 is a new video codec by the Alliance for Open Media, composed of most of the important Web companies (Google, Facebook, Netflix, Amazon, Microsoft, Mozilla...) and VideoLAN.
AV1 has the potential to be up to 20% better than the HEVC codec, but the patents' license is totally free, while HEVC patents licenses are insanely high and very confusing.
This is the first time, where the Open community
is ahead of the MPEG community
, notably because AV1 and Opus are both great codecs.
AV1 has mappings to wrap it inside MP4 or MKV. And other mappings are coming, notably for RTP or TS.
So, of course, the open source community has developed tools to support AV1. This post is about how to use those tools.
For FFmpeg, integration with libaom was done for both encoding and decoding (and now also dav1d for decoding).
To encode, it is important to activate the --enable-libaom
option at ./configure
time.
You can get all the options for encoding by using ffmpeg -h encoder=libaom-av1
.
To encode any file that is played by ffmpeg, just use the -c:v libaom-av1
option:
ffmpeg -i input.mp4 -c:v libaom-av1 -crf 30 -b:v 0 -strict experimental av1_test.mkv
This works, of course, for the mp4 and mkv output formats.
For 2-pass encoding, use the usual commands, but with the -c:v libaom-av1
:
ffmpeg -i input.mp4 -c:v libaom-av1 -strict experimental -b:v 2M -pass 1 -an -f matroska /dev/null && ffmpeg -i input.mp4 -c:v libaom-av1 -strict experimental -b:v 2M -pass 2 -c:a libopus output.mkv
To know more about AV1 in FFmpeg, please use the help or the official documentation.
For GPAC, the integration of libaom was done too, and it is quite simple.
For example, to add an av1 stream inside an MP4, just use MP4Box:
MP4Box -add file.av1 file_av1.mp4
And, you can even prepare those AV1/MP4 files for DASH streaming:
MP4Box -dash 1000 -profile onDemand file_av1.mp4
If you want more details, or try encrypting of those streams, please read the GPAC blog.
Gstreamer made several releases supporting the AV1 plugins.
To play an MP4 AV1 file, just use gst-play-1.0 av1.mp4
.
To do a simple encode and mux it in MP4:
gst-launch-1.0 videotestsrc num-buffers=300 ! video/x-raw, framerate=30/1, width=320, height=240 ! av1enc ! mp4mux ! filesink location=av1file.mp4
Or, if you want to transmux from MKV to MP4, or vice-versa:
gst-launch-1.0 filesrc location=av1.mkv ! matroskademux ! mp4mux ! filesink location=av1.mp4
gst-launch-1.0 filesrc location=av1.mp4 ! qtdemux ! matroskamux ! filesink location=av1.mkv
Finally, to transcode to AV1 and mux in MP4:
gst-launch-1.0 uridecodebin uri=file:///home/toto/file.avi ! av1enc ! mp4mux ! location=video-av1.mp4
Of course, VLC has full decoding integration, with libaom, and with dav1d (starting in 3.0.5, in a few days). This will work on all platforms, starting with desktop releases first.
But VLC 4.0 also has full encoding and muxing, in both MP4 and MKV, in the nightly builds. To use, you can try this:
vlc file.avi --sout "#transcode{vcodec=av01}:std{access=file,mux=mkv,dst='output.mkv'}"
It's quite important to mention that Mediainfo already supports AV1, since version 18.08.
You can use the GUI, or the CLI: mediainfo av1.mkv
.
Last, but not least, MKVtoolnix, supports AV1 muxing, since v28.0.0.
Please try those tools, to create and play AV1/OPUS files everywhere.
Also, please report any bug you would find in those tools.
If you want a quick summary of this post, about our AV1 decoder:
Read the following for more details...
If you follow this blog, you should know everything about dav1d.
AV1 is a new video codec by the Alliance for Open Media, composed of most of the important Web companies (Google, Facebook, Netflix, Amazon, Microsoft, Mozilla...).
AV1 has the potential to be up to 20% better than the HEVC codec, but the patents license is totally free, while HEVC patents licenses are insanely high and very confusing.
The reference decoder for AV1 is great, but it's a research codebase, so it has a lot to improve.
Therefore, the VideoLAN, VLC and FFmpeg communities have started to work on a new decoder, sponsored by the Alliance of Open Media, in order to create the reference optimized decoder for AV1.
Today, we release the first usable version of dav1d, called 0.1.0, Gazelle
.
It means you can use the API, ship the decoder, and expect a bit of support on it.
We launched dav1d, more than 2 months ago, during VDD, and we've been working a lot since:
On modern desktop, dav1d is very fast, compared to other decoders:
You can see more details on my previous post.
But, since the previous blogpost, we've added more assembly for desktop, and we've merged some assembly for ARMv8, and for older machines (SSSE3).
We're now as fast as libaom, in single-thread, on ARMv8, and faster with more threads.
We've been also merging more SSSE3 code. (I haven't had enough time to bench it).
Which means that we will soon be faster than other decoders, on all platforms.
And, we've been experimenting with shaders, notably for the Film Grain feature.
You can get the tarball on our FTP: dav1d 0.1.0.
You can get the code and report issues on our gitlab project.
You can also join the project, or sponsor our work, by contacting me
If you want a quick summary of this post:
Now is the right time to integrate it, in your products!
Read the following for more details...
AV1 is a new video codec by the Alliance for Open Media, composed of most of the important Web companies (Google, Facebook, Netflix, Amazon, Microsoft, Mozilla...).
AV1 has the potential to be up to 20% better than the HEVC codec, but the patents license is totally free, while HEVC patents licenses are insanely high and very confusing.
The reference decoder for AV1 is great, but it's a research codebase, so it has a lot to improve.
Therefore, the VideoLAN, VLC and FFmpeg communities have started to work on a new decoder, sponsored by the Alliance of Open Media, in order to create the reference optimized decoder for AV1.
We launched dav1d, exactly 2 months ago, during VDD.
We did a lot of work since. And by "we", I mean mostly the others.
There are now more than 500 commits from 29 contributors from different open source communities. This is a good result for a new open source project.
First, we've completed all the features, including Film Grain, Super-Res, Scaled References, and other more obscure features of the bitstream. This covers both 8 and 10bits, of course.
We also improved the public API.
Then, we've fuzzed the decoder a lot: we are now above 99% of functions covered, and 97% of lines covered on OSS-FUZZ; and we usually fix all the issues in a couple of days. This should assure you a secure decoding for AV1.
Finally, we've written a lot of assembly, mostly for modern desktop CPUs, but the work has been started for mobile and older desktop CPUs.
We even reduced the size of the C code!
Today, dav1d is very fast on AVX2 processors, which should cover a bit more than 50% of the CPUs used on the desktop. We wrote 95% of the code needed for AVX2, but there is still a bit more achievable.
We're readying the SSE and the ARM optimizations, to do the same. They will be very fast too, in the next weeks.
The following graphs are comparing dav1d and aomdec top-of-the-tree on master branches. (and yes, aomdec has CONFIG_LOWBITDEPTH=1).
This was done on Windows 10 64bits, using precompiled binaries.
The clips are taken from Netflix, Elecard, and Youtube, because they don't use the same parameters in the encoder, and don't have the same bitstream features.
Film Grain is not run on the CPU, so it is not visible here.
Here, on Haswell (i7-4710, a 4 year old CPU with 4 cores), are the results:
And reported to in percentage compared to libaom:
We got in average 2.49x, and we even get 3.48x on the Youtube Summer clip!
With a more modern Zen machine (Ryzen 5 1600, 6 cores HT), here are the results:
And reported to in percentage compared to libaom:
The average is even higher at 3.49x, and we even get 5.27x on the Youtube Summer clip!
If we put both on the same graphs, here is what we have:
If you listened to our talks during VDD or during demuxed, we explained that dav1d threading was quite innovative, and should scale way better than libaom.
On an even less powerful machine, an i5-4590, with 4 cores/4 threads, here are our results, for the Youtube Summer clip:
You see that dav1d can scale better, in terms of threading, than libaom.
dav1d is very fast, dav1d is almost complete, dav1d is cool.
We're finishing the rough edges for a release soon, so that we can hope that Firefox 65 will ship with dav1d for AV1 decoding.
On the other platforms, SSE and ARM assembly will follow very quickly, and we're already as fast on ARMv8. Stay tuned for more!
I would like to thank Ewout ter Hoeven (EwoutH) from the community who did all the testing, numbers and computations.
I'm very proud to present to you the VLC Technical Committee, as elected during the last VDD conference: Denis Charmet, Rémi Denis-Courmont, Hugo Beauzée-Luyssen, Thomas Guillem and David Fuhrmann.
The role of the VLC Technical Committee (TC), is mostly a technical resolution committee, that will arise and decide when there are disagreements and bike-shedding in our community.
The glorious members of this Technical Committee are:
As you can see, in all fairness:
there are 2 people of the board in the TC, 2 out of 5 are VideoLabs employees, no roots are part of the TC, nor am I. They know about C, C++, obj-C, and Linux, Windows, macOS.
The fine prints of this Technical Committee:
May they do good work! Good luck to them!
Have you ever wondered what was the difference between the -fpic and -fPIC compiler command line flags were?
AV1 is a new video codec by the Alliance for Open Media, composed of most of the important Web companies (Google, Facebook, Netflix, Amazon, Microsoft, Mozilla...).
AV1 has the potential to be up to 20% better than the HEVC codec, but the patents license is totally free, while HEVC patents licenses are insanely high and very confusing.
The reference decoder for AV1 is great, but it's a research codebase, so it has a lot to improve.
Therefore, the VideoLAN, VLC and FFmpeg communities have started to work on a new decoder, sponsored by the Alliance of Open Media.
The goal of this new decoder is:
Without further ado, the code: https://code.videolan.org/videolan/dav1d
dav1d is called dav1d, because Dav1d is an AV1 Decoder
(Yes, that is a recursive acronym, no need to tell us...)
You can see a talk during VDD 2018 about dav1d:
Some technical details about dav1d:
Currently the source code of dav1d is 1/10th of lines of code compared to libaom and its weight is 1/3rd of the binary size of libaom.
It currently uses 1/4th of the memory usage of libaom and uses a very limited amount of stack.
Depending on the threads conditions (see the video talk linked above), dav1d is more or less faster than libaom 1.0.0, but slower than libaom HEAD.
dav1d having almost no assembly code yet, this is not surprising, and is actually a good starting point for the future.
Of course, those metrics will evolve once we add more assembly code, and when the project evolves a bit more.
Not yet, but you can start testing it and check how the API works for you.
Yes! We need C, ASM developers, but also app integrators and testers to give us feedback.
Yes. dav1d is licensed under BSD for this very reason.
Please talk to us, if you need to get adaptations for your use-case (hybrid decoders, or specific platforms, for example).
We want AV1 to be as popular as possible. This requires fast decoders, running everywhere. Therefore, we want to help everyone, even non-open-source software.
See RMS opinion on this subject.
After a few months since the release of VLC 3.0, today we release VLC 3.1.0 on 2 mobile OSes: iOS and Windows Store (UWP).
This release brings ChromeCast integration to iOS and UWP, like it was present on desktop and Android versions.
However, it supports ChromeCast in a more performant way, because we added hardware encoders to those 2 platforms.
Indeed, here, for local streaming, we care more about speed and battery saving than we care about bandwidth efficiency, si hardware encoding is a good fit.
On iOS, we're using the standard VideoToolbox hardware encoding to produce H.264 streams, muxed in MKV.
On UWP, we're using Quick Sync Video for intel CPUs (that covers almost all CPUs since 3rd core generation).
In fact, VLC has a QSV encoder since 2013, but it's very rarely used, because people usually prefer software encode (x264). Here, we fixed it and modified it to work inside the UWP sandbox.
You should really read Caro's blogpost here!
But in that version you have:
The version is similar to the iOS version, in the fact that it has hardware encoding and ChromeCast integration.
As explained, the hardware encoding is done using QSV.
But it features also a large rework of the codebase and fixes a very large number of crashes.
Also, funnily enough, we've worked on the 8.1 version too, and we will push that one soon on the store. This includes SurfaceRT devices, even if Microsoft has forgotten them!
So VLC 3.1.0, UWP version will be out for:
Once we fixed an issue, we might even do Windows Phone 8.1.
The Windows 10 versions are on the store today, and we're waiting for a deployment issue to be fixed to push the 8.1 versions!
(Note: if you are from Windows Central, you can contact me for more details)
Have fun!
After quite a bit of time far from the blog, I am back around here.
The biggest reason for this silence was that this was taking a lot of my time, but I had almost no positive feedback on those posts.
Let's see if we can do better this time
Here is a small cone, to make you more happy:
Current Java/Android concurrency framework leads to callback hells and blocking states because we do not have any other simple way to guarantee thread safety.
With coroutines, kotlin brings a very efficient and complete framework to manage concurrency in a more performant and simple way.
Coroutines do not replace threads, it’s more like a framework to manage it.
Its philosophy is to define an execution context which allows to wait for background operations to complete, without blocking the original thread.
The goal here is to avoid callbacks and make concurrency easier.
Very simple first example, we launch a coroutine in the Main
context (main thread). In it, we retrieve an image from the IO
one, and process it back in Main
.
launch(Dispatchers.Main) {
val image = withContext(Dispatchers.IO) { getImage() } // Get from IO context
imageView.setImageBitmap(image) // Back on main thread
}
Staightforward code, like a single threaded function. And while getImage
runs in IO
dedicated threadpool, the main thread is free for any other job!
withContext
function suspends the current coroutine while its action (getImage()
) is running. As soon as getImage()
returns and main looper is available, coroutine resumes on main thread, and imageView.setImageBitmap(image)
is called.
Second example, we now want 2 background works done to use them. We will use the async
/await
duo to make them run in parallel and use their result in main thread as soon as both are ready:
val job = launch(Dispatchers.Main) {
val deferred1 = async(Dispatchers.Default) { getFirstValue() }
val deferred2 = async(Dispatchers.IO) { getSecondValue() }
useValues(deferred1.await(), deferred2.await())
}
job.join() // suspends current coroutine until job is done
async
is similar to launch
but returns a deferred
(which is the Kotlin equivalent of Future
), so we can get its result with await()
. Called with no parameter, it runs in current scope default context.
And once again, the main thread is free while we are waiting for our 2 values.
As you can see, launch
funtion returns a Job
that can be used to wait for the operation to be over, with the join()
function. It works like in any other language, except that it suspends the coroutine instead of blocking the thread.
Dispatching is a key notion with coroutines, it’s the action to ‘jump’ from a thread to another one.
Let’s look at our current java equivalent of Main
dispatching, which is runOnUiThread
:
public final void runOnUiThread(Runnable action) {
if (Thread.currentThread() != mUiThread) {
mHandler.post(action); // Dispatch
} else {
action.run(); // Immediate execution
}
}
Android implementation of Main
context is a dispatcher based on a Handler. So this really is the matching implementation:
launch(Dispatchers.Main) { ... }
vs
launch(Dispatchers.Main, CoroutineStart.UNDISPATCHED) { ... }
// Since kotlinx 0.26:
launch(Dispatchers.Main.immediate) { ... }
launch(Dispatchers.Main)
posts a Runnable
in a Handler
, so its code execution is not immediate.
launch(Dispatchers.Main, CoroutineStart.UNDISPATCHED)
will immediately execute its lambda expression in the current thread.
Dispatchers.Main
guarantees that coroutine is dispatched on main thread when it resumes, and it uses a Handler
as the native Android implementation to post in the application event loop.
Its actual implementation looks like:
val Main: HandlerDispatcher = HandlerContext(mainHandler, "Main")
To get a better understanding of Android dispatching, you can read this blog post on Understanding Android Core: Looper, Handler, and HandlerThread.
A couroutine context (aka coroutine dispatcher) defines on which thread its code will execute, what to do in case of thrown exception and refers to a parent context, to propagate cancellation.
val job = Job()
val exceptionHandler = CoroutineExceptionHandler {
coroutineContext, throwable -> whatever(throwable)
}
launch(Disaptchers.Default+exceptionHandler+job) { ... }
job.cancel()
will cancel all coroutines that have job
as a parent. And exceptionHandler
will receive all thrown exceptions in these coroutines.
A coroutineScope
makes errors handling easier:
If any child coroutine fails, the entire scope fails and all of children coroutines are cancelled.
In the async
example, if the retrieval of a value failed, the other one continued then we would have a broken state to manage.
With a coroutineScope
, useValues
will be called only if both values retrieval succeeded. Also, if deferred2
fails, deferred1
is cancelled.
coroutineScope {
val deferred1 = async(Dispatchers.Default) { getFirstValue() }
val deferred2 = async(Dispatchers.IO) { getSecondValue() }
useValues(deferred1.await(), deferred2.await())
}
We also can “scope” an entire class to define its default CoroutineContext
and leverage it.
Example of a class implementing CoroutineScope
:
open class ScopedViewModel : ViewModel(), CoroutineScope {
protected val job = Job()
override val coroutineContext = Dispatchers.Main+job
override fun onCleared() {
super.onCleared()
job.cancel()
}
}
Launching coroutines in a CoroutineScope
:
launch
or async
default dispatcher is now the current scope dispatcher. And we can still choose a different one the same way we did before.
launch {
val foo = withContext(Dispatchers.IO) { … }
// lambda runs within scope's CoroutineContext
…
}
launch(Dispatchers.Default) {
// lambda runs in default threadpool.
…
}
Standalone coroutine launching (outside of any CoroutineScope
):
GlobalScope.launch(Dispatchers.Main) {
// lambda runs in main thread.
…
}
Dispatchers.Default
(and Main
…)Dispatchers.IO
designed for thisDispatchers.Default
is based on a ForkJoinPool on Android 5+Channel definition from JetBrain documentation:
A Channel
is conceptually very similar to BlockingQueue
. One key difference is that instead of a blocking put operation it has a suspending send
(or a non-blocking offer
), and instead of a blocking take operation it has a suspending receive
.
Let’s start with a simple tool to use Channels, the Actor
.
We already saw it in this blog with the DiffUtil kotlin implementation.
Actor
is, yet again, very similar to Handler
: we define a coroutine context (so, the tread where to execute actions) and it will execute it in a sequencial order.
Difference is it uses coroutines of course :), we can specify a capacity and executed code can suspend.
An actor
will basically forward any order to a coroutine Channel
. It will guaranty the order execution and confine operations in its context. It greatly helps to remove synchronize
calls and keep all threads free!
protected val updateActor by lazy {
actor<Update>(capacity = Channel.UNLIMITED) {
for (update in channel) when (update) {
Refresh -> updateList()
is Filter -> filter.filter(update.query)
is MediaUpdate -> updateItems(update.mediaList as List<T>)
is MediaAddition -> addMedia(update.media as T)
is MediaListAddition -> addMedia(update.mediaList as List<T>)
is MediaRemoval -> removeMedia(update.media as T)
}
}
}
// usage
fun filter(query: String?) = updateActor.offer(Filter(query))
//or
suspend fun filter(query: String?) = updateActor.send(Filter(query))
In this example, we take advantage of the Kotlin sealed classes feature to select which action to execute.
sealed class Update
object Refresh : Update()
class Filter(val query: String?) : Update()
class MediaAddition(val media: Media) : Update()
And all this actions will be queued, they will never run in parallel. That’s a good way to achieve mutability confinement.
Actors can be profitable for Android UI management too, they can ease tasks cancellation and prevent overloading of the main thread.
Let’s implement it and call job.cancel()
when activity is destroyed.
class MyActivity : AppCompatActivity(), CoroutineScope {
protected val job = SupervisorJob() // the instance of a Job for this activity
override val coroutineContext = Dispatchers.Main.immediate+job
override fun onDestroy() {
super.onDestroy()
job.cancel() // cancel the job when activity is destroyed
}
}
A SupervisorJob
is similar to a regular Job with the only exception that cancellation is propagated only downwards.
So we do not cancel all coroutines in the Activity
, when one fails.
A bit better, with an extension function, we can make this CoroutineContext
accessible from any View
of a CoroutineScope
val View.coroutineContext: CoroutineContext?
get() = (context as? CoroutineScope)?.coroutineContext
We can now combine all this, setOnClick
function creates a conflated actor
to manage its onClick
actions. In case of multiple clicks, intermediates actions will be ignored, preventing any ANR, and these actions will be executed in Activity
’s scope. So it will be cancelled when
Activity` is destroyed 😎
fun View.setOnClick(action: suspend () -> Unit) {
// launch one actor as a parent of the context job
val eventActor = (context as? CoroutineScope)?.actor<Unit>(
capacity = Channel.CONFLATED) {
for (event in channel) action()
} ?: GlobalScope.actor<Unit>(
Dispatchers.Main,
capacity = Channel.CONFLATED) {
for (event in channel) action()
}
// install a listener to activate this actor
setOnClickListener { eventActor.offer(Unit) }
}
In this example, we set the Channel
as Conflated to ignore events when we have too much of them. You can change it to Channel.UNLIMITED
if you prefer to queue events without missing anyone of them, but still protect your app from ANR
We also can combine coroutines and Lifecycle
frameworks to automate UI tasks cancellation:
val LifecycleOwner.untilDestroy: Job get() {
val job = Job()
lifecycle.addObserver(object: LifecycleObserver {
@OnLifecycleEvent(Lifecycle.Event.ON_DESTROY)
fun onDestroy() { job.cancel() }
})
return job
}
//usage
GlobalScope.launch(Dispatchers.Main, parent = untilDestroy) {
/* amazing things happen here! */
}
Example of a callback based API use transformed thank to a Channel
.
API works like this:
requestBrowsing(url, listener)
triggers the parsing of folder at url
address.listener
receives onMediaAdded(media: Media)
for each discovered media in this folder.listener.onBrowseEnd()
is called once folder parsing is done.Here is the old refresh
function in VLC browser provider:
private val refreshList = mutableListOf<Media>()
fun refresh() = requestBrowsing(url, refreshListener)
private val refreshListener = object : EventListener{
override fun onMediaAdded(media: Media) {
refreshList.add(media))
}
override fun onBrowseEnd() {
val list = refreshList.toMutableList()
refreshList.clear()
launch {
dataset.value = list
parseSubDirectories()
}
}
}
How to improve this?
We create a channel, which will be initiated in refresh
. Browser callbacks will now only forward media to this channel then close it.
Refresh
function is now easier to understand. It sets the channel, calls the VLC browser then fills a list with the media and processes it.
Instead of the select
or consumeEach
functions, we can use for
to wait for media and it will break once browserChannel
is closed
private lateinit var browserChannel : Channel<Media>
override fun onMediaAdded(media: Media) {
browserChannel.offer(media)
}
override fun onBrowseEnd() {
browserChannel.close()
}
suspend fun refresh() {
browserChannel = Channel(Channel.UNLIMITED)
val refreshList = mutableListOf<Media>()
requestBrowsing(url)
//Suspends at every iteration to wait for media
for (media in browserChannel) refreshList.add(media)
//Channel has been closed
dataset.value = refreshList
parseSubDirectories()
}
Second approach, we don’t use kotlinx-coroutines at all but the coroutine core framework.
Let’s see how coroutines really work!
retrofitSuspendCall
function wraps a Retrofit Call
request to make it a suspend
function.
With suspendCoroutine
we call the Call.enqueue
method and suspend the coroutine. The provided callback will call continuation.resume(response)
to resume the coroutine with the server response once received.
Then, we just have to bundle our Retrofit functions in retrofitSuspendCall
to have a suspending functions returning the requests result.
suspend inline fun <reified T> retrofitSuspendCall(request: () -> Call<T>
) : Response<T> = suspendCoroutine { continuation ->
request.invoke().enqueue(object : Callback<T> {
override fun onResponse(call: Call<T>, response: Response<T>) {
continuation.resume(response)
}
override fun onFailure(call: Call<T>, t: Throwable) {
continuation.resumeWithException(t)
}
})
}
suspend fun browse(path: String?) = retrofitSuspendCall {
ApiClient.browse(path)
}
// usage (within Main coroutine context)
livedata.value = Repo.browse(path)
This way, the network blocking call is done in Retrofit dedicated thread, coroutine is here to wait for the response, and in-app usage couldn’t be simpler!
This implementation is inspired by gildor/kotlin-coroutines-retrofit library, which makes it ready to use.
JakeWharton/retrofit2-kotlin-coroutines-adapter is also available with another implementation, for the same result.
Channel
framework can be used in many other ways, you can look at BroadcastChannel for more powerful implementations according to your needs.
We can also create channels with the Produce function.
It can also be useful for communication between UI components: an adapter can pass click events to its Fragment/Activity via a Channel
or an Actor
for example.
Related readings: