Installing qml caches

[Rolf Eike Beer]

Hi all,
It would be nice to install .qmlc files in addition to .qml files to
reduce start-up time of applications.
qmlcachegen -o example.qmlc qxample.qml
install(DIRECTORY qml/ DESTINATION ${KDE_INSTALL_QMLDIR}/org/kde/kcm)
So this could be changed to something like
ecm_install_qml(qml org/kde/kcm)
I wonder whether someone already working on this or want to implement
such a function?
If not, I'll try to do it myself.

Just a quick note: the quick compiler macros provided by Qt 5.11.0 are
broken for cross-compilation (QTBUG-68724).

Eike

[Sune Vuorela]

This can be made optional, as it is done in Qt.
So distros will have a choice whether to install qmlc files or not.
Debian installs them.

I think Debian only installs the qmlc files for Qt packages (where the
versioning already is tightly coupled to Qt stuff)

/Sune

[Lisandro Damián Nicanor Pérez Meyer]

Post by Sune Vuorela

This can be made optional, as it is done in Qt.
So distros will have a choice whether to install qmlc files or not.
Debian installs them.

I think Debian only installs the qmlc files for Qt packages (where the
versioning already is tightly coupled to Qt stuff)

Exactly. But if it the gain is *really important* we might come up with a
solution maybe, like triggering rebuilds in the user's machine. But the gain
needs to be really significant.

[Александр Волков]

Post by Lisandro Damián Nicanor Pérez Meyer
Exactly. But if it the gain is *really important* we might come up with a
solution maybe, like triggering rebuilds in the user's machine. But the gain
needs to be really significant.

Using triggers seems to be the best option.
A package can be created that will create/update qml caches.
Thus there is no need to modify existing packages.

Why system caches are better than per-user caches?
1. There is no extra delay when a user starts an application for the
first time
or after updating libs.
2. Out-of-date caches files are automatically removed.

[Volker Krause]

Hi all,
It would be nice to install .qmlc files in addition to .qml files to
reduce
start-up time of applications.
qmlcachegen -o example.qmlc qxample.qml
install(DIRECTORY qml/ DESTINATION ${KDE_INSTALL_QMLDIR}/org/kde/kcm)
So this could be changed to something like
ecm_install_qml(qml org/kde/kcm)
I wonder whether someone already working on this or want to implement such
a function?
If not, I'll try to do it myself.
Cheers,
Alexander.

I'm not sure this makes sense, as far as I know it's not a
binary-compatible format, so distros would have to recompile every
application with qmlc files upon Qt update.

It might indeed not be useful for distros, but it looks like an interesting
option to reduce the package size (as AFAIK we don't need to ship the qml
files anymore then) and improve the first start performance for our APKs for
example. Probably not the change with the highest impact/gain atm, but
nevertheless worth exploring IMHO.

Regards,
Volker

[Volker Krause]

Hi all,
It would be nice to install .qmlc files in addition to .qml files to
reduce start-up time of applications.
qmlcachegen -o example.qmlc qxample.qml
install(DIRECTORY qml/ DESTINATION ${KDE_INSTALL_QMLDIR}/org/kde/kcm)
So this could be changed to something like
ecm_install_qml(qml org/kde/kcm)
I wonder whether someone already working on this or want to implement
such a function?
If not, I'll try to do it myself.

Qt 5.11 actually has something like this, but only covering QML files compiled
in via qrc, see qtquick_compiler_add_resources() (I needed a very recent 5.11
branch build though for this to work, post 5.11.1).

I have no performance measurements yet, but at least the package size goes up
contrary to what one might expect, as the compiled binary is 2-3x larger than
the (non-minified) QML source code. That space would be needed by the qmlc
files anyway in the end though, so overall disk space cost goes down a bit
once deployed.

With the QML sources no longer included you end up with a non-starting
application after updating Qt though.

So, interesting for APKs etc where you fully control Qt, but probably not for
distros.

Regards,
Volker

[Lisandro Damián Nicanor Pérez Meyer]

El martes, 3 de julio de 2018 05:21:56 -03 Volker Krause escribió:
[snip]

Post by Volker Krause
Qt 5.11 actually has something like this, but only covering QML files
compiled in via qrc, see qtquick_compiler_add_resources() (I needed a very
recent 5.11 branch build though for this to work, post 5.11.1).
I have no performance measurements yet, but at least the package size goes
up contrary to what one might expect, as the compiled binary is 2-3x larger
than the (non-minified) QML source code. That space would be needed by the
qmlc files anyway in the end though, so overall disk space cost goes down a
bit once deployed.
With the QML sources no longer included you end up with a non-starting
application after updating Qt though.
So, interesting for APKs etc where you fully control Qt, but probably not
for distros.

I was actually thinking in packages shipping qml source files as usual, make
package register them with some tool and recompile them with any qt update.

Now I don't know if that happens automatically, ie, qml files become compiled
at runtime in a cache. If that's the case then only the first run for each
user would just take time.

But again, the benefit really needs to be interesting to deploy something like
this.

[Elv1313 .]

Post by Volker Krause
It might indeed not be useful for distros, but it looks like an interesting
option to reduce the package size

I played with this last year at Randa after we spoke about it. I
heavily patched everything to bring the size down and improve startup
time. QML cache doesn't help that goal at all. I ended up not using it
and it made both size and startup time worst on mobile devices. The
startup time is I/O bound on modern phones, not CPU bound. I am
leaving for the airport so I can't write a 10 page tutorial and it
isn't related to shipping QML cache.

## What doesn't help:

* Compile QML (it does nothing)
* Minify QML (the bundle are often already compressed)

## What help, but is broken

* Using musl libc because libGl cannot be bundled and they must have
the same libc
* Static glibc causes 2 instances of static vars to exist and it crashes

## What is lost

* The way many KF5 apps use plugins can't work in static modes (like
Kio, kded5, kdeinit5) and have to be patched out.

## What helps:

* Using static libraries
* Using full bundle LTO (link time optimization) (reduces the bundle
size by 75%+ with this alone)
* Enable PGO/FDO (massive improvment in statup time)
* Optimize all SVG for loading speed (reduce the Inkscape created ones
by 96% and Qt loads them faster)
* QRC *everything* to avoid all syscalls and blocking I/O `stat()`
* Create a custom qplatform with more aggressive CXXFLAGS and LDFLAGS
* Using Qt-lite
* Using Gentoo "USE" system to disable optional dependency features
instead of using what ships with Qt
* Stripping the fonts from all the colored icons madness they recently
added and QRC it.

Here's an AppImage produced with the patched frameworks:

http://lvindustries.net:8080/

In both case it is not fully minified. The 45mb ones lack -Os and have
debug symbols.
The 18mb ones are not fully stripped to get useful backtraces. I can
make them ~13mb if I want, but it makes debugging impossible. To have
a good comparaison, that software has mostly the same dependencies as
KDenLive and their AppImage is more than an order of magnitude bigger.
Note that I don't run the PGO/FDO on those builds as I havn't
automated that part yet.

If there is enough interest, I guess I can create a small BoF at
Akademy to demonstrate and talk about how willing we are to makes this
work. If we get serious about this, a lot of things have to change and
new library guidelines enforced. Some people will find the new
limitations imposed on them very frustrating. But the bright side is
85%+ smaller APK/FlatPak/AppImage/DMG/Snap that load multiple time
faster.

(related: https://phabricator.kde.org/D13816)

[Simon Hausmann]

Hi,

I suspect that at the point when you measured, the qml files created ahead
of time contained the native code. Note that if they were big for your
taste, then remember that every byte there without cache files occupies
unshared memory (that can only be evicted to swap).

Since 5.11 the files are much smaller and contain byte code, in addition to
data structures that need creation otherwise.

I share Volker’s recommendation as well as yours:

Use static linking, LTO etc. for apps. And use the QtQuickCompiler build
option with 5.11.


Simon

Post by Volker Krause

Post by Volker Krause
It might indeed not be useful for distros, but it looks like an

interesting

Post by Volker Krause
option to reduce the package size

I played with this last year at Randa after we spoke about it. I
heavily patched everything to bring the size down and improve startup
time. QML cache doesn't help that goal at all. I ended up not using it
and it made both size and startup time worst on mobile devices. The
startup time is I/O bound on modern phones, not CPU bound. I am
leaving for the airport so I can't write a 10 page tutorial and it
isn't related to shipping QML cache.
* Compile QML (it does nothing)
* Minify QML (the bundle are often already compressed)
## What help, but is broken
* Using musl libc because libGl cannot be bundled and they must have
the same libc
* Static glibc causes 2 instances of static vars to exist and it crashes
## What is lost
* The way many KF5 apps use plugins can't work in static modes (like
Kio, kded5, kdeinit5) and have to be patched out.
* Using static libraries
* Using full bundle LTO (link time optimization) (reduces the bundle
size by 75%+ with this alone)
* Enable PGO/FDO (massive improvment in statup time)
* Optimize all SVG for loading speed (reduce the Inkscape created ones
by 96% and Qt loads them faster)
* QRC *everything* to avoid all syscalls and blocking I/O `stat()`
* Create a custom qplatform with more aggressive CXXFLAGS and LDFLAGS
* Using Qt-lite
* Using Gentoo "USE" system to disable optional dependency features
instead of using what ships with Qt
* Stripping the fonts from all the colored icons madness they recently
added and QRC it.
http://lvindustries.net:8080/
In both case it is not fully minified. The 45mb ones lack -Os and have
debug symbols.
The 18mb ones are not fully stripped to get useful backtraces. I can
make them ~13mb if I want, but it makes debugging impossible. To have
a good comparaison, that software has mostly the same dependencies as
KDenLive and their AppImage is more than an order of magnitude bigger.
Note that I don't run the PGO/FDO on those builds as I havn't
automated that part yet.
If there is enough interest, I guess I can create a small BoF at
Akademy to demonstrate and talk about how willing we are to makes this
work. If we get serious about this, a lot of things have to change and
new library guidelines enforced. Some people will find the new
limitations imposed on them very frustrating. But the bright side is
85%+ smaller APK/FlatPak/AppImage/DMG/Snap that load multiple time
faster.
(related: https://phabricator.kde.org/D13816)

[Thiago Macieira]

Post by Simon Hausmann
And use the QtQuickCompiler build
option with 5.11.

Is that just running qmlcachegen on all the *.qml?