Discussion:
FOSDEM 19 Debian Java talk
(too old to reply)
Markus Koschany
2019-01-29 18:10:01 UTC
Permalink
Hi all,

I will attend FOSDEM 19 in Brussels this weekend (03.02.2019, 12:40
local time) and give a lightning talk (15 min) about our heroics.

https://fosdem.org/2019/schedule/event/debian_java/

Naturally there is not enough time to explain everything in detail but
it should be adequate to put our message across. Here is your chance.
What do you consider important or what would you like other people from
the community to know?

I would like to reuse some of the scripts that we used for our last blog
post in 2016 [1] to fetch some packaging statistics. Are those scripts
still available? Otherwise I intend to use UDD a lot.

I want to answer the following questions:

How does the Java language rank in Debian?

https://sources.debian.org/stats/sid/

How many Java source packages / binary packages do we maintain or are
maintained in total including by other teams?

How many contributors are there?

How many uploads were made by those contributors?

How many bugs were fixed by them?

How many RC bugs were reported/fixed during the
Wheezy/Jessie/Stretch/Buster release cycle? My guess is the total
numbers are increasing.

How many RC bugs were caused by OpenJDK7/8/9/10/11? We have our tagged
bugs list for example.

https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=debian-***@lists.debian.org;tag=default-java9

What specific OpenJDK changes caused the most grief?

Why we all love JavaDoc? :E

etc.

If you have some other fun statistic, please let me know.

Regards,

Markus

[1]
https://java.debian.net/blog/2016/06/wheezy-lts-and-the-switch-to-openjdk-7.html
Markus Koschany
2019-01-30 20:10:02 UTC
Permalink
Great that you are giving that talk! I think the Java Team's work is
generally under-appreciated, so getting the word out should help with
I think one thing to mention is how the Debian Java Team has to
consistently fight the Java standard practice of bundling all deps into
a single JAR. This means there is no shared security updates, each dev
has to update every dependency themselves in that model. That works
great for large companies with staff devoted to doing that.
For the majority of Debian use cases, that works poorly. Debian
delivers on the promise that people can just "apt install foo" and have
it work, and receive security updates. The user does not even need to
know what language the program is written in, it just works.
Java developers need to learn the value of these use cases, and help
Debian by making it easier to package Java projects in the standard
distro method, with shared dependencies that are indepentently updated.
Thanks for your feedback. Yes, that's a very good point and I will
mention it!

Cheers,

Markus
Emmanuel Bourg
2019-01-30 22:10:01 UTC
Permalink
https://salsa.debian.org/publicity-team/bits/merge_requests/16
Nice, remember the Java Team also has its own blog [1], that might be
nice to announce the FOSDEM talk there. It's just a matter of updating
the blog repository [2], the update is published automatically by a
Gitlab trigger.


[1] https://java.debian.net/blog/
[2] https://salsa.debian.org/java-team/pkg-java-blog
Emmanuel Bourg
2019-02-01 13:30:01 UTC
Permalink
Hi Markus,
Post by Markus Koschany
I will attend FOSDEM 19 in Brussels this weekend (03.02.2019, 12:40
local time) and give a lightning talk (15 min) about our heroics.
https://fosdem.org/2019/schedule/event/debian_java/
Thanks a lot for holding that talk. I won't be able to attend by I can
make some suggestions.
Post by Markus Koschany
Naturally there is not enough time to explain everything in detail but
it should be adequate to put our message across. Here is your chance.
What do you consider important or what would you like other people from
the community to know?
You could maybe summarize the upcoming Java related changes in Buster. I
started thinking about a blog post on this topic, the freeze is the good
time to publish it. Some items worth mentioning:
- the transition to OpenJDK 11 and the massive effort involved (more
than 400 package updates the last time I counted)
- the state of the build tools. Ant and Maven are up to date, Gradle is
stuck at the last pre-Kotlin version. SBT still being worked on.
- the state of the JVM languages: Groovy 2.14, Scala 2.11.12 (2.12
requires SBT), Clojure 1.9. Kotlin is wanted but difficult to bootstrap.
- the state of the IDEs: Eclipse is gone (lack of maintainers), Netbeans 10
- on the application servers side: Jetty 9.4 and Tomcat 9, fully up to
date, and now with systemd integration.
- the reproducibility rate is now around 85% (Stretch was around 75%).
Post by Markus Koschany
I would like to reuse some of the scripts that we used for our last blog
post in 2016 [1] to fetch some packaging statistics. Are those scripts
still available? Otherwise I intend to use UDD a lot.
I have a script that compares the packages between two releases but I
haven't published it yet. I'll run it and post the result for the
stretch/testing differences (it doesn't addresses the maintainer/bug
counts though).
Post by Markus Koschany
What specific OpenJDK changes caused the most grief?
- New layout of the JDK installation
- Removal of tools.jar
- Removal of the JavaEE APIs (activation, JAXB, JAXWS...)
- Removal of javah
- javac source/target pre 1.6 removal.
- ByteBuffer return type changes (Java 11 compiled code breaks with 8)
- sun.misc.Unsafe breaking changes
- Javadoc pendantic errors, JQuery embedding

But let's not rant too much and keep the talk positive :)
Post by Markus Koschany
Why we all love JavaDoc? :E
Grrr :)


Emmanuel Bourg
Emmanuel Bourg
2019-02-01 20:40:02 UTC
Permalink
Post by Emmanuel Bourg
I have a script that compares the packages between two releases but I
haven't published it yet. I'll run it and post the result for the
stretch/testing differences.
Here is the summary for the stretch -> testing changes (the libraries
are not listed) :

Packages removed from testing:

* androidsdk-tools
* eclipse
* eclipse-anyedit
* eclipse-cdt
* eclipse-cdt-pkg-config
* eclipse-eclox
* eclipse-egit
* eclipse-gef
* eclipse-linuxtools
* eclipse-mercurialeclipse
* eclipse-mylyn
* eclipse-mylyn-tasks-github
* eclipse-ptp
* eclipse-pydev
* eclipse-remote-services-api
* eclipse-rse
* eclipse-subclipse
* eclipse-wtp
* eclipse-xsd
* glassfish
* jsymphonic
* jta
* netbeans
* openjdk-8-jre-dcevm
* pirl
* pleiades
* procyon
* sikulix
* triplea

Packages added to testing:

* gradle-completion (1.3.1)
* hibiscus (2.8.8)
* jameica (2.8.2)
* jaxws (2.3.0.2)
* maven-cache-cleanup (1.0.4)
* omegat (3.6.0.10)
* openjdk-11 (11.0.2+9)
* sbt-ivy (2.4.0~rc1)
* tomcat9 (9.0.14)

Packages upgraded in testing:

* activemq (5.15.8)
* ant (1.10.5)
* antlr4 (4.7.1)
* apache-directory-server (2.0.0~M24)
* aspectj (1.9.2)
* bnd (3.5.0)
* checkstyle (8.15)
* cobertura (2.1.1)
* cup (0.11b-20160615)
* derby (10.14.2.0)
* ecj (3.16.0)
* fop (2.3)
* freeplane (1.7.2)
* gradle (4.4.1)
* groovy (2.4.16)
* hsqldb (2.4.1)
* ivyplusplus (1.28)
* jabref (3.8.2+ds)
* japi-compliance-checker (2.4)
* java-common (0.71)
* java-wrappers (0.3)
* jaxb (2.3.0.1)
* jblas (1.2.4)
* jedit (5.5.0)
* jetty9 (9.4.14)
* jflex (1.7.0)
* jhove (1.20.1)
* jruby (9.1.13.0)
* jtreg (4.2-b13)
* jython (2.7.1+repack1)
* libapache-mod-jk (1.2.46)
* maven (3.6.0)
* nailgun (0.9.3)
* openjdk-8 (8u171-b11)
* openjfx (11.0.1+1)
* pdfsam (4.0.1)
* proguard (6.0.3)
* robocode (1.9.3.3)
* scala (2.11.12)
* simplyhtml (0.17.3)
* stegosuite (0.8.0)
* sweethome3d (6.0)
* sweethome3d-furniture (1.6.4)
* sweethome3d-furniture-editor (1.23)
* tomcat-native (1.2.19)
* tomcat8 (8.5.37) -> will be removed before the release
* uimaj (2.10.2)
* visualvm (1.4.2)
* xmlbeans (3.0.2)
* yui-compressor (2.4.8)
* zookeeper (3.4.13)

Total packages added to testing: 179
Total packages removed from testing: 95
Total packages updated in testing: 291
Total packages upgraded in testing: 283

Number of packages maintained: 1003
Markus Koschany
2019-02-01 21:50:02 UTC
Permalink
Great. Thanks!
Post by Emmanuel Bourg
Total packages added to testing: 179
Total packages removed from testing: 95
Total packages updated in testing: 291
Total packages upgraded in testing: 283
What numbers do we have if you include the libraries too?
Post by Emmanuel Bourg
Number of packages maintained: 1003
Hmm. Why is this different from

https://qa.debian.org/developer.php?email=pkg-java-maintainers%40lists.alioth.debian.org

The page lists 1191 packages in main.

Cheers,

Markus
Emmanuel Bourg
2019-02-01 22:40:02 UTC
Permalink
Post by Markus Koschany
What numbers do we have if you include the libraries too?
These numbers include the libraries (but they weren't listed above).
Post by Markus Koschany
Hmm. Why is this different from
https://qa.debian.org/developer.php?email=pkg-java-maintainers%40lists.alioth.debian.org
The page lists 1191 packages in main.
Because it includes oldstable+stable+testing+unstable. 1003 is for
testing only.

That said, I forgot to include the new Clojure packages. It brings
Leiningen 2.8.1 to the list of available build tools, and the numbers
become:

Total packages added to testing: 195
Total packages removed from testing: 95
Total packages updated in testing: 296
Total packages upgraded in testing: 291

Number of packages in stable: 932
Number of packages in testing: 1033 (+10,84%)

Emmanuel Bourg
Markus Koschany
2019-02-01 21:50:02 UTC
Permalink
Hi!
Post by Emmanuel Bourg
Hi Markus,
Post by Markus Koschany
I will attend FOSDEM 19 in Brussels this weekend (03.02.2019, 12:40
local time) and give a lightning talk (15 min) about our heroics.
https://fosdem.org/2019/schedule/event/debian_java/
Thanks a lot for holding that talk. I won't be able to attend by I can
make some suggestions.
Would have been great to see you there. Next time!
Post by Emmanuel Bourg
Post by Markus Koschany
Naturally there is not enough time to explain everything in detail but
it should be adequate to put our message across. Here is your chance.
What do you consider important or what would you like other people from
the community to know?
You could maybe summarize the upcoming Java related changes in Buster. I
started thinking about a blog post on this topic, the freeze is the good
- the transition to OpenJDK 11 and the massive effort involved (more
than 400 package updates the last time I counted)
- the state of the build tools. Ant and Maven are up to date, Gradle is
stuck at the last pre-Kotlin version. SBT still being worked on.
- the state of the JVM languages: Groovy 2.14, Scala 2.11.12 (2.12
requires SBT), Clojure 1.9. Kotlin is wanted but difficult to bootstrap.
- the state of the IDEs: Eclipse is gone (lack of maintainers), Netbeans 10
- on the application servers side: Jetty 9.4 and Tomcat 9, fully up to
date, and now with systemd integration.
- the reproducibility rate is now around 85% (Stretch was around 75%).
Perfect. Will be mentioned.
Post by Emmanuel Bourg
Post by Markus Koschany
I would like to reuse some of the scripts that we used for our last blog
post in 2016 [1] to fetch some packaging statistics. Are those scripts
still available? Otherwise I intend to use UDD a lot.
I have a script that compares the packages between two releases but I
haven't published it yet. I'll run it and post the result for the
stretch/testing differences (it doesn't addresses the maintainer/bug
counts though).
I tried to gather some fine grained statistics about bugs and used

https://udd.debian.org/bugs/

for that. However the results appear very strange to me. There is even a
selection for "Java team". I just wanted the total bugs reported in the
past 600 days and all RC bugs and all RC bugs fixed (done). It is not
very important but if you have some spare time in the next month we
could figure this out and create some sort of statistics page like

https://blends.debian.org/games/maintstats/

just with more information and for fun.
Post by Emmanuel Bourg
Post by Markus Koschany
What specific OpenJDK changes caused the most grief?
- New layout of the JDK installation
- Removal of tools.jar
- Removal of the JavaEE APIs (activation, JAXB, JAXWS...)
- Removal of javah
- javac source/target pre 1.6 removal.
- ByteBuffer return type changes (Java 11 compiled code breaks with 8)
- sun.misc.Unsafe breaking changes
- Javadoc pendantic errors, JQuery embedding
But let's not rant too much and keep the talk positive :)
I can't promise that. :)
Post by Emmanuel Bourg
Post by Markus Koschany
Why we all love JavaDoc? :E
Grrr :)
:)
Emmanuel Bourg
2019-02-01 23:00:01 UTC
Permalink
It is not very important but if you have some spare time in the next month
we could figure this out and create some sort of statistics page like
https://blends.debian.org/games/maintstats/
just with more information and for fun.
The same charts are generated for the Java Team if that helps:

Loading Image...
Loading Image...
Loading Image...
Loading Image...
Loading Image...

Emmanuel Bourg
Markus Koschany
2019-02-01 23:10:03 UTC
Permalink
Post by Emmanuel Bourg
It is not very important but if you have some spare time in the next month
we could figure this out and create some sort of statistics page like
https://blends.debian.org/games/maintstats/
just with more information and for fun.
http://blends.debian.net/liststats/authorstat_debian-java.png
http://blends.debian.net/liststats/authorstat_pkg-java-maintainers.png
http://blends.debian.net/liststats/bugs_pkg-java.png
http://blends.debian.net/liststats/commitstat_pkg-java.png
http://blends.debian.net/liststats/maintainer_per_package_pkg-java.png
Ah, interesting. I think the bugs closed graph is the most interesting
one. I always had the feeling the bugs rate increased at an exponential
rate for me. Still that would be more than 1000 bugs closed in the past
two years for all of us. I have to find a way to verify that at some
point in time but it doesn't feel too absurd right now.

Thanks,

Markus
Markus Koschany
2019-02-06 20:10:03 UTC
Permalink
Hi,

I'm still recovering from the infamous conference flu but I wanted to
give you a short report about the talk and the discussions afterwards.

You can find all the relevant links here. [1]

While I was going through all the stuff we talked about, I realized that
I would easily exceed my time limit. Thus I focused on Emmanuel's
suggestion to present our achievements during the Buster release cycle
in the first part and then I tried to explain the difference between
Java's version centric approach and Debian's preferences.

Apparently that aroused some interest and I got feedback from four
audience members after the talk.

The Fedora maintainer of Tycho and Eclipse offered help and support if
someone wants to update and maintain Eclipse.

Someone wanted to get more involved in Debian Java and I suggested to
get involved by contacting debian-***@lists.debian.org.

Andrej Shadura told me that he was working on Kotlin but discovered the
same issues I mentioned during my talk.

Another Java developer approached me and told me that he was impressed
by the amount of work we are doing but couldn't understand why we would
waste our talent on fighting Java build systems. ;) He was employed in a
company with 1500 employees that only does Java stuff and he told me
that 90% of his use cases are uberjars and docker images. :)

In conclusion I can highly recommend to visit FOSDEM once at least
because it is a great opportunity to connect with people and to broaden
one's mind.

I can also recommend to view some of these talks:

https://fosdem.org/2019/schedule/track/free_java/

especially

https://fosdem.org/2019/schedule/event/java_language_futures/

where Brian Goetz talks about what we can expect from Java language-wise
in the future. Project "Valhalla" and "Metropolis" (integrating GraalVM
into OpenJDK?) sounded promising. In a nutshell nobody should have to
worry about Java. It looks like it will last for another 20 years at least.

That's it for now.

Markus


[1]
https://fosdem.org/2019/schedule/event/debian_java/

https://fosdem.org/2019/schedule/event/debian_java/attachments/slides/3231/export/events/attachments/debian_java/slides/3231/FOSDEM19_Debian_Java.odp

https://fosdem.org/2019/schedule/event/debian_java/attachments/paper/3232/export/events/attachments/debian_java/paper/3232/FOSDEM19_Debian_Java.pdf

https://video.fosdem.org/2019/H.2215/debian_java.mp4
https://video.fosdem.org/2019/H.2215/debian_java.webm
Markus Koschany
2019-02-12 11:40:01 UTC
Permalink
Hello,

Dalibor Topic (Oracle) and Robert Scholte (Apache Maven) contacted me
and were so kind to agree to make this discussion public, so that others
can chime in too. I would like to use the opportunity to answer the
initial question "what we are interested in seeing better supported from
build tools" and give some general feedback about integrating Java into
Debian.


First of all Ant and Maven are most likely the best supported build
systems at the moment. We carry only two patches for Maven, one because
we use a newer version of SLF4j [1] and the second one is to make Maven
builds reproducible. [2] It looks like [1] has been already merged
upstream but [2] has not been forwarded yet. It would be great of
course, if Maven builds would be reproducible out-of-the-box. In general
I would like to see reproducible builds everywhere.

Otherwise we have two build tools / Maven plugins called
maven-debian-helper [3] and maven-repo-helper [4] that do all the Debian
specific operations. Maven already supports local repositories and
offline mode but I would like to see native support for unversioned jar
files and dependencies too. At the moment we create our own local
repository in /usr/share/maven-repo and in addition to the normal
version, we have a so called "debian" version. Other Java projects in
Debian only reference the debian version, so that we have to maintain
only one library or application and when we decide to upgrade a package,
reverse-depdencies continue to work because they use the unversioned
"debian" instead of a specific version. In my experience other languages
like Perl or Python, which are less version-centric, support this use
case better.

Regarding javadoc generation we would like to see an option that
basically reverts to pre OpenJDK8 and simply is less strict than the
current implementation. We currently use the undocumented and
unsupported --ignore-source-errors option when we build javadoc. It is
not feasible for us to fix all those syntax errors ourselves and we will
rather ditch our documentation packages should --ignore-source-errors go
away.

Our biggest challenge is Gradle. If Robert wants to help us then he
should never rewrite parts of Maven in Kotlin or encourage developers to
use a specific prebuilt version of Maven and to ship a script in every
project that downloads it from the internet (gradle-wrapper). ;)

That is all off the top of my head. Maybe someone else on the list wants
to chime in here.

Regards,

Markus


[1]
https://sources.debian.org/src/maven/3.6.0-1/debian/patches/slf4j-compatibility.patch/

[2]
https://sources.debian.org/src/maven/3.6.0-1/debian/patches/reproducible-build-timestamp.patch/

[3] https://tracker.debian.org/pkg/maven-debian-helper

[4] https://tracker.debian.org/pkg/maven-repo-helper
Robert Scholte
2019-02-12 19:30:01 UTC
Permalink
Post by Markus Koschany
Hello,
Dalibor Topic (Oracle) and Robert Scholte (Apache Maven) contacted me
and were so kind to agree to make this discussion public, so that othe=
rs
Post by Markus Koschany
can chime in too. I would like to use the opportunity to answer the
initial question "what we are interested in seeing better supported fr=
om
Post by Markus Koschany
build tools" and give some general feedback about integrating Java int=
o
Post by Markus Koschany
Debian.
First of all Ant and Maven are most likely the best supported build
systems at the moment. We carry only two patches for Maven, one becaus=
e
Post by Markus Koschany
we use a newer version of SLF4j [1] and the second one is to make Mave=
n
Post by Markus Koschany
builds reproducible. [2] It looks like [1] has been already merged
upstream but [2] has not been forwarded yet. It would be great of
course, if Maven builds would be reproducible out-of-the-box. In gener=
al
Post by Markus Koschany
I would like to see reproducible builds everywhere.
Hi Markus,

first of all thanks for the insights, it is important for us to know how=
=

Maven is used and in which way we can improve that way-of-work. Herv=E9 =
is =

already working hard on the reproducible builds specs with your team in =
=

order to find out how we can improve our maven-plugins to get reproducib=
le =

artifacts.

Maven itself is not 100% reproducible. We've learned that some Linux =

vendors rebuild Maven and the presentation confirmed that Debian is one =
of =

those vendors. What we've seen in the past is that sometimes people are =
=

having issues with Maven and after a while we discovered that they were =
=

not using the official Apache Maven distribution[1]. For us it is quite =
=

easy to say: sorry, not our official distribution, please contact your =

Linux distributor.
In such case we have 3 losers: the user, the Apache Maven project and th=
e =

Linux Distributor. If only the official Maven distribution was used, the=
n =

we would have had 3 winners.

When you decide to rebuild Maven, you're also taking all related =

responsibilities. I'm also wondering how you build Maven, since Maven is=
=

being built with Maven. That should be a challenge to also rebuild all =

plugins, etc.
And how do you test this and confirm that it works as the official =

distribution?
Sure, *IF* Maven is 100% reproducible then you can rely on our test-infr=
a, =

but that's not the situation.

So here are my main questions:
- Are you making clear that you're not using the official Maven =

distribution, e.g. by adjust the info from 'mvn --version'?
- What is the optimum way of distributing Maven sources? For example: al=
so =

providing compile and package scripts (instead of calling maven-plugins)=
?

[1] https://maven.apache.org/download.cgi
Post by Markus Koschany
Otherwise we have two build tools / Maven plugins called
maven-debian-helper [3] and maven-repo-helper [4] that do all the Debi=
an
Post by Markus Koschany
specific operations. Maven already supports local repositories and
offline mode but I would like to see native support for unversioned ja=
r
Post by Markus Koschany
files and dependencies too. At the moment we create our own local
repository in /usr/share/maven-repo and in addition to the normal
version, we have a so called "debian" version. Other Java projects in
Debian only reference the debian version, so that we have to maintain
only one library or application and when we decide to upgrade a packag=
e,
Post by Markus Koschany
reverse-depdencies continue to work because they use the unversioned
"debian" instead of a specific version. In my experience other languag=
es
Post by Markus Koschany
like Perl or Python, which are less version-centric, support this use
case better.
Regarding javadoc generation we would like to see an option that
basically reverts to pre OpenJDK8 and simply is less strict than the
current implementation. We currently use the undocumented and
unsupported --ignore-source-errors option when we build javadoc. It is=
not feasible for us to fix all those syntax errors ourselves and we wi=
ll
Post by Markus Koschany
rather ditch our documentation packages should --ignore-source-errors =
go
Post by Markus Koschany
away.
Our biggest challenge is Gradle. If Robert wants to help us then he
should never rewrite parts of Maven in Kotlin or encourage developers =
to
Post by Markus Koschany
use a specific prebuilt version of Maven and to ship a script in every=
project that downloads it from the internet (gradle-wrapper). ;)
Interesting :) We've been discussing how we could get more contributors =
=

and Kotlin was one idea, but that one didn't make it.
Even though we as Maven developers don't like the wrapper, the community=
=

is asking for it, so we're seriously considering to add it as part of =

Maven Core.

regards,
Robert
Post by Markus Koschany
That is all off the top of my head. Maybe someone else on the list wan=
ts
Post by Markus Koschany
to chime in here.
Regards,
Markus
[1]
https://sources.debian.org/src/maven/3.6.0-1/debian/patches/slf4j-comp=
atibility.patch/
Post by Markus Koschany
[2]
https://sources.debian.org/src/maven/3.6.0-1/debian/patches/reproducib=
le-build-timestamp.patch/
Post by Markus Koschany
[3] https://tracker.debian.org/pkg/maven-debian-helper
[4] https://tracker.debian.org/pkg/maven-repo-helper
Markus Koschany
2019-02-13 11:20:01 UTC
Permalink
Hi Robert,

Am 12.02.19 um 20:09 schrieb Robert Scholte:
[...]
Post by Emmanuel Bourg
Hi Markus,
first of all thanks for the insights, it is important for us to know how
Maven is used and in which way we can improve that way-of-work. Hervé is
already working hard on the reproducible builds specs with your team in
order to find out how we can improve our maven-plugins to get
reproducible artifacts.
Maven itself is not 100% reproducible. We've learned that some Linux
vendors rebuild Maven and the presentation confirmed that Debian is one
of those vendors. What we've seen in the past is that sometimes people
are having issues with Maven and after a while we discovered that they
were not using the official Apache Maven distribution[1]. For us it is
quite easy to say: sorry, not our official distribution, please contact
your Linux distributor.
In such case we have 3 losers: the user, the Apache Maven project and
the Linux Distributor. If only the official Maven distribution was used,
then we would have had 3 winners.
When you decide to rebuild Maven, you're also taking all related
responsibilities.
We hear this a lot and it seems to be more common in the Java community.
From Debian's point of view (other distributions like Fedora share the
same view) it is essential being able to rebuild software from source.
The prerequisite is the availability of source code of course. Most of
us find it even strange when upstream developers recommend to use their
prebuilt versions. Do they have something to hide? Why can't they just
make building from source easy and trivial?

We believe that every user should have the ability to modify software
but this is only possible if she can build it from source. We go to
great lengths to ensure that all software complies with Debian's free
software guidelines. For the enduser the language and build tools of a
certain piece of software almost become meaningless. They just type "apt
source maven", change into the maven directory and rebuild the software
with another one-liner.

In case of Maven I don't see where our release differs fundamentally
from your binary releases. As I said there is only one reproducibility
patch left that doesn't change the functionality at all. So what we do
is grab your sources from https://github.com/apache/maven/releases or
maven.apache.org. In our opinion, without making any significant
changes, it should just behave as your binary release when we build it
from source. Otherwise there is source code missing or different.
Post by Emmanuel Bourg
I'm also wondering how you build Maven, since Maven is
being built with Maven. That should be a challenge to also rebuild all
plugins, etc.
And how do you test this and confirm that it works as the official
distribution?
Sure, *IF* Maven is 100% reproducible then you can rely on our
test-infra, but that's not the situation.
It is a challenge to build Maven from source. We solved the
bootstrapping problem and now we can use Debian's Maven version to build
newer versions but we have to follow a certain build order when we make
an update.
Post by Emmanuel Bourg
- Are you making clear that you're not using the official Maven
distribution, e.g. by adjust the info from 'mvn --version'?
This is how it looks on Debian 9 "Stretch" at the moment.

Apache Maven 3.3.9
Maven home: /usr/share/maven
Java version: 1.8.0_181, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: de_DE, platform encoding: UTF-8
OS name: "linux", version: "4.9.0-8-amd64", arch: "amd64", family: "unix"

It is obvious from Maven home I guess but there is no special emphasis
on Debian because when you install Maven from Debian you will never get
a prebuilt binary release, this is implicit for all software in Debian's
main distribution.
Post by Emmanuel Bourg
also providing compile and package scripts (instead of calling
maven-plugins)?
The major headache for us is the initial bootstrapping of new compilers
or build tools. We often write our own Ant scripts to solve the chicken
and egg problem. For Maven this is currently solved but if I recall
correctly there are sometimes plugins that require a certain Maven
version which in turn requires a specific plugin version, a classic
dependency loop. So if there was a way to build Maven without Maven or
certain plugins that would obviously make our life easier.

[...]
Post by Emmanuel Bourg
Post by Markus Koschany
Our biggest challenge is Gradle. If Robert wants to help us then he
should never rewrite parts of Maven in Kotlin or encourage developers to
use a specific prebuilt version of Maven and to ship a script in every
project that downloads it from the internet (gradle-wrapper). ;)
Interesting :) We've been discussing how we could get more contributors
and Kotlin was one idea, but that one didn't make it.
Even though we as Maven developers don't like the wrapper, the community
is asking for it, so we're seriously considering to add it as part of
Maven Core.
Uh, don't give in to those developers. :) This is really a bad habit in
the Java community. A build system should be merely a tool to build your
software and maybe for simplifying project management but imagine all
C/C++/Python and Perl developers would be like that. What a nightmare.

Regards,

Markus
Gary Gregory
2019-02-14 13:50:02 UTC
Permalink
Jar manifest files carry data like "built-by" and implementation
information:

https://docs.oracle.com/javase/tutorial/deployment/jar/packageman.html

Why not reuse "Implementation-Vendor" or invent a new entry and put
"Debian" in there. Maven can display this additional information on "mvn
-version".

Gary
Post by Markus Koschany
Hi Robert,
[...]
Post by Emmanuel Bourg
Hi Markus,
first of all thanks for the insights, it is important for us to know how
Maven is used and in which way we can improve that way-of-work. Hervé is
already working hard on the reproducible builds specs with your team in
order to find out how we can improve our maven-plugins to get
reproducible artifacts.
Maven itself is not 100% reproducible. We've learned that some Linux
vendors rebuild Maven and the presentation confirmed that Debian is one
of those vendors. What we've seen in the past is that sometimes people
are having issues with Maven and after a while we discovered that they
were not using the official Apache Maven distribution[1]. For us it is
quite easy to say: sorry, not our official distribution, please contact
your Linux distributor.
In such case we have 3 losers: the user, the Apache Maven project and
the Linux Distributor. If only the official Maven distribution was used,
then we would have had 3 winners.
When you decide to rebuild Maven, you're also taking all related
responsibilities.
We hear this a lot and it seems to be more common in the Java community.
From Debian's point of view (other distributions like Fedora share the
same view) it is essential being able to rebuild software from source.
The prerequisite is the availability of source code of course. Most of
us find it even strange when upstream developers recommend to use their
prebuilt versions. Do they have something to hide? Why can't they just
make building from source easy and trivial?
We believe that every user should have the ability to modify software
but this is only possible if she can build it from source. We go to
great lengths to ensure that all software complies with Debian's free
software guidelines. For the enduser the language and build tools of a
certain piece of software almost become meaningless. They just type "apt
source maven", change into the maven directory and rebuild the software
with another one-liner.
In case of Maven I don't see where our release differs fundamentally
from your binary releases. As I said there is only one reproducibility
patch left that doesn't change the functionality at all. So what we do
is grab your sources from https://github.com/apache/maven/releases or
maven.apache.org. In our opinion, without making any significant
changes, it should just behave as your binary release when we build it
from source. Otherwise there is source code missing or different.
Post by Emmanuel Bourg
I'm also wondering how you build Maven, since Maven is
being built with Maven. That should be a challenge to also rebuild all
plugins, etc.
And how do you test this and confirm that it works as the official
distribution?
Sure, *IF* Maven is 100% reproducible then you can rely on our
test-infra, but that's not the situation.
It is a challenge to build Maven from source. We solved the
bootstrapping problem and now we can use Debian's Maven version to build
newer versions but we have to follow a certain build order when we make
an update.
Post by Emmanuel Bourg
- Are you making clear that you're not using the official Maven
distribution, e.g. by adjust the info from 'mvn --version'?
This is how it looks on Debian 9 "Stretch" at the moment.
Apache Maven 3.3.9
Maven home: /usr/share/maven
Java version: 1.8.0_181, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-8-openjdk-amd64/jre
Default locale: de_DE, platform encoding: UTF-8
OS name: "linux", version: "4.9.0-8-amd64", arch: "amd64", family: "unix"
It is obvious from Maven home I guess but there is no special emphasis
on Debian because when you install Maven from Debian you will never get
a prebuilt binary release, this is implicit for all software in Debian's
main distribution.
Post by Emmanuel Bourg
also providing compile and package scripts (instead of calling
maven-plugins)?
The major headache for us is the initial bootstrapping of new compilers
or build tools. We often write our own Ant scripts to solve the chicken
and egg problem. For Maven this is currently solved but if I recall
correctly there are sometimes plugins that require a certain Maven
version which in turn requires a specific plugin version, a classic
dependency loop. So if there was a way to build Maven without Maven or
certain plugins that would obviously make our life easier.
[...]
Post by Emmanuel Bourg
Post by Markus Koschany
Our biggest challenge is Gradle. If Robert wants to help us then he
should never rewrite parts of Maven in Kotlin or encourage developers to
use a specific prebuilt version of Maven and to ship a script in every
project that downloads it from the internet (gradle-wrapper). ;)
Interesting :) We've been discussing how we could get more contributors
and Kotlin was one idea, but that one didn't make it.
Even though we as Maven developers don't like the wrapper, the community
is asking for it, so we're seriously considering to add it as part of
Maven Core.
Uh, don't give in to those developers. :) This is really a bad habit in
the Java community. A build system should be merely a tool to build your
software and maybe for simplifying project management but imagine all
C/C++/Python and Perl developers would be like that. What a nightmare.
Regards,
Markus
Michael Osipov
2019-02-13 19:00:02 UTC
Permalink
Post by Emmanuel Bourg
Post by Markus Koschany
Hello,
Dalibor Topic (Oracle) and Robert Scholte (Apache Maven) contacted me
and were so kind to agree to make this discussion public, so that others
can chime in too. I would like to use the opportunity to answer the
initial question "what we are interested in seeing better supported from
build tools" and give some general feedback about integrating Java into
Debian.
First of all Ant and Maven are most likely the best supported build
systems at the moment. We carry only two patches for Maven, one because
we use a newer version of SLF4j [1] and the second one is to make Maven
builds reproducible. [2] It looks like [1] has been already merged
upstream but [2] has not been forwarded yet. It would be great of
course, if Maven builds would be reproducible out-of-the-box. In general
I would like to see reproducible builds everywhere.
Hi Markus,
first of all thanks for the insights, it is important for us to know how
Maven is used and in which way we can improve that way-of-work. Hervé is
already working hard on the reproducible builds specs with your team in
order to find out how we can improve our maven-plugins to get
reproducible artifacts.
Maven itself is not 100% reproducible. We've learned that some Linux
vendors rebuild Maven and the presentation confirmed that Debian is one
of those vendors. What we've seen in the past is that sometimes people
are having issues with Maven and after a while we discovered that they
were not using the official Apache Maven distribution[1]. For us it is
quite easy to say: sorry, not our official distribution, please contact
your Linux distributor.
In such case we have 3 losers: the user, the Apache Maven project and
the Linux Distributor. If only the official Maven distribution was used,
then we would have had 3 winners.
When you decide to rebuild Maven, you're also taking all related
responsibilities. I'm also wondering how you build Maven, since Maven is
being built with Maven. That should be a challenge to also rebuild all
plugins, etc.
And how do you test this and confirm that it works as the official
distribution?
Sure, *IF* Maven is 100% reproducible then you can rely on our
test-infra, but that's not the situation.
- Are you making clear that you're not using the official Maven
distribution, e.g. by adjust the info from 'mvn --version'?
I expressed my proposal to Hervé that we need a new property:
maven.vendor. Our official distribution will carry the value: ASF.
Everyone else who modifies the content must change the value in the
build.properties. Thus, we will quickly know that this distro has been
modified by someone else.

Michael
Bernd Eckenfels
2019-02-13 20:00:01 UTC
Permalink
Hello,

according to the Apache Release Policy a release is the source and while it allows and defines convinience binaries there is not really a Notion of „official binaries“ from the ASF Point of view. So Maybe the new property should be something like „binary Vendor“ or „packager“ (similiar what many package Managers have?)

I think the number of additional support Problems because of distribution specific packages and the number of solved Problems by distributions doing a good Integration Job can not be clearly tallied.

And therefore I would stay away from language like „modified“, „official“ to avoid those Groups to feel unwelcome (especially in the ASF spirit of open SOURCE).

Gruss
Bernd
--
http://bernd.eckenfels.net

Von: Michael Osipov
Gesendet: Mittwoch, 13. Februar 2019 19:36
An: Maven Developers List; Robert Scholte; Dalibor Topic; Markus Koschany
Cc: debian-***@lists.debian.org; Matthias Klose
Betreff: Re: Fwd: FOSDEM 19 Debian Java talk
Post by Emmanuel Bourg
Post by Markus Koschany
Hello,
Dalibor Topic (Oracle) and Robert Scholte (Apache Maven) contacted me
and were so kind to agree to make this discussion public, so that others
can chime in too. I would like to use the opportunity to answer the
initial question "what we are interested in seeing better supported from
build tools" and give some general feedback about integrating Java into
Debian.
First of all Ant and Maven are most likely the best supported build
systems at the moment. We carry only two patches for Maven, one because
we use a newer version of SLF4j [1] and the second one is to make Maven
builds reproducible. [2] It looks like [1] has been already merged
upstream but [2] has not been forwarded yet. It would be great of
course, if Maven builds would be reproducible out-of-the-box. In general
I would like to see reproducible builds everywhere.
Hi Markus,
first of all thanks for the insights, it is important for us to know how
Maven is used and in which way we can improve that way-of-work. Hervé is
already working hard on the reproducible builds specs with your team in
order to find out how we can improve our maven-plugins to get
reproducible artifacts.
Maven itself is not 100% reproducible. We've learned that some Linux
vendors rebuild Maven and the presentation confirmed that Debian is one
of those vendors. What we've seen in the past is that sometimes people
are having issues with Maven and after a while we discovered that they
were not using the official Apache Maven distribution[1]. For us it is
quite easy to say: sorry, not our official distribution, please contact
your Linux distributor.
In such case we have 3 losers: the user, the Apache Maven project and
the Linux Distributor. If only the official Maven distribution was used,
then we would have had 3 winners.
When you decide to rebuild Maven, you're also taking all related
responsibilities. I'm also wondering how you build Maven, since Maven is
being built with Maven. That should be a challenge to also rebuild all
plugins, etc.
And how do you test this and confirm that it works as the official
distribution?
Sure, *IF* Maven is 100% reproducible then you can rely on our
test-infra, but that's not the situation.
- Are you making clear that you're not using the official Maven
distribution, e.g. by adjust the info from 'mvn --version'?
I expressed my proposal to Hervé that we need a new property:
maven.vendor. Our official distribution will carry the value: ASF.
Everyone else who modifies the content must change the value in the
build.properties. Thus, we will quickly know that this distro has been
modified by someone else.

Michael


---------------------------------------------------------------------
To unsubscribe, e-mail: dev-***@maven.apache.org
For additional commands, e-mail: dev-***@maven.apache.org
Emmanuel Bourg
2019-02-14 22:00:02 UTC
Permalink
Hi Robert,

Some comments with my Maven package maintainer hat.
Post by Robert Scholte
I'm also wondering how you build Maven, since Maven is
being built with Maven. That should be a challenge to also rebuild all
plugins, etc.
The maven package is rebuilt with the version of Maven previously
packaged, that's not really a problem now. In the past we used a
convoluted Ant build mimicking the Maven phases but that's history now.

The source code is fetched from the release tags on the Git repositories
(the tags are polled everyday and we're warned when a new release is
available). One project (library, plugin, etc) is mapped to one package,
and the upgrade process is done manually. It isn't terribly difficult
but that's quite time consuming. If there are Debian/Ubuntu users in the
Maven community interested in helping they are very welcome.
Post by Robert Scholte
And how do you test this and confirm that it works as the official
distribution?
Sure, *IF* Maven is 100% reproducible then you can rely on our
test-infra, but that's not the situation.
Debian has a CI infra rebuilding 550+ packages using Maven as build
tool, so regressions tend to be caught fairly quickly (and immediately
[1] reported [2][3]). Also the version provided in the stable release is
available after months of user testing. So I'm pretty confident the
Debian package is true to the Apache binaries.
Post by Robert Scholte
- Are you making clear that you're not using the official Maven
distribution, e.g. by adjust the info from 'mvn --version'?
No we aren't, but that's worth considering. Note that as the Maven
reproducibility improves this will become unnecessary because at some
point we'll be able to build binaries strictly identical to the Apache ones.
Post by Robert Scholte
also providing compile and package scripts (instead of calling
maven-plugins)?
The current system is mostly fine to us. The only issue I got recently
was the embedding of the SLF4J sources at build time, because we don't
build binary packages installing the sources artifacts of the libraries
and they aren't available at build time (we could though, that's a
little more work). In this case I patched the build [4] to embed the
compiled classes with the shade plugin instead (the patch was refused
though [5], but that's a fairly minor divergence).
Post by Robert Scholte
Interesting :) We've been discussing how we could get more contributors
and Kotlin was one idea, but that one didn't make it.
Even though we as Maven developers don't like the wrapper, the community
is asking for it, so we're seriously considering to add it as part of
Maven Core.
The wrapper would have no significant impact on the Debian packaging, we
just remove it before assembling the source package (no binaries are
allowed).

Emmanuel Bourg

[1] https://issues.apache.org/jira/browse/MJAVADOC-504
[2] https://issues.apache.org/jira/browse/MPLUGIN-339
[3] https://issues.apache.org/jira/browse/SUREFIRE-1422
[4]
https://salsa.debian.org/java-team/maven/blob/master/debian/patches/slf4j-compatibility.patch
[5] https://github.com/apache/maven/pull/118

Dalibor Topic
2019-02-13 15:20:02 UTC
Permalink
Hi Markus,
Post by Markus Koschany
Regarding javadoc generation we would like to see an option that
basically reverts to pre OpenJDK8 and simply is less strict than the
current implementation.
Unfortunately, we don't plan to go back to the pre-Java 8 Javadoc
implementation in OpenJDK.

We do plan to remove the old doclet API in a future release, though:
https://mail.openjdk.java.net/pipermail/javadoc-dev/2019-February/000848.html
, and the support for HTML4 output has been removed two weeks ago:
https://hg.openjdk.java.net/jdk/jdk/rev/0d9dee001667 .

With respect to strictness, the doclint feature can be turned off
generally or selectively. A talklet/demo can be seen at


cheers,
dalibor topic
--
Oracle <http://www.oracle.com>
Dalibor Topic | Principal Product Manager
Phone: +494089091214 <tel:+494089091214> | Mobile: +491737185961
<tel:+491737185961>

ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | D-22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Nederland, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher

Green Oracle <http://www.oracle.com/commitment> Oracle is committed to
developing practices and products that help protect the environment
Continue reading on narkive:
Loading...