Reproducible JARs

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Reproducible JARs

George Bateman
Dear List,

I'm currently trying to make Processing <https://processing.org> build
reproducibly (so building it twice yields the exact same output file).
Currently this involves, as far as I can tell, unzipping every JAR
file, touching the files with a constant modification time, and
re-zipping it. (Even if you were to touch the .class files in advance,
there's MANIFEST.MF, which is created by Ant and which you have to
unzip to access.)

Unless there's some efficient way of doing this that I've missed,
could you advise me on how I'd go about writing a patch for Ant that
makes reproducible JARs easier? I'd been thinking of adding a
"modificationtime" attribute to the jar and zip tasks, and giving that
time to all the files, but I'd be grateful if you could give me a
rough idea of how Ant works and which files I'd need to be looking at
and editing.

Thanks,
George Bateman.

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reproducible JARs

Stefan Bodewig
On 2017-07-22, George Bateman wrote:

> I'm currently trying to make Processing <https://processing.org> build
> reproducibly (so building it twice yields the exact same output file).
> Currently this involves, as far as I can tell, unzipping every JAR
> file, touching the files with a constant modification time, and
> re-zipping it. (Even if you were to touch the .class files in advance,
> there's MANIFEST.MF, which is created by Ant and which you have to
> unzip to access.)

The manifest may contain the current time stamp, to make things worse
for you :-)

I'm not sure what you want can be achieved at all. The best you can get
is the exact same jar when the build has been run on the same OS with
the same version of java, javac and zlib (and probably a few other
things I'm missing right now).

Different javacs will create different class files. ZIP (and thus JAR)
creation uses zlib under the covers and different versions may result in
different deflated output.

> Unless there's some efficient way of doing this that I've missed,
> could you advise me on how I'd go about writing a patch for Ant that
> makes reproducible JARs easier? I'd been thinking of adding a
> "modificationtime" attribute to the jar and zip tasks, and giving that
> time to all the files, but I'd be grateful if you could give me a
> rough idea of how Ant works and which files I'd need to be looking at
> and editing.

First of all you'd have to modify the class
org.apache.tools.ant.taskdefs.Zip and add a setModificationtime method
to it - this will create the attribute for both tasks, as the
implementation of <jar> (org.apache.tools.ant.taskdefs.Jar) is a
subclass of the Zip class. You'll need to think about what the value of
the attribute should be - milliseconds since epoch? A formatted string
containing the timestamp to set?

And then you need to look for all places where Zip or Jar invoke setTime
on a ZipEntry instance.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Reproducible JARs

Gintautas Grigelionis
I believe the easier way would be to diff the fileset of classes to be
packed to the fileset of classes in the jar before creating a new jar.
There's a tool called zipdiff that could do the trick (it may ignore
timestamps, etc). That would correspond to a javac deciding on whether to
recompile the classes (which is based on timestamps, I guess ;-). It sounds
to me like you're looking for an equivalent of ClearCase winkin. If that's
the case, have you read [1]?

Gintas

[1]
https://www.packtpub.com/books/content/ibm-rational-clearcase-challenges-java-development

2017-07-23 12:35 GMT+02:00 Stefan Bodewig <[hidden email]>:

> On 2017-07-22, George Bateman wrote:
>
> > I'm currently trying to make Processing <https://processing.org> build
> > reproducibly (so building it twice yields the exact same output file).
> > Currently this involves, as far as I can tell, unzipping every JAR
> > file, touching the files with a constant modification time, and
> > re-zipping it. (Even if you were to touch the .class files in advance,
> > there's MANIFEST.MF, which is created by Ant and which you have to
> > unzip to access.)
>
> The manifest may contain the current time stamp, to make things worse
> for you :-)
>
> I'm not sure what you want can be achieved at all. The best you can get
> is the exact same jar when the build has been run on the same OS with
> the same version of java, javac and zlib (and probably a few other
> things I'm missing right now).
>
> Different javacs will create different class files. ZIP (and thus JAR)
> creation uses zlib under the covers and different versions may result in
> different deflated output.
>
> > Unless there's some efficient way of doing this that I've missed,
> > could you advise me on how I'd go about writing a patch for Ant that
> > makes reproducible JARs easier? I'd been thinking of adding a
> > "modificationtime" attribute to the jar and zip tasks, and giving that
> > time to all the files, but I'd be grateful if you could give me a
> > rough idea of how Ant works and which files I'd need to be looking at
> > and editing.
>
> First of all you'd have to modify the class
> org.apache.tools.ant.taskdefs.Zip and add a setModificationtime method
> to it - this will create the attribute for both tasks, as the
> implementation of <jar> (org.apache.tools.ant.taskdefs.Jar) is a
> subclass of the Zip class. You'll need to think about what the value of
> the attribute should be - milliseconds since epoch? A formatted string
> containing the timestamp to set?
>
> And then you need to look for all places where Zip or Jar invoke setTime
> on a ZipEntry instance.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Loading...