Ant support for users who use JUnit5

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Ant support for users who use JUnit5

Jaikiran Pai
I've started some experiments in adding support for the recently
released JUnit5 [1] in Ant. I don't mean using JUnit5 for test cases
written within our Ant project, instead I am talking about allowing
users to use a Ant task to be able to run JUnit5 based test cases as
requested here [2].

Before I get into the details of what I have tried so far, I would like
to quickly explain what JUnit5 brings for us (build tool developers). As
they note in their documentation, JUnit 5 has introduced proper APIs
targetted towards build tools and IDEs to be able to launch tests, in a
simple way. Furthermore, they have separated out these APIs from the
core of the tests itself. What that means is, the JUnit platform
launcher APIs, allow build tools to launch the JUnit platform and then
once launched, the ability to "identify" tests that need to be run
depends on the plugged in engine, which is/can be controlled by the
user's environment. So essentially the launched platform can be used to
run JUnit 5 test cases (written using JUnit 5 test APIs) or can be used
to run JUnit 4 test cases (they call these vintage). This decision is
done by a pluggable test engine, typically placed within the classpath.

With that context, I would like to explain what I have attempted so far
and I would like inputs on whether I should follow this path and/or if
there are other suggestions on how we should go ahead. As you are
probably aware, the JUnitTask (the Ant task backing the <junit> element)
has very complex logic which deals with launching the test runners
responsible for running JUnit tests (3.x or 4.x). IMO, most this logic
can now be handed off to the JUnit 5 platform launcher, using their APIs.

So one way to add JUnit 5 support would probably be to introduce a new
attribtue on the <junit> task, something like "use-junit-launcher" which
when set, will let us rely on this JUnit 5 launcher APIs. Of course, we
would also document that the usage of this attribute would imply that
they will have to setup their library dependencies to include all
relevant JUnit 5 related jars that provide this support, plus the
relevant JUnit test engine that suits their needs. This will allow them
to use the existing <junit> task and just add a new attribtue to deal
with JUnit5.

The other approach is to introduce a new task (say <junitlauncher>)
which has the exact same support in terms of what attributes and nested
elements are supported but its implementation is completely going to
rely on the JUnit 5 platform launcher to deal with the test execution.
Practically, that means, all the existing complex code in JUnitTask need
no longer be used with this task and instead it will be a much simpler
implementation. Users who want "pure" junit 3.x or 4.x can still stick
to <junit> task, whereas users who want to use JUnit 5 based platform,
either to run JUnit 4.x or 5.x can use this newer <junitlauncher> task.
Over time, in some release, we might probably just advertise
<junitlauncher> and deprecate <junit> (that's a long way away of course,
if we do indeed do that)

In my experiment, I have implemented the latter approach to show its
usage. The code/commit resides here [3]. It by no means is complete, but
does currently support JUnit 5 (and JUnit 4) in its current form. Before
I actually spend more time on this approach, I would like inputs and
thoughts on it. The usage of this task would be something like this
(just a snippet from my experimental build.xml):

<target name="test" depends="compile-tests">
         <junitlauncher>
             <!-- Multiple tests contained in the build.classes.dir-->
             <batchtest>
               <fileset dir="${build.classes.dir}"/>
             </batchtest>
             <!-- A specific test -->
             <test name="org.myapp.FooTest"/>
             <!-- classpath to be used for the tests -->
             <classpath refid="test.classpath" />
         </junitlauncher>
</target>


[1] http://junit.org/junit5/
[2] https://bz.apache.org/bugzilla/show_bug.cgi?id=61796
[3]
https://github.com/jaikiran/ant/commit/42c7f221a8cd0848df0a2d6cac2897c07d446bda

-Jaikiran


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Jaikiran Pai
After sending this mail, the more I think of it, I am not too sure
introducing a new task (junitlauncher) is serving any useful purpose. Of
course it does allow me to keep the task implementation simpler without
using the existing JUnitTask as an entry point of the call. But that is
an internal implementation detail, now that I think of it. So perhaps
just using the existing <junit> task and introducing relevant
attribute(s) to support this JUnit 5 launcher might be a better idea.
Even if we do stop supporting non-launcher based approach in some
distant future, we could still do that just by changing the internal
implementation detail of this task (and maybe removing support for
certain attributes).

Either way, I would still like to hear others thoughts on this.

-Jaikiran


On 14/12/17 3:39 PM, Jaikiran Pai wrote:

> I've started some experiments in adding support for the recently
> released JUnit5 [1] in Ant. I don't mean using JUnit5 for test cases
> written within our Ant project, instead I am talking about allowing
> users to use a Ant task to be able to run JUnit5 based test cases as
> requested here [2].
>
> Before I get into the details of what I have tried so far, I would
> like to quickly explain what JUnit5 brings for us (build tool
> developers). As they note in their documentation, JUnit 5 has
> introduced proper APIs targetted towards build tools and IDEs to be
> able to launch tests, in a simple way. Furthermore, they have
> separated out these APIs from the core of the tests itself. What that
> means is, the JUnit platform launcher APIs, allow build tools to
> launch the JUnit platform and then once launched, the ability to
> "identify" tests that need to be run depends on the plugged in engine,
> which is/can be controlled by the user's environment. So essentially
> the launched platform can be used to run JUnit 5 test cases (written
> using JUnit 5 test APIs) or can be used to run JUnit 4 test cases
> (they call these vintage). This decision is done by a pluggable test
> engine, typically placed within the classpath.
>
> With that context, I would like to explain what I have attempted so
> far and I would like inputs on whether I should follow this path
> and/or if there are other suggestions on how we should go ahead. As
> you are probably aware, the JUnitTask (the Ant task backing the
> <junit> element) has very complex logic which deals with launching the
> test runners responsible for running JUnit tests (3.x or 4.x). IMO,
> most this logic can now be handed off to the JUnit 5 platform
> launcher, using their APIs.
>
> So one way to add JUnit 5 support would probably be to introduce a new
> attribtue on the <junit> task, something like "use-junit-launcher"
> which when set, will let us rely on this JUnit 5 launcher APIs. Of
> course, we would also document that the usage of this attribute would
> imply that they will have to setup their library dependencies to
> include all relevant JUnit 5 related jars that provide this support,
> plus the relevant JUnit test engine that suits their needs. This will
> allow them to use the existing <junit> task and just add a new
> attribtue to deal with JUnit5.
>
> The other approach is to introduce a new task (say <junitlauncher>)
> which has the exact same support in terms of what attributes and
> nested elements are supported but its implementation is completely
> going to rely on the JUnit 5 platform launcher to deal with the test
> execution. Practically, that means, all the existing complex code in
> JUnitTask need no longer be used with this task and instead it will be
> a much simpler implementation. Users who want "pure" junit 3.x or 4.x
> can still stick to <junit> task, whereas users who want to use JUnit 5
> based platform, either to run JUnit 4.x or 5.x can use this newer
> <junitlauncher> task. Over time, in some release, we might probably
> just advertise <junitlauncher> and deprecate <junit> (that's a long
> way away of course, if we do indeed do that)
>
> In my experiment, I have implemented the latter approach to show its
> usage. The code/commit resides here [3]. It by no means is complete,
> but does currently support JUnit 5 (and JUnit 4) in its current form.
> Before I actually spend more time on this approach, I would like
> inputs and thoughts on it. The usage of this task would be something
> like this (just a snippet from my experimental build.xml):
>
> <target name="test" depends="compile-tests">
>         <junitlauncher>
>             <!-- Multiple tests contained in the build.classes.dir-->
>             <batchtest>
>               <fileset dir="${build.classes.dir}"/>
>             </batchtest>
>             <!-- A specific test -->
>             <test name="org.myapp.FooTest"/>
>             <!-- classpath to be used for the tests -->
>             <classpath refid="test.classpath" />
>         </junitlauncher>
> </target>
>
>
> [1] http://junit.org/junit5/
> [2] https://bz.apache.org/bugzilla/show_bug.cgi?id=61796
> [3]
> https://github.com/jaikiran/ant/commit/42c7f221a8cd0848df0a2d6cac2897c07d446bda
>
> -Jaikiran
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Stefan Bodewig
In reply to this post by Jaikiran Pai
On 2017-12-14, Jaikiran Pai wrote:

> With that context, I would like to explain what I have attempted so
> far and I would like inputs on whether I should follow this path
> and/or if there are other suggestions on how we should go ahead. As
> you are probably aware, the JUnitTask (the Ant task backing the
> <junit> element) has very complex logic which deals with launching the
> test runners responsible for running JUnit tests (3.x or 4.x). IMO,
> most this logic can now be handed off to the JUnit 5 platform
> launcher, using their APIs.

In addition some of the complexity inside of JUnitTask stems from the
fact that it must not refer to any JUnit 4.x API in order to be usable
in a pure JUnit 3.x environment. If you try to add JUnit 5 on top of
that this sounds diffciult.

I too had toyed with the idea of adding JUnit5 support but it is quite a
bit further down my TODO list and I've not looked into JUnit5 as deeply
as you did. What you've got so far looks promising, please please push
ahead.

My idea was to create a completely new task and mainly try to keep the
XML output the way it has been for some fifteen years so junitreport
would still work. I even though about placing it in a separate antlib
rather than Ant's core to allow it get released independently.

My vote would go to a separate task.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Gintautas Grigelionis
FWIW, I'm for a separate task. It may simplify dropping tasks that become
mired in legacy (who needs Jasper from Tomcat 4 nowadays?)
Or maybe we should start an ant-legacy antlib? :-)

Gintas

2017-12-14 12:44 GMT+01:00 Stefan Bodewig <[hidden email]>:

> On 2017-12-14, Jaikiran Pai wrote:
>
> > With that context, I would like to explain what I have attempted so
> > far and I would like inputs on whether I should follow this path
> > and/or if there are other suggestions on how we should go ahead. As
> > you are probably aware, the JUnitTask (the Ant task backing the
> > <junit> element) has very complex logic which deals with launching the
> > test runners responsible for running JUnit tests (3.x or 4.x). IMO,
> > most this logic can now be handed off to the JUnit 5 platform
> > launcher, using their APIs.
>
> In addition some of the complexity inside of JUnitTask stems from the
> fact that it must not refer to any JUnit 4.x API in order to be usable
> in a pure JUnit 3.x environment. If you try to add JUnit 5 on top of
> that this sounds diffciult.
>
> I too had toyed with the idea of adding JUnit5 support but it is quite a
> bit further down my TODO list and I've not looked into JUnit5 as deeply
> as you did. What you've got so far looks promising, please please push
> ahead.
>
> My idea was to create a completely new task and mainly try to keep the
> XML output the way it has been for some fifteen years so junitreport
> would still work. I even though about placing it in a separate antlib
> rather than Ant's core to allow it get released independently.
>
> My vote would go to a separate task.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Jaikiran Pai
In reply to this post by Stefan Bodewig
Thank you. I'm moving ahead with the new proposed task. Hopefully I
should be able to have something functional in a few days.

-Jaikiran


On 14/12/17 5:14 PM, Stefan Bodewig wrote:

> On 2017-12-14, Jaikiran Pai wrote:
>
>> With that context, I would like to explain what I have attempted so
>> far and I would like inputs on whether I should follow this path
>> and/or if there are other suggestions on how we should go ahead. As
>> you are probably aware, the JUnitTask (the Ant task backing the
>> <junit> element) has very complex logic which deals with launching the
>> test runners responsible for running JUnit tests (3.x or 4.x). IMO,
>> most this logic can now be handed off to the JUnit 5 platform
>> launcher, using their APIs.
> In addition some of the complexity inside of JUnitTask stems from the
> fact that it must not refer to any JUnit 4.x API in order to be usable
> in a pure JUnit 3.x environment. If you try to add JUnit 5 on top of
> that this sounds diffciult.
>
> I too had toyed with the idea of adding JUnit5 support but it is quite a
> bit further down my TODO list and I've not looked into JUnit5 as deeply
> as you did. What you've got so far looks promising, please please push
> ahead.
>
> My idea was to create a completely new task and mainly try to keep the
> XML output the way it has been for some fifteen years so junitreport
> would still work. I even though about placing it in a separate antlib
> rather than Ant's core to allow it get released independently.
>
> My vote would go to a separate task.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Jaikiran Pai
In reply to this post by Stefan Bodewig
Here's an update on where this effort now stands. As of yesterday, I
have the basic minimal functionality that I had planned for this (new)
task ready. There are other enhancements that this task will need as we
go along but at this point, this should be usable.

While working on this new task, I kept thinking whether it would make
more sense to just have this task as a separate project under the Ant
umbrella and have its own release/versioning cycle. Plus maybe a bit of
ease in building/testing it given that it won't need some of the
conditional logic that we do in Ant build itself when it comes to such
tasks. It did look like a good idea to separate it out but ultimately I
decided _not_ to do it mainly for the reason that I think it makes more
sense to have a task in Ant which allows users to write tests (using a
modern test framework) and test them, right out of the box of an Ant
installation. This first class experience, IMO, outweighs all the other
"advantages" that seem to come with having this as a separate project.

Coming to the task itself, the task is called "junitlauncher". In the
current state of this task[1], the following goals/features are
implemented and functional:

- Ability to launch the JUnit 5 platform

- Ability to specify a classpath to use for the launched tests

- Ability to specify one or more single test classes that need to be run
as part of this launch

- Ability to specify specific methods on these test classses that need
to be run as part of the launch

- Ability to specify "batch tests" which essentially is a way to use
Ant's resource collections to pattern match files that should be passed
on to the JUnit platform to be evaluated and run as test cases.

- Ability to specify "listeners" for tests. These listeners are expected
to implement (intentionally) JUnit platform's interface and _not_ any of
Ant's interfaces.

- Test result formatters are implemented as "listeners". This task comes
with 2 (for now) implementations out of the box, "plain" and "brief".
The idea behind these 2 is the same as that of what we current have with
the "junit" task formatters.

- Users can define custom formatters as "listeners" by specifying a
class which implements the JUnit's test listener plus (optionally) one
of Ant's own (new) custom interface. This custom interface will let them
have access to output stream to which they might want to write out the
results.

(I might be missing a few more details, but these are the major
functional features. The manual that I plan to write, will have the
whole details)

In its current state the task should be able to run both "vintage"
(JUnit 4.x) and "jupiter" (JUnit 5.x) based tests.

TODO/enhancements for later (a few days/weeks down the line):

- Ability to "fork" these tests in a separate JVM. I haven't fully
thought about this and might need some inputs on whether we need this or
not. If we do add this, I might do it slightly differently that what we
current do with "junit" where there are numerous attributes to the
task/test elements which are only applicable if fork mode is enabled. I
might perhaps just introduce a new element within the task's element
which specifically is meant to deal with any forked VM characteristics.
But that's something I will get to after the other easier enhancements
are done.

- Provide the XML formatter out of the box. I was planning to do this in
the first version itself, but I haven't had enough time to understand
the schema of this XML plus whether or not the details that we put in
here are available through the new JUnit launcher APIs.

- Ability to more specifically say which JUnit test engine needs to be
used for the tests. Right now, the classpath decides which engine gets
used and thus which classes are considered as tests. If the classpath
has both "vintage" and "jupiter" engines then such tests will be run.
However, I want users to be able to say "just use jupiter engine for
these tests" without having to worry about checking if the classpath is
polluted with some other engines.

- Make the current "junitreport" task be usable with the XML results of
this "junitlauncher" task. I haven't yet got to this but this should be
doable I think, of course once the XML formatter itself is ready and
functional. I don't plan to introduce a new task for this and instead
plan to reuse/enhance the existing junitreport task to work seamlessly
both with existing "junit" task and the "junitlauncher" task.

(Few other minor enhancements here and there and any user feedback reports)

For those of you curious to see what the task usage is going to look
like, here's an example[2] build file which shows its many usages. That
file will be cleaned up a bit, before I send a PR for review/merge.

[1] https://github.com/apache/ant/compare/master...jaikiran:junit5

[2]
https://github.com/apache/ant/compare/master...jaikiran:junit5#diff-3e0911ed925129e68c543aca0abc5d7aR52

-Jaikiran


On 14/12/17 5:14 PM, Stefan Bodewig wrote:

> On 2017-12-14, Jaikiran Pai wrote:
>
>> With that context, I would like to explain what I have attempted so
>> far and I would like inputs on whether I should follow this path
>> and/or if there are other suggestions on how we should go ahead. As
>> you are probably aware, the JUnitTask (the Ant task backing the
>> <junit> element) has very complex logic which deals with launching the
>> test runners responsible for running JUnit tests (3.x or 4.x). IMO,
>> most this logic can now be handed off to the JUnit 5 platform
>> launcher, using their APIs.
> In addition some of the complexity inside of JUnitTask stems from the
> fact that it must not refer to any JUnit 4.x API in order to be usable
> in a pure JUnit 3.x environment. If you try to add JUnit 5 on top of
> that this sounds diffciult.
>
> I too had toyed with the idea of adding JUnit5 support but it is quite a
> bit further down my TODO list and I've not looked into JUnit5 as deeply
> as you did. What you've got so far looks promising, please please push
> ahead.
>
> My idea was to create a completely new task and mainly try to keep the
> XML output the way it has been for some fifteen years so junitreport
> would still work. I even though about placing it in a separate antlib
> rather than Ant's core to allow it get released independently.
>
> My vote would go to a separate task.
>
> Stefan
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Jaikiran Pai
One minor but important detail - this feature/task will only be
available in the 1.10.x releases (whichever release we decide to include
this in) and won't be available in 1.9.x releases of Ant.

-Jaikiran


On 27/01/18 8:40 AM, Jaikiran Pai wrote:

> Here's an update on where this effort now stands. As of yesterday, I
> have the basic minimal functionality that I had planned for this (new)
> task ready. There are other enhancements that this task will need as
> we go along but at this point, this should be usable.
>
> While working on this new task, I kept thinking whether it would make
> more sense to just have this task as a separate project under the Ant
> umbrella and have its own release/versioning cycle. Plus maybe a bit
> of ease in building/testing it given that it won't need some of the
> conditional logic that we do in Ant build itself when it comes to such
> tasks. It did look like a good idea to separate it out but ultimately
> I decided _not_ to do it mainly for the reason that I think it makes
> more sense to have a task in Ant which allows users to write tests
> (using a modern test framework) and test them, right out of the box of
> an Ant installation. This first class experience, IMO, outweighs all
> the other "advantages" that seem to come with having this as a
> separate project.
>
> Coming to the task itself, the task is called "junitlauncher". In the
> current state of this task[1], the following goals/features are
> implemented and functional:
>
> - Ability to launch the JUnit 5 platform
>
> - Ability to specify a classpath to use for the launched tests
>
> - Ability to specify one or more single test classes that need to be
> run as part of this launch
>
> - Ability to specify specific methods on these test classses that need
> to be run as part of the launch
>
> - Ability to specify "batch tests" which essentially is a way to use
> Ant's resource collections to pattern match files that should be
> passed on to the JUnit platform to be evaluated and run as test cases.
>
> - Ability to specify "listeners" for tests. These listeners are
> expected to implement (intentionally) JUnit platform's interface and
> _not_ any of Ant's interfaces.
>
> - Test result formatters are implemented as "listeners". This task
> comes with 2 (for now) implementations out of the box, "plain" and
> "brief". The idea behind these 2 is the same as that of what we
> current have with the "junit" task formatters.
>
> - Users can define custom formatters as "listeners" by specifying a
> class which implements the JUnit's test listener plus (optionally) one
> of Ant's own (new) custom interface. This custom interface will let
> them have access to output stream to which they might want to write
> out the results.
>
> (I might be missing a few more details, but these are the major
> functional features. The manual that I plan to write, will have the
> whole details)
>
> In its current state the task should be able to run both "vintage"
> (JUnit 4.x) and "jupiter" (JUnit 5.x) based tests.
>
> TODO/enhancements for later (a few days/weeks down the line):
>
> - Ability to "fork" these tests in a separate JVM. I haven't fully
> thought about this and might need some inputs on whether we need this
> or not. If we do add this, I might do it slightly differently that
> what we current do with "junit" where there are numerous attributes to
> the task/test elements which are only applicable if fork mode is
> enabled. I might perhaps just introduce a new element within the
> task's element which specifically is meant to deal with any forked VM
> characteristics. But that's something I will get to after the other
> easier enhancements are done.
>
> - Provide the XML formatter out of the box. I was planning to do this
> in the first version itself, but I haven't had enough time to
> understand the schema of this XML plus whether or not the details that
> we put in here are available through the new JUnit launcher APIs.
>
> - Ability to more specifically say which JUnit test engine needs to be
> used for the tests. Right now, the classpath decides which engine gets
> used and thus which classes are considered as tests. If the classpath
> has both "vintage" and "jupiter" engines then such tests will be run.
> However, I want users to be able to say "just use jupiter engine for
> these tests" without having to worry about checking if the classpath
> is polluted with some other engines.
>
> - Make the current "junitreport" task be usable with the XML results
> of this "junitlauncher" task. I haven't yet got to this but this
> should be doable I think, of course once the XML formatter itself is
> ready and functional. I don't plan to introduce a new task for this
> and instead plan to reuse/enhance the existing junitreport task to
> work seamlessly both with existing "junit" task and the
> "junitlauncher" task.
>
> (Few other minor enhancements here and there and any user feedback
> reports)
>
> For those of you curious to see what the task usage is going to look
> like, here's an example[2] build file which shows its many usages.
> That file will be cleaned up a bit, before I send a PR for review/merge.
>
> [1] https://github.com/apache/ant/compare/master...jaikiran:junit5
>
> [2]
> https://github.com/apache/ant/compare/master...jaikiran:junit5#diff-3e0911ed925129e68c543aca0abc5d7aR52
>
> -Jaikiran
>
>
> On 14/12/17 5:14 PM, Stefan Bodewig wrote:
>> On 2017-12-14, Jaikiran Pai wrote:
>>
>>> With that context, I would like to explain what I have attempted so
>>> far and I would like inputs on whether I should follow this path
>>> and/or if there are other suggestions on how we should go ahead. As
>>> you are probably aware, the JUnitTask (the Ant task backing the
>>> <junit> element) has very complex logic which deals with launching the
>>> test runners responsible for running JUnit tests (3.x or 4.x). IMO,
>>> most this logic can now be handed off to the JUnit 5 platform
>>> launcher, using their APIs.
>> In addition some of the complexity inside of JUnitTask stems from the
>> fact that it must not refer to any JUnit 4.x API in order to be usable
>> in a pure JUnit 3.x environment. If you try to add JUnit 5 on top of
>> that this sounds diffciult.
>>
>> I too had toyed with the idea of adding JUnit5 support but it is quite a
>> bit further down my TODO list and I've not looked into JUnit5 as deeply
>> as you did. What you've got so far looks promising, please please push
>> ahead.
>>
>> My idea was to create a completely new task and mainly try to keep the
>> XML output the way it has been for some fifteen years so junitreport
>> would still work. I even though about placing it in a separate antlib
>> rather than Ant's core to allow it get released independently.
>>
>> My vote would go to a separate task.
>>
>> Stefan
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Stefan Bodewig
In reply to this post by Jaikiran Pai
On 2018-01-27, Jaikiran Pai wrote:

> Here's an update on where this effort now stands. As of yesterday, I
> have the basic minimal functionality that I had planned for this (new)
> task ready. There are other enhancements that this task will need as
> we go along but at this point, this should be usable.

I'm not really familiar enough with junit5 and in particualr not with
its internals to perform a review of your code. I can try to spot simple
mistakes but won't be able to see whether you are using junit5's API
correctly.

> While working on this new task, I kept thinking whether it would make
> more sense to just have this task as a separate project under the Ant
> umbrella and have its own release/versioning cycle. Plus maybe a bit
> of ease in building/testing it given that it won't need some of the
> conditional logic that we do in Ant build itself when it comes to such
> tasks.

This is what I would have expected to happen :-)

Apart from what you mention, it may also make it easier for people who
want to contribute without having to get a full built of Ant itself
going.

> It did look like a good idea to separate it out but ultimately I
> decided _not_ to do it mainly for the reason that I think it makes
> more sense to have a task in Ant which allows users to write tests
> (using a modern test framework) and test them, right out of the box of
> an Ant installation.

Understood.

> ...

> TODOs

> ...

> - Ability to "fork" these tests in a separate JVM. I haven't fully
> thought about this and might need some inputs on whether we need this
> or not.

We probably do unless the junit5 runner can do that for us. You may want
to use different JVM settings or just be able to allow tests to crash
without crashing the build.

> - Make the current "junitreport" task be usable with the XML results
> of this "junitlauncher" task. I haven't yet got to this but this
> should be doable I think, of course once the XML formatter itself is
> ready and functional.

If you create the same kind of XML things will just work.

Many thanks for the time you put in, this is great

        Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Jaikiran Pai
In reply to this post by Jaikiran Pai
I'm now done with the initial goals that I had in mind for this task.
I've opened a PR[1] for review. I've included a manual for this task and
it can be currently found here[2]. I would suggest reading the manual
first, before reviewing the PR, since the manual will give an overall
idea of what's being attempted with this task.

Except for the "fork" mode which is available in our existing junit task
and which was was in the TODO list, in my previous mail, the rest have
been implemented in this version. I plan to focus on the "fork" mode
after this task is made available to users and any feedback received.

I have tried to address some of the (internal) technicalities, based on
some discussions that I have seen about JUnit, XML reporter and such,
are around OOMs issues when it comes to sysout and syserr handling.
Result formatters, which can be configured as listeners, are allowed to
say "sendSysOut" and/or "sendSysErr". In such cases, the task will
(internally) redirect the sysout/syserr to a PipedOutputStream (which is
backed by a PipedInputStream). The task also sets up 2 separate threads
- one which reads the PipedInputStream and "feeds" it to the second
thread which "delivers" this content to such result formatters. The
necessity to use a separate thread to deliver the content of
sysout/syserr is so that we don't run into deadlocks (as noted in the
Javadoc of Piped(output/input)Stream class) if/when the result
formatters themselves write something out to sysout/syserr when they are
working on this incoming sysout/syserr content.

We have 3 pre-defined formatters all of which are capable of receiving
this streamed sysout/syserr data. Each of these do _not_ hold on to this
sysout/syserr data in-memory during the execution and instead stream it
out a temporary file. Once the execution completes and it's time to
present the result, these formatters stream/read back the content from
the temp file and write it out in a formatted manner to the target
report files.

Furthermore, the XML formatter doesn't use DOM and instead is based on
Stax for writing out the report. However, even with all this, there's
one place where I haven't yet been able to avoid reading the whole
generated sysout/syserr data into memory (thus potentially triggering an
OOM) - it's the XMLStreamWriter API which for writing out a CDATA
section (which is what we use for sysout/syserr content) in XML, expects
the entire String object. I'll have to see if there are ways to avoid
it, but I think this still is an improvement since this loaded content
is only held on for a short while in memory, during the report writing
and will be immediately garbage collected once that's done.

Please review the PR - mostly the Ant specific constructs and
implementation details.

[1] https://github.com/apache/ant/pull/60

[2]
https://builds.apache.org/job/Ant-Build-Jaikiran/ws/manual/Tasks/junitlauncher.html

[3]
https://github.com/apache/ant/pull/60/files#diff-f985f89ce4779eff2cdcf164d09d5396R308

-Jaikiran




On 27/01/18 8:40 AM, Jaikiran Pai wrote:

> Here's an update on where this effort now stands. As of yesterday, I
> have the basic minimal functionality that I had planned for this (new)
> task ready. There are other enhancements that this task will need as
> we go along but at this point, this should be usable.
>
> While working on this new task, I kept thinking whether it would make
> more sense to just have this task as a separate project under the Ant
> umbrella and have its own release/versioning cycle. Plus maybe a bit
> of ease in building/testing it given that it won't need some of the
> conditional logic that we do in Ant build itself when it comes to such
> tasks. It did look like a good idea to separate it out but ultimately
> I decided _not_ to do it mainly for the reason that I think it makes
> more sense to have a task in Ant which allows users to write tests
> (using a modern test framework) and test them, right out of the box of
> an Ant installation. This first class experience, IMO, outweighs all
> the other "advantages" that seem to come with having this as a
> separate project.
>
> Coming to the task itself, the task is called "junitlauncher". In the
> current state of this task[1], the following goals/features are
> implemented and functional:
>
> - Ability to launch the JUnit 5 platform
>
> - Ability to specify a classpath to use for the launched tests
>
> - Ability to specify one or more single test classes that need to be
> run as part of this launch
>
> - Ability to specify specific methods on these test classses that need
> to be run as part of the launch
>
> - Ability to specify "batch tests" which essentially is a way to use
> Ant's resource collections to pattern match files that should be
> passed on to the JUnit platform to be evaluated and run as test cases.
>
> - Ability to specify "listeners" for tests. These listeners are
> expected to implement (intentionally) JUnit platform's interface and
> _not_ any of Ant's interfaces.
>
> - Test result formatters are implemented as "listeners". This task
> comes with 2 (for now) implementations out of the box, "plain" and
> "brief". The idea behind these 2 is the same as that of what we
> current have with the "junit" task formatters.
>
> - Users can define custom formatters as "listeners" by specifying a
> class which implements the JUnit's test listener plus (optionally) one
> of Ant's own (new) custom interface. This custom interface will let
> them have access to output stream to which they might want to write
> out the results.
>
> (I might be missing a few more details, but these are the major
> functional features. The manual that I plan to write, will have the
> whole details)
>
> In its current state the task should be able to run both "vintage"
> (JUnit 4.x) and "jupiter" (JUnit 5.x) based tests.
>
> TODO/enhancements for later (a few days/weeks down the line):
>
> - Ability to "fork" these tests in a separate JVM. I haven't fully
> thought about this and might need some inputs on whether we need this
> or not. If we do add this, I might do it slightly differently that
> what we current do with "junit" where there are numerous attributes to
> the task/test elements which are only applicable if fork mode is
> enabled. I might perhaps just introduce a new element within the
> task's element which specifically is meant to deal with any forked VM
> characteristics. But that's something I will get to after the other
> easier enhancements are done.
>
> - Provide the XML formatter out of the box. I was planning to do this
> in the first version itself, but I haven't had enough time to
> understand the schema of this XML plus whether or not the details that
> we put in here are available through the new JUnit launcher APIs.
>
> - Ability to more specifically say which JUnit test engine needs to be
> used for the tests. Right now, the classpath decides which engine gets
> used and thus which classes are considered as tests. If the classpath
> has both "vintage" and "jupiter" engines then such tests will be run.
> However, I want users to be able to say "just use jupiter engine for
> these tests" without having to worry about checking if the classpath
> is polluted with some other engines.
>
> - Make the current "junitreport" task be usable with the XML results
> of this "junitlauncher" task. I haven't yet got to this but this
> should be doable I think, of course once the XML formatter itself is
> ready and functional. I don't plan to introduce a new task for this
> and instead plan to reuse/enhance the existing junitreport task to
> work seamlessly both with existing "junit" task and the
> "junitlauncher" task.
>
> (Few other minor enhancements here and there and any user feedback
> reports)
>
> For those of you curious to see what the task usage is going to look
> like, here's an example[2] build file which shows its many usages.
> That file will be cleaned up a bit, before I send a PR for review/merge.
>
> [1] https://github.com/apache/ant/compare/master...jaikiran:junit5
>
> [2]
> https://github.com/apache/ant/compare/master...jaikiran:junit5#diff-3e0911ed925129e68c543aca0abc5d7aR52
>
> -Jaikiran
>
>
> On 14/12/17 5:14 PM, Stefan Bodewig wrote:
>> On 2017-12-14, Jaikiran Pai wrote:
>>
>>> With that context, I would like to explain what I have attempted so
>>> far and I would like inputs on whether I should follow this path
>>> and/or if there are other suggestions on how we should go ahead. As
>>> you are probably aware, the JUnitTask (the Ant task backing the
>>> <junit> element) has very complex logic which deals with launching the
>>> test runners responsible for running JUnit tests (3.x or 4.x). IMO,
>>> most this logic can now be handed off to the JUnit 5 platform
>>> launcher, using their APIs.
>> In addition some of the complexity inside of JUnitTask stems from the
>> fact that it must not refer to any JUnit 4.x API in order to be usable
>> in a pure JUnit 3.x environment. If you try to add JUnit 5 on top of
>> that this sounds diffciult.
>>
>> I too had toyed with the idea of adding JUnit5 support but it is quite a
>> bit further down my TODO list and I've not looked into JUnit5 as deeply
>> as you did. What you've got so far looks promising, please please push
>> ahead.
>>
>> My idea was to create a completely new task and mainly try to keep the
>> XML output the way it has been for some fifteen years so junitreport
>> would still work. I even though about placing it in a separate antlib
>> rather than Ant's core to allow it get released independently.
>>
>> My vote would go to a separate task.
>>
>> Stefan
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Jaikiran Pai
 From a release point of view - I know we are currently sorting out some
regressions and issues that have been uncovered by the latest release.
So if it makes things easier from a stability point of view, we can wait
to merge this only after we have done another release which addresses
only the regressions that we have found in our latest releases.

-Jaikiran


On 15/02/18 7:21 PM, Jaikiran Pai wrote:

> I'm now done with the initial goals that I had in mind for this task.
> I've opened a PR[1] for review. I've included a manual for this task
> and it can be currently found here[2]. I would suggest reading the
> manual first, before reviewing the PR, since the manual will give an
> overall idea of what's being attempted with this task.
>
> Except for the "fork" mode which is available in our existing junit
> task and which was was in the TODO list, in my previous mail, the rest
> have been implemented in this version. I plan to focus on the "fork"
> mode after this task is made available to users and any feedback
> received.
>
> I have tried to address some of the (internal) technicalities, based
> on some discussions that I have seen about JUnit, XML reporter and
> such, are around OOMs issues when it comes to sysout and syserr
> handling. Result formatters, which can be configured as listeners, are
> allowed to say "sendSysOut" and/or "sendSysErr". In such cases, the
> task will (internally) redirect the sysout/syserr to a
> PipedOutputStream (which is backed by a PipedInputStream). The task
> also sets up 2 separate threads - one which reads the PipedInputStream
> and "feeds" it to the second thread which "delivers" this content to
> such result formatters. The necessity to use a separate thread to
> deliver the content of sysout/syserr is so that we don't run into
> deadlocks (as noted in the Javadoc of Piped(output/input)Stream class)
> if/when the result formatters themselves write something out to
> sysout/syserr when they are working on this incoming sysout/syserr
> content.
>
> We have 3 pre-defined formatters all of which are capable of receiving
> this streamed sysout/syserr data. Each of these do _not_ hold on to
> this sysout/syserr data in-memory during the execution and instead
> stream it out a temporary file. Once the execution completes and it's
> time to present the result, these formatters stream/read back the
> content from the temp file and write it out in a formatted manner to
> the target report files.
>
> Furthermore, the XML formatter doesn't use DOM and instead is based on
> Stax for writing out the report. However, even with all this, there's
> one place where I haven't yet been able to avoid reading the whole
> generated sysout/syserr data into memory (thus potentially triggering
> an OOM) - it's the XMLStreamWriter API which for writing out a CDATA
> section (which is what we use for sysout/syserr content) in XML,
> expects the entire String object. I'll have to see if there are ways
> to avoid it, but I think this still is an improvement since this
> loaded content is only held on for a short while in memory, during the
> report writing and will be immediately garbage collected once that's
> done.
>
> Please review the PR - mostly the Ant specific constructs and
> implementation details.
>
> [1] https://github.com/apache/ant/pull/60
>
> [2]
> https://builds.apache.org/job/Ant-Build-Jaikiran/ws/manual/Tasks/junitlauncher.html
>
> [3]
> https://github.com/apache/ant/pull/60/files#diff-f985f89ce4779eff2cdcf164d09d5396R308
>
> -Jaikiran
>
>
>
>
> On 27/01/18 8:40 AM, Jaikiran Pai wrote:
>> Here's an update on where this effort now stands. As of yesterday, I
>> have the basic minimal functionality that I had planned for this
>> (new) task ready. There are other enhancements that this task will
>> need as we go along but at this point, this should be usable.
>>
>> While working on this new task, I kept thinking whether it would make
>> more sense to just have this task as a separate project under the Ant
>> umbrella and have its own release/versioning cycle. Plus maybe a bit
>> of ease in building/testing it given that it won't need some of the
>> conditional logic that we do in Ant build itself when it comes to
>> such tasks. It did look like a good idea to separate it out but
>> ultimately I decided _not_ to do it mainly for the reason that I
>> think it makes more sense to have a task in Ant which allows users to
>> write tests (using a modern test framework) and test them, right out
>> of the box of an Ant installation. This first class experience, IMO,
>> outweighs all the other "advantages" that seem to come with having
>> this as a separate project.
>>
>> Coming to the task itself, the task is called "junitlauncher". In the
>> current state of this task[1], the following goals/features are
>> implemented and functional:
>>
>> - Ability to launch the JUnit 5 platform
>>
>> - Ability to specify a classpath to use for the launched tests
>>
>> - Ability to specify one or more single test classes that need to be
>> run as part of this launch
>>
>> - Ability to specify specific methods on these test classses that
>> need to be run as part of the launch
>>
>> - Ability to specify "batch tests" which essentially is a way to use
>> Ant's resource collections to pattern match files that should be
>> passed on to the JUnit platform to be evaluated and run as test cases.
>>
>> - Ability to specify "listeners" for tests. These listeners are
>> expected to implement (intentionally) JUnit platform's interface and
>> _not_ any of Ant's interfaces.
>>
>> - Test result formatters are implemented as "listeners". This task
>> comes with 2 (for now) implementations out of the box, "plain" and
>> "brief". The idea behind these 2 is the same as that of what we
>> current have with the "junit" task formatters.
>>
>> - Users can define custom formatters as "listeners" by specifying a
>> class which implements the JUnit's test listener plus (optionally)
>> one of Ant's own (new) custom interface. This custom interface will
>> let them have access to output stream to which they might want to
>> write out the results.
>>
>> (I might be missing a few more details, but these are the major
>> functional features. The manual that I plan to write, will have the
>> whole details)
>>
>> In its current state the task should be able to run both "vintage"
>> (JUnit 4.x) and "jupiter" (JUnit 5.x) based tests.
>>
>> TODO/enhancements for later (a few days/weeks down the line):
>>
>> - Ability to "fork" these tests in a separate JVM. I haven't fully
>> thought about this and might need some inputs on whether we need this
>> or not. If we do add this, I might do it slightly differently that
>> what we current do with "junit" where there are numerous attributes
>> to the task/test elements which are only applicable if fork mode is
>> enabled. I might perhaps just introduce a new element within the
>> task's element which specifically is meant to deal with any forked VM
>> characteristics. But that's something I will get to after the other
>> easier enhancements are done.
>>
>> - Provide the XML formatter out of the box. I was planning to do this
>> in the first version itself, but I haven't had enough time to
>> understand the schema of this XML plus whether or not the details
>> that we put in here are available through the new JUnit launcher APIs.
>>
>> - Ability to more specifically say which JUnit test engine needs to
>> be used for the tests. Right now, the classpath decides which engine
>> gets used and thus which classes are considered as tests. If the
>> classpath has both "vintage" and "jupiter" engines then such tests
>> will be run. However, I want users to be able to say "just use
>> jupiter engine for these tests" without having to worry about
>> checking if the classpath is polluted with some other engines.
>>
>> - Make the current "junitreport" task be usable with the XML results
>> of this "junitlauncher" task. I haven't yet got to this but this
>> should be doable I think, of course once the XML formatter itself is
>> ready and functional. I don't plan to introduce a new task for this
>> and instead plan to reuse/enhance the existing junitreport task to
>> work seamlessly both with existing "junit" task and the
>> "junitlauncher" task.
>>
>> (Few other minor enhancements here and there and any user feedback
>> reports)
>>
>> For those of you curious to see what the task usage is going to look
>> like, here's an example[2] build file which shows its many usages.
>> That file will be cleaned up a bit, before I send a PR for review/merge.
>>
>> [1] https://github.com/apache/ant/compare/master...jaikiran:junit5
>>
>> [2]
>> https://github.com/apache/ant/compare/master...jaikiran:junit5#diff-3e0911ed925129e68c543aca0abc5d7aR52
>>
>> -Jaikiran
>>
>>
>> On 14/12/17 5:14 PM, Stefan Bodewig wrote:
>>> On 2017-12-14, Jaikiran Pai wrote:
>>>
>>>> With that context, I would like to explain what I have attempted so
>>>> far and I would like inputs on whether I should follow this path
>>>> and/or if there are other suggestions on how we should go ahead. As
>>>> you are probably aware, the JUnitTask (the Ant task backing the
>>>> <junit> element) has very complex logic which deals with launching the
>>>> test runners responsible for running JUnit tests (3.x or 4.x). IMO,
>>>> most this logic can now be handed off to the JUnit 5 platform
>>>> launcher, using their APIs.
>>> In addition some of the complexity inside of JUnitTask stems from the
>>> fact that it must not refer to any JUnit 4.x API in order to be usable
>>> in a pure JUnit 3.x environment. If you try to add JUnit 5 on top of
>>> that this sounds diffciult.
>>>
>>> I too had toyed with the idea of adding JUnit5 support but it is
>>> quite a
>>> bit further down my TODO list and I've not looked into JUnit5 as deeply
>>> as you did. What you've got so far looks promising, please please push
>>> ahead.
>>>
>>> My idea was to create a completely new task and mainly try to keep the
>>> XML output the way it has been for some fifteen years so junitreport
>>> would still work. I even though about placing it in a separate antlib
>>> rather than Ant's core to allow it get released independently.
>>>
>>> My vote would go to a separate task.
>>>
>>> Stefan
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Jaikiran Pai
In reply to this post by Jaikiran Pai
I just realized that the manual that I linked in my mail isn't
accessible to everyone. So here's an alternative link to the same
https://home.apache.org/~jaikiran/temp_workspace/manual/Tasks/junitlauncher.html

-Jaikiran


On 15/02/18 7:21 PM, Jaikiran Pai wrote:

> I'm now done with the initial goals that I had in mind for this task.
> I've opened a PR[1] for review. I've included a manual for this task
> and it can be currently found here[2]. I would suggest reading the
> manual first, before reviewing the PR, since the manual will give an
> overall idea of what's being attempted with this task.
>
> Except for the "fork" mode which is available in our existing junit
> task and which was was in the TODO list, in my previous mail, the rest
> have been implemented in this version. I plan to focus on the "fork"
> mode after this task is made available to users and any feedback
> received.
>
> I have tried to address some of the (internal) technicalities, based
> on some discussions that I have seen about JUnit, XML reporter and
> such, are around OOMs issues when it comes to sysout and syserr
> handling. Result formatters, which can be configured as listeners, are
> allowed to say "sendSysOut" and/or "sendSysErr". In such cases, the
> task will (internally) redirect the sysout/syserr to a
> PipedOutputStream (which is backed by a PipedInputStream). The task
> also sets up 2 separate threads - one which reads the PipedInputStream
> and "feeds" it to the second thread which "delivers" this content to
> such result formatters. The necessity to use a separate thread to
> deliver the content of sysout/syserr is so that we don't run into
> deadlocks (as noted in the Javadoc of Piped(output/input)Stream class)
> if/when the result formatters themselves write something out to
> sysout/syserr when they are working on this incoming sysout/syserr
> content.
>
> We have 3 pre-defined formatters all of which are capable of receiving
> this streamed sysout/syserr data. Each of these do _not_ hold on to
> this sysout/syserr data in-memory during the execution and instead
> stream it out a temporary file. Once the execution completes and it's
> time to present the result, these formatters stream/read back the
> content from the temp file and write it out in a formatted manner to
> the target report files.
>
> Furthermore, the XML formatter doesn't use DOM and instead is based on
> Stax for writing out the report. However, even with all this, there's
> one place where I haven't yet been able to avoid reading the whole
> generated sysout/syserr data into memory (thus potentially triggering
> an OOM) - it's the XMLStreamWriter API which for writing out a CDATA
> section (which is what we use for sysout/syserr content) in XML,
> expects the entire String object. I'll have to see if there are ways
> to avoid it, but I think this still is an improvement since this
> loaded content is only held on for a short while in memory, during the
> report writing and will be immediately garbage collected once that's
> done.
>
> Please review the PR - mostly the Ant specific constructs and
> implementation details.
>
> [1] https://github.com/apache/ant/pull/60
>
> [2]
> https://builds.apache.org/job/Ant-Build-Jaikiran/ws/manual/Tasks/junitlauncher.html
>
> [3]
> https://github.com/apache/ant/pull/60/files#diff-f985f89ce4779eff2cdcf164d09d5396R308
>
> -Jaikiran
>
>
>
>
> On 27/01/18 8:40 AM, Jaikiran Pai wrote:
>> Here's an update on where this effort now stands. As of yesterday, I
>> have the basic minimal functionality that I had planned for this
>> (new) task ready. There are other enhancements that this task will
>> need as we go along but at this point, this should be usable.
>>
>> While working on this new task, I kept thinking whether it would make
>> more sense to just have this task as a separate project under the Ant
>> umbrella and have its own release/versioning cycle. Plus maybe a bit
>> of ease in building/testing it given that it won't need some of the
>> conditional logic that we do in Ant build itself when it comes to
>> such tasks. It did look like a good idea to separate it out but
>> ultimately I decided _not_ to do it mainly for the reason that I
>> think it makes more sense to have a task in Ant which allows users to
>> write tests (using a modern test framework) and test them, right out
>> of the box of an Ant installation. This first class experience, IMO,
>> outweighs all the other "advantages" that seem to come with having
>> this as a separate project.
>>
>> Coming to the task itself, the task is called "junitlauncher". In the
>> current state of this task[1], the following goals/features are
>> implemented and functional:
>>
>> - Ability to launch the JUnit 5 platform
>>
>> - Ability to specify a classpath to use for the launched tests
>>
>> - Ability to specify one or more single test classes that need to be
>> run as part of this launch
>>
>> - Ability to specify specific methods on these test classses that
>> need to be run as part of the launch
>>
>> - Ability to specify "batch tests" which essentially is a way to use
>> Ant's resource collections to pattern match files that should be
>> passed on to the JUnit platform to be evaluated and run as test cases.
>>
>> - Ability to specify "listeners" for tests. These listeners are
>> expected to implement (intentionally) JUnit platform's interface and
>> _not_ any of Ant's interfaces.
>>
>> - Test result formatters are implemented as "listeners". This task
>> comes with 2 (for now) implementations out of the box, "plain" and
>> "brief". The idea behind these 2 is the same as that of what we
>> current have with the "junit" task formatters.
>>
>> - Users can define custom formatters as "listeners" by specifying a
>> class which implements the JUnit's test listener plus (optionally)
>> one of Ant's own (new) custom interface. This custom interface will
>> let them have access to output stream to which they might want to
>> write out the results.
>>
>> (I might be missing a few more details, but these are the major
>> functional features. The manual that I plan to write, will have the
>> whole details)
>>
>> In its current state the task should be able to run both "vintage"
>> (JUnit 4.x) and "jupiter" (JUnit 5.x) based tests.
>>
>> TODO/enhancements for later (a few days/weeks down the line):
>>
>> - Ability to "fork" these tests in a separate JVM. I haven't fully
>> thought about this and might need some inputs on whether we need this
>> or not. If we do add this, I might do it slightly differently that
>> what we current do with "junit" where there are numerous attributes
>> to the task/test elements which are only applicable if fork mode is
>> enabled. I might perhaps just introduce a new element within the
>> task's element which specifically is meant to deal with any forked VM
>> characteristics. But that's something I will get to after the other
>> easier enhancements are done.
>>
>> - Provide the XML formatter out of the box. I was planning to do this
>> in the first version itself, but I haven't had enough time to
>> understand the schema of this XML plus whether or not the details
>> that we put in here are available through the new JUnit launcher APIs.
>>
>> - Ability to more specifically say which JUnit test engine needs to
>> be used for the tests. Right now, the classpath decides which engine
>> gets used and thus which classes are considered as tests. If the
>> classpath has both "vintage" and "jupiter" engines then such tests
>> will be run. However, I want users to be able to say "just use
>> jupiter engine for these tests" without having to worry about
>> checking if the classpath is polluted with some other engines.
>>
>> - Make the current "junitreport" task be usable with the XML results
>> of this "junitlauncher" task. I haven't yet got to this but this
>> should be doable I think, of course once the XML formatter itself is
>> ready and functional. I don't plan to introduce a new task for this
>> and instead plan to reuse/enhance the existing junitreport task to
>> work seamlessly both with existing "junit" task and the
>> "junitlauncher" task.
>>
>> (Few other minor enhancements here and there and any user feedback
>> reports)
>>
>> For those of you curious to see what the task usage is going to look
>> like, here's an example[2] build file which shows its many usages.
>> That file will be cleaned up a bit, before I send a PR for review/merge.
>>
>> [1] https://github.com/apache/ant/compare/master...jaikiran:junit5
>>
>> [2]
>> https://github.com/apache/ant/compare/master...jaikiran:junit5#diff-3e0911ed925129e68c543aca0abc5d7aR52
>>
>> -Jaikiran
>>
>>
>> On 14/12/17 5:14 PM, Stefan Bodewig wrote:
>>> On 2017-12-14, Jaikiran Pai wrote:
>>>
>>>> With that context, I would like to explain what I have attempted so
>>>> far and I would like inputs on whether I should follow this path
>>>> and/or if there are other suggestions on how we should go ahead. As
>>>> you are probably aware, the JUnitTask (the Ant task backing the
>>>> <junit> element) has very complex logic which deals with launching the
>>>> test runners responsible for running JUnit tests (3.x or 4.x). IMO,
>>>> most this logic can now be handed off to the JUnit 5 platform
>>>> launcher, using their APIs.
>>> In addition some of the complexity inside of JUnitTask stems from the
>>> fact that it must not refer to any JUnit 4.x API in order to be usable
>>> in a pure JUnit 3.x environment. If you try to add JUnit 5 on top of
>>> that this sounds diffciult.
>>>
>>> I too had toyed with the idea of adding JUnit5 support but it is
>>> quite a
>>> bit further down my TODO list and I've not looked into JUnit5 as deeply
>>> as you did. What you've got so far looks promising, please please push
>>> ahead.
>>>
>>> My idea was to create a completely new task and mainly try to keep the
>>> XML output the way it has been for some fifteen years so junitreport
>>> would still work. I even though about placing it in a separate antlib
>>> rather than Ant's core to allow it get released independently.
>>>
>>> My vote would go to a separate task.
>>>
>>> Stefan
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Stefan Bodewig
In reply to this post by Jaikiran Pai
On 2018-02-15, Jaikiran Pai wrote:

> I'm now done with the initial goals that I had in mind for this
> task. I've opened a PR[1] for review. I've included a manual for this
> task and it can be currently found here[2]. I would suggest reading
> the manual first, before reviewing the PR, since the manual will give
> an overall idea of what's being attempted with this task.

It may be a while until I manage to look at it - in particular since I
need to "read the manual first" :-)

The review of the Java8 refactoring patch is going to take a bit longer
still, but I expect to finish it this weekend (am already at
src/main/org/apache/tools/ant/types/mappers). Whether we want to include
the new task with an upcoming release depends on how comfortable you
feel with it from my point of view. It probably won't see much use
unless we include it in a release.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Jaikiran Pai
On 17/02/18 12:51 AM, Stefan Bodewig wrote:
> On 2018-02-15, Jaikiran Pai wrote:
>
>> I'm now done with the initial goals that I had in mind for this
>> task. I've opened a PR[1] for review. I've included a manual for this
>> task and it can be currently found here[2]. I would suggest reading
>> the manual first, before reviewing the PR, since the manual will give
>> an overall idea of what's being attempted with this task.
> It may be a while until I manage to look at it - in particular since I
> need to "read the manual first" :-)
Please take your time. Although I am happy with the way this task has
shaped up, the one area I would like to have your feedback is the
sysout/syserr handling implementation in that PR. The intention is to
try and avoid potential OOMs and other such issues, that have been noted
in our bug tracker, but at the same time I don't want to end up having
an overly complicated logic or something that could have been done
better in a different way.
>
> Whether we want to include
> the new task with an upcoming release depends on how comfortable you
> feel with it from my point of view. It probably won't see much use
> unless we include it in a release.
>

Given the changes, I am confident that it won't break existing tasks or
anything else in Ant. I also believe that this new task will help people
get started with using JUnit 5 with Ant to an extent that they can get
the tests run (both JUnit 4 and 5) with basic reporting features and the
ability to use the junitreport for the generated XML reports. So I think
it's in a good enough state to start off with. I do plan to try this
task as a "user" myself this weekend, outside of our testsuite and see
if there are any issues or anything basic missing.

-Jaikiran

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Stefan Bodewig
In reply to this post by Jaikiran Pai
On 2018-02-15, Jaikiran Pai wrote:

> I'm now done with the initial goals that I had in mind for this
> task. I've opened a PR[1] for review. I've included a manual for this
> task and it can be currently found here[2]. I would suggest reading
> the manual first, before reviewing the PR, since the manual will give
> an overall idea of what's being attempted with this task.

> Except for the "fork" mode which is available in our existing junit
> task and which was was in the TODO list, in my previous mail, the rest
> have been implemented in this version. I plan to focus on the "fork"
> mode after this task is made available to users and any feedback
> received.

I'm through with the manual and can see how the lack of fork is a real
limitation. Setting the classloaders up may be a pain with several
classes of Ant leaking into the test's classpath. IMHO forking new VMs
is the only clean way of running tests. And then there are timeouts or
tests that crash the VM, memory settings and all that.

This is not to say the task shouldn't be merged without the fork
feature, I just expect it to get requested pretty soon :-)

> We have 3 pre-defined formatters all of which are capable of receiving
> this streamed sysout/syserr data. Each of these do _not_ hold on to
> this sysout/syserr data in-memory during the execution and instead
> stream it out a temporary file. Once the execution completes and it's
> time to present the result, these formatters stream/read back the
> content from the temp file and write it out in a formatted manner to
> the target report files.

Sounds expensive.

> Furthermore, the XML formatter doesn't use DOM and instead is based on
> Stax for writing out the report. However, even with all this, there's
> one place where I haven't yet been able to avoid reading the whole
> generated sysout/syserr data into memory (thus potentially triggering
> an OOM) - it's the XMLStreamWriter API which for writing out a CDATA
> section (which is what we use for sysout/syserr content) in XML,
> expects the entire String object. I'll have to see if there are ways
> to avoid it, but I think this still is an improvement since this
> loaded content is only held on for a short while in memory, during the
> report writing and will be immediately garbage collected once that's
> done.

CDATA is really just an implementation detail. The stylesheets should
work with properly escaped nested text as well. So as an alternative you
could escape all special characters with entity references and use
writeCharacters.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Jaikiran Pai
On 19/02/18 1:37 AM, Stefan Bodewig wrote:

> On 2018-02-15, Jaikiran Pai wrote:
>
>> I'm now done with the initial goals that I had in mind for this
>> task. I've opened a PR[1] for review. I've included a manual for this
>> task and it can be currently found here[2]. I would suggest reading
>> the manual first, before reviewing the PR, since the manual will give
>> an overall idea of what's being attempted with this task.
>> Except for the "fork" mode which is available in our existing junit
>> task and which was was in the TODO list, in my previous mail, the rest
>> have been implemented in this version. I plan to focus on the "fork"
>> mode after this task is made available to users and any feedback
>> received.
> I'm through with the manual and can see how the lack of fork is a real
> limitation. Setting the classloaders up may be a pain with several
> classes of Ant leaking into the test's classpath. IMHO forking new VMs
> is the only clean way of running tests. And then there are timeouts or
> tests that crash the VM, memory settings and all that.
>
> This is not to say the task shouldn't be merged without the fork
> feature, I just expect it to get requested pretty soon :-)
Agreed. I have started the "fork" mode work, but it's probably going to
take at least acouple of more weekends to get it usable.

>
>> We have 3 pre-defined formatters all of which are capable of receiving
>> this streamed sysout/syserr data. Each of these do _not_ hold on to
>> this sysout/syserr data in-memory during the execution and instead
>> stream it out a temporary file. Once the execution completes and it's
>> time to present the result, these formatters stream/read back the
>> content from the temp file and write it out in a formatted manner to
>> the target report files.
> Sounds expensive.
Indeed. In 2 ways actually:

     1. The creation of these files.
     2. The fact that we write it out to these files (although in a
separate thread) and then read it while formatting the result, finally.

It's only applicable for result formatters which want to have the sysout
and syserr content sent to them, but I expect that most users will want
this enabled anyway. So yes, it's expensive.

The alternate approach is perhaps to have this content maintained
in-memory till a fixed size limit (relatively large enough but not too
much that it will cause OOM) and only create and stream into temporary
files if that size limit is exceeded. That way the temp file creation
and writing/reading perhaps can be avoided/controlled in most of the cases.

>
>> Furthermore, the XML formatter doesn't use DOM and instead is based on
>> Stax for writing out the report. However, even with all this, there's
>> one place where I haven't yet been able to avoid reading the whole
>> generated sysout/syserr data into memory (thus potentially triggering
>> an OOM) - it's the XMLStreamWriter API which for writing out a CDATA
>> section (which is what we use for sysout/syserr content) in XML,
>> expects the entire String object. I'll have to see if there are ways
>> to avoid it, but I think this still is an improvement since this
>> loaded content is only held on for a short while in memory, during the
>> report writing and will be immediately garbage collected once that's
>> done.
> CDATA is really just an implementation detail. The stylesheets should
> work with properly escaped nested text as well. So as an alternative you
> could escape all special characters with entity references and use
> writeCharacters.
>
Thank you - I hadn't thought of it and my mind was just fixed on the
CDATA usage. I have updated that PR to encode the characters and then
use writeCharacters to write them out.

Thanks for the review Stefan.

-Jaikiran


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Ant support for users who use JUnit5

Jaikiran Pai
On 19/02/18 9:23 AM, Jaikiran Pai wrote:

> On 19/02/18 1:37 AM, Stefan Bodewig wrote:
>> On 2018-02-15, Jaikiran Pai wrote:
>>
>>> We have 3 pre-defined formatters all of which are capable of receiving
>>> this streamed sysout/syserr data. Each of these do _not_ hold on to
>>> this sysout/syserr data in-memory during the execution and instead
>>> stream it out a temporary file. Once the execution completes and it's
>>> time to present the result, these formatters stream/read back the
>>> content from the temp file and write it out in a formatted manner to
>>> the target report files.
>> Sounds expensive.
> Indeed. In 2 ways actually:
>
>     1. The creation of these files.
>     2. The fact that we write it out to these files (although in a
> separate thread) and then read it while formatting the result, finally.
>
> It's only applicable for result formatters which want to have the
> sysout and syserr content sent to them, but I expect that most users
> will want this enabled anyway. So yes, it's expensive.
>
> The alternate approach is perhaps to have this content maintained
> in-memory till a fixed size limit (relatively large enough but not too
> much that it will cause OOM) and only create and stream into temporary
> files if that size limit is exceeded. That way the temp file creation
> and writing/reading perhaps can be avoided/controlled in most of the
> cases.
I just updated the PR to include an additional commit which introduces
this idea. The (common internal) formatter class now has a "store" which
is used to store the sysout/syserr content. The store is initially
backed by a fixed size ByteBuffer and the incoming sysout/syserr content
is pushed to that. When/if the size limit is reached, the store is
switched to create a new temporary file and write out the already held
data and any new incoming data. Once the file is created, the in-memory
buffer is destroyed. I've intentionally kept this as a completely
internal implementation detail which means that the size limit of this
internal buffer isn't configurable. I (somewhat randomly) decided to use
50KB as the size limit for (each) sysout/syserr store but am willing to
change it to anything lower (or maybe even higher) that might make sense.

The PR is the same one, that we have been using so far for review
here[1] and the specific commit is this one[2]. I'll be squashing all
those commits into one before merging, once the review is done.

Thanks Stefan for the review so far, it has been of real help and
uncovered some bugs as well as my lack of knowledge of some of the Ant
task creation guidelines.

[1] https://github.com/apache/ant/pull/60
[2]
https://github.com/apache/ant/pull/60/commits/641b873b9e9db5d78c7cd7953da135fb0b9c097e

-Jaikiran


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]