Tests in Surefire

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Tests in Surefire

Gintautas Grigelionis
I believe that in order to execute Ant test suite in Surefire one must
configure basedir which unfortunately affects Ant test projects. Perhaps it
would make sense to have a magic property that (in combination with
detection of Surefire environment) would allow BuildFileRule to call
System.clearProperty("basedir")?

Gintas
Reply | Threaded
Open this post in threaded view
|

Re: Tests in Surefire

Jaikiran Pai-2
Could you tell us a bit more about what is being attempted? Are you
saying you plan to run the tests that are part of the Ant project,
through the mvn command, while building Ant?

-Jaikiran


On 20/10/18 3:31 PM, Gintautas Grigelionis wrote:
> I believe that in order to execute Ant test suite in Surefire one must
> configure basedir which unfortunately affects Ant test projects. Perhaps it
> would make sense to have a magic property that (in combination with
> detection of Surefire environment) would allow BuildFileRule to call
> System.clearProperty("basedir")?
>
> Gintas
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tests in Surefire

Gintautas Grigelionis
Here's what happened: there's a Jenkins job that builds Ant using Maven,
since Ant project has POMs for all components.

I happened to create a new component, and realised I forgot to add it to
the master POM.
While doing that, I saw that the Jenkins job has been broken, and started
digging deeper.
It seems that once upon a time Ant was buildable with Maven including unit
tests
(even though the latter part was cludgy, mainly because the launcher sets
some magical properties,
but also due to limitations in earlier versions of Surefire).

However, running the tests independently through Surefire would help to
improve them,
partly because that would expose all magic that's happening under the cover,
and partly because the exclude/include rules are degraded to the point that
apache-ant Maven module
still tries to execute too many tests, while some tests (like those for ssh
task) seems to be ignored by pure Ant builds.

In order to get this sorted out, I came to the conclusion that I would like
BuildFileTest/BuildFileRule
to clear basedir property based on whether another magic property is set or
not before project.init() in configureProject().
I propose to call it "ant.test.basedir.ignore" or something like that; I
can open a PR for that.

Gintas

On Mon, 22 Oct 2018 at 07:34, Jaikiran Pai <[hidden email]> wrote:

> Could you tell us a bit more about what is being attempted? Are you
> saying you plan to run the tests that are part of the Ant project,
> through the mvn command, while building Ant?
>
> -Jaikiran
>
>
> On 20/10/18 3:31 PM, Gintautas Grigelionis wrote:
> > I believe that in order to execute Ant test suite in Surefire one must
> > configure basedir which unfortunately affects Ant test projects. Perhaps
> it
> > would make sense to have a magic property that (in combination with
> > detection of Surefire environment) would allow BuildFileRule to call
> > System.clearProperty("basedir")?
> >
> > Gintas
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Tests in Surefire

Stefan Bodewig
In reply to this post by Gintautas Grigelionis
On 2018-10-20, Gintautas Grigelionis wrote:

> I believe that in order to execute Ant test suite in Surefire one must
> configure basedir which unfortunately affects Ant test projects.

I'm not sure I share the goal of trying to run the tests via Surefire at
all.

Antoine has been the one who set up the maven poms and I don't remember
us ever discussing what we wanted to achieve there. To me they've been a
vehicle to get decent POMs for the jars we publish to Maven Central, not
more. Here "decent" means "specifies the correct dependencies" and not
even "can be used to compile hte jar". I think it has outgrown my
personal minimal requirement by far, so others must have had bigger
plans :-)

I do not believe we've been aware of Surefire bugs that prevented
running. We've most likely never cared (and I still don't, obviously :-)

> Perhaps it would make sense to have a magic property that (in
> combination with detection of Surefire environment) would allow
> BuildFileRule to call System.clearProperty("basedir")?

If that's all it would take to make them work, that's fine with me. If
we need further modifications to the tests them I'm not sure.

Stefan

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Tests in Surefire

Gintautas Grigelionis
On Tue, 23 Oct 2018 at 12:25, Stefan Bodewig <[hidden email]> wrote:

> On 2018-10-20, Gintautas Grigelionis wrote:
>
> > I believe that in order to execute Ant test suite in Surefire one must
> > configure basedir which unfortunately affects Ant test projects.
>
> I'm not sure I share the goal of trying to run the tests via Surefire at
> all.
>
> Antoine has been the one who set up the maven poms and I don't remember
> us ever discussing what we wanted to achieve there. To me they've been a
> vehicle to get decent POMs for the jars we publish to Maven Central, not
> more. Here "decent" means "specifies the correct dependencies" and not
> even "can be used to compile the jar". I think it has outgrown my
> personal minimal requirement by far, so others must have had bigger
> plans :-)
>
> I do not believe we've been aware of Surefire bugs that prevented
> running. We've most likely never cared (and I still don't, obviously :-)
>

It was in the README :-) the inability to set basedir/workingDir.


> > Perhaps it would make sense to have a magic property that (in
> > combination with detection of Surefire environment) would allow
> > BuildFileRule to call System.clearProperty("basedir")?
>
> If that's all it would take to make them work, that's fine with me. If
> we need further modifications to the tests them I'm not sure.
>

Fair enough. I've hit some snags and I commented on them in the POMs.

Gintas
Reply | Threaded
Open this post in threaded view
|

Re: Tests in Surefire

Gintautas Grigelionis
On Wed, 24 Oct 2018 at 00:29, Gintautas Grigelionis <[hidden email]>
wrote:

> On Tue, 23 Oct 2018 at 12:25, Stefan Bodewig <[hidden email]> wrote:
>
>> On 2018-10-20, Gintautas Grigelionis wrote:
>>
>> > I believe that in order to execute Ant test suite in Surefire one must
>> > configure basedir which unfortunately affects Ant test projects.
>>
>> I'm not sure I share the goal of trying to run the tests via Surefire at
>> all.
>>
>> Antoine has been the one who set up the maven poms and I don't remember
>> us ever discussing what we wanted to achieve there. To me they've been a
>> vehicle to get decent POMs for the jars we publish to Maven Central, not
>> more. Here "decent" means "specifies the correct dependencies" and not
>> even "can be used to compile the jar". I think it has outgrown my
>> personal minimal requirement by far, so others must have had bigger
>> plans :-)
>>
>> I do not believe we've been aware of Surefire bugs that prevented
>> running. We've most likely never cared (and I still don't, obviously :-)
>>
>
> It was in the README :-) the inability to set basedir/workingDir.
>
>
>> > Perhaps it would make sense to have a magic property that (in
>> > combination with detection of Surefire environment) would allow
>> > BuildFileRule to call System.clearProperty("basedir")?
>>
>> If that's all it would take to make them work, that's fine with me. If
>> we need further modifications to the tests them I'm not sure.
>>
>
> Fair enough. I've hit some snags and I commented on them in the POMs.
>

I am at the point now where I must say "this is the farthest I could get
without changing the tests".
Please see the comments in src/etc/poms/ant/pom.xml, especially
testExcludes section.
Some tests, in particular in IncludeTest, are not robust (failure modes
depend on XML parser,
which may throw an exception because encoding is not specified, which was
not the original intention).
AntTest hardcodes directory structure (build/testcases) which apparently
breaks modular builds in Jenkins
(works locally). Also, current test failures in Jenkins apparently are an
oddity of Maven,
which uses ANT_HOME/jre/java on Java 8 or older SDKs precluding use of
javac.

Thanks, Gintas
Reply | Threaded
Open this post in threaded view
|

Re: Tests in Surefire

Gintautas Grigelionis
On Thu, 25 Oct 2018 at 09:05, Gintautas Grigelionis <[hidden email]>
wrote:

> On Wed, 24 Oct 2018 at 00:29, Gintautas Grigelionis <
> [hidden email]> wrote:
>
>> On Tue, 23 Oct 2018 at 12:25, Stefan Bodewig <[hidden email]> wrote:
>>
>>> On 2018-10-20, Gintautas Grigelionis wrote:
>>>
>>> > I believe that in order to execute Ant test suite in Surefire one must
>>> > configure basedir which unfortunately affects Ant test projects.
>>>
>>> I'm not sure I share the goal of trying to run the tests via Surefire at
>>> all.
>>>
>>> Antoine has been the one who set up the maven poms and I don't remember
>>> us ever discussing what we wanted to achieve there. To me they've been a
>>> vehicle to get decent POMs for the jars we publish to Maven Central, not
>>> more. Here "decent" means "specifies the correct dependencies" and not
>>> even "can be used to compile the jar". I think it has outgrown my
>>> personal minimal requirement by far, so others must have had bigger
>>> plans :-)
>>>
>>> I do not believe we've been aware of Surefire bugs that prevented
>>> running. We've most likely never cared (and I still don't, obviously :-)
>>>
>>
>> It was in the README :-) the inability to set basedir/workingDir.
>>
>>
>>> > Perhaps it would make sense to have a magic property that (in
>>> > combination with detection of Surefire environment) would allow
>>> > BuildFileRule to call System.clearProperty("basedir")?
>>>
>>> If that's all it would take to make them work, that's fine with me. If
>>> we need further modifications to the tests them I'm not sure.
>>>
>>
>> Fair enough. I've hit some snags and I commented on them in the POMs.
>>
>
> I am at the point now where I must say "this is the farthest I could get
> without changing the tests".
> Please see the comments in src/etc/poms/ant/pom.xml, especially
> testExcludes section.
> Some tests, in particular in IncludeTest, are not robust (failure modes
> depend on XML parser,
> which may throw an exception because encoding is not specified, which was
> not the original intention).
> AntTest hardcodes directory structure (build/testcases) which apparently
> breaks modular builds in Jenkins
> (works locally). Also, current test failures in Jenkins apparently are an
> oddity of Maven,
> which uses ANT_HOME/jre/java on Java 8 or older SDKs precluding use of
> javac.
>
> Thanks, Gintas
>

It has been eerily quiet here... anyway, here's the state of the test suite
vis-a-vis Surefire and modular Maven builds:

   - most cases that were dependent on assumed structure of build file tree
   are refactored to use build.tests.value property (corresponding to
   testOutputDirectory in Maven);
   - build.tests.value property is redundant where it is used in
   combination with java.class.path property and therefore is removed so that
   the corresponding test classes do not need to assert it;
   - root property is removed from all test build files except one,
   javadoc, where it is used to access Java sources; POMs do not set it, and
   perhaps build.xml needs not set it either;
   - I added build.classes.value property (corresponding to outputDirectory
   in Maven), it's only used in signjar tests currently; I chose build.*.value
   names for the properties in order to avoid accidental name collisions.

Currently, certain Ant core and junit/junitlauncher tests still fail in
Surefire. I would like to get them all working; however, I am afraid that
their logic should be changed quite drastically. Also. please see comments
in pom.xml about some dependencies that must be eliminated (launcher on
core, or core on junit).

I also activated test analysis in Ant_Nightly (build from POMs presents
test analysis automagically). Currently, it seems that Maven executes more
tests than Nightly, and I will look into that discrepancy, too.

Thanks, Gintas