Questions about tests in v8 (trunk)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Questions about tests in v8 (trunk)

Georges Racinet
Hi there, I'm starting to take a look at the future v8, and have a few
questions about testing practices.

First, openerpcommand's run_test option has been removed (actually in
saas-3) with this comment :

  "remove run_tests command, use -d -u modname --test-enable or --test-file"

Was it considered a useless duplication, or is there a deeper reason to
that ?

For the record, at Anybox, we're using mostly nosetests to run openerp
modules tests, because of its superior flexibility (test selection,
replay, coverage, profiling), but our integration is a bit crude for the
time being.

I also noticed the addition of grunt tests. Is there a reference doc or
statement about these ?

Lastly, 'oe initialize' now has a mandatory port argument. My guess is
that it's for PhantomJS tests. If true, wouldn't it be better to skip
the phantomjs tests if no port is supplied ? Some people may want to run
them separately. Some other people may want to be able to init a db
without reserving a port.

Thanks all for the hard work !

Regards,

--
Georges Racinet
Anybox SAS, http://anybox.fr
Bureau: 09 72 39 50 97 / 09 72 39 13 06
Portable: 06 51 32 07 27
GPG: 0x33AB0A35, sur serveurs publics


_______________________________________________
Mailing list: https://launchpad.net/~openerp-expert-framework
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-expert-framework
More help   : https://help.launchpad.net/ListHelp

signature.asc (915 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Questions about tests in v8 (trunk)

Leonardo Pistone
Hi Georges.

Not an answer to your questions, but on the subject on how to run tests: when writing OpenERP tests, and especially when trying to do some TDD, I did feel the need for somewhat better output. In particular, having test failures mixed into the logging messages is a bit inconvenient, assertRaises shows an ERROR message even if the test passes, and there is no exit value in case of failure, so some kind of grepping is needed for CI or for running "git bisect".

Your mention to nose grabbed my attention: even if as you say it is a bit crude, could you please elaborate on your setup with nose?

Thanks!


On Wed, Apr 30, 2014 at 11:58 AM, Georges Racinet <[hidden email]> wrote:
Hi there, I'm starting to take a look at the future v8, and have a few
questions about testing practices.

First, openerpcommand's run_test option has been removed (actually in
saas-3) with this comment :

  "remove run_tests command, use -d -u modname --test-enable or --test-file"

Was it considered a useless duplication, or is there a deeper reason to
that ?

For the record, at Anybox, we're using mostly nosetests to run openerp
modules tests, because of its superior flexibility (test selection,
replay, coverage, profiling), but our integration is a bit crude for the
time being.

I also noticed the addition of grunt tests. Is there a reference doc or
statement about these ?

Lastly, 'oe initialize' now has a mandatory port argument. My guess is
that it's for PhantomJS tests. If true, wouldn't it be better to skip
the phantomjs tests if no port is supplied ? Some people may want to run
them separately. Some other people may want to be able to init a db
without reserving a port.

Thanks all for the hard work !

Regards,

--
Georges Racinet
Anybox SAS, http://anybox.fr
Bureau: 09 72 39 50 97 / 09 72 39 13 06
Portable: 06 51 32 07 27
GPG: 0x33AB0A35, sur serveurs publics


_______________________________________________
Mailing list: https://launchpad.net/~openerp-expert-framework
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-expert-framework
More help   : https://help.launchpad.net/ListHelp



_______________________________________________
Mailing list: https://launchpad.net/~openerp-expert-framework
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-expert-framework
More help   : https://help.launchpad.net/ListHelp
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Questions about tests in v8 (trunk)

Georges Racinet
Hi Leonardo,

On 04/30/2014 12:14 PM, Leonardo Pistone wrote:

> Hi Georges.
>
> Not an answer to your questions, but on the subject on how to run tests:
> when writing OpenERP tests, and especially when trying to do some TDD, I
> did feel the need for somewhat better output. In particular, having test
> failures mixed into the logging messages is a bit inconvenient,
> assertRaises shows an ERROR message even if the test passes, and there
> is no exit value in case of failure, so some kind of grepping is needed
> for CI or for running "git bisect".
>
> Your mention to nose grabbed my attention: even if as you say it is a
> bit crude, could you please elaborate on your setup with nose?
This is for users of the buildout recipe only, but you happen to be one…

It's explained there, as a use-case example (well you'll need to add at
least 'nose' to the eggs):
http://pythonhosted.org/anybox.recipe.openerp/scripts.html#command-line-options

A few precisions on top of that:

- you'll probably need to restrict the scope of search for tests to your
addon, and even (maybe not always) to the precise directory holding the
tests. Something like

  bin/nosetests -d mydb -- my_addons/the_addon/tests

- nose does not care about fast_suite. Conversely, it's easy to forget
to register the tests in fast_suite if you develop with nose, meaning
that the builtin OpenERP runner will ignore them
- for the current trunk, you'll need the fix for this
https://bugs.launchpad.net/anybox.recipe.openerp/+bug/1314566 (later today)

That's it ! Enjoy the options I can't imagine to live without: --pdb,
--pdb-failures, --with-id, --failed etc.

This is also how the integrations tests of the recipe work:
http://docs.anybox.fr/anybox.recipe.openerp/trunk/contributing.html#integration-tests
(in this case, the tests are not even in a proper addon)

Actually, if you launch nosetests with the python interpreter built by
the recipe, it'll work, but you'll get strange effects if nose happens
to import your modules before the registry (through the base test class)
does.

That's why I introduced this "command-line-options=-d" modifier in the
recipe : to load the database before exec'ing nose (previous versions
required protection of the addon against double import).

Alas, the side effect is that coverage doesn't witness the loading,
leading to a mostly unsuable report (that's the crude part)
It's been a while since I've focused on that, but I think to correct the
problem would require writing a dedicated nose plugin.

Cheers,
--
Georges Racinet
Anybox SAS, http://anybox.fr
Bureau: 09 72 39 50 97 / 09 72 39 13 06
Portable: 06 51 32 07 27
GPG: 0x33AB0A35, sur serveurs publics


_______________________________________________
Mailing list: https://launchpad.net/~openerp-expert-framework
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-expert-framework
More help   : https://help.launchpad.net/ListHelp

signature.asc (915 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Questions about tests in v8 (trunk)

Valentin LAB
In reply to this post by Georges Racinet
I recent commits, the whole "openerpcommand" disappeared also. My first
concerns is also about tests.

I believe that tests command should be as slick and easy to run than it
could be and I'm not really keen of a runbot only solution (but this is
better than anything). The run_tests command was a way to try to gather
some tests in a single simple command that anyone could run in various
scenarios.

I'm wondering what is the general view of the openerp on tests policy.

Here some points that seems to suggest that "runbot" perspective is the
only one in town:
- the command to run unittest is the command to run the server.
- the server is run, HTTP server is setup (port is open), database
   installed even to run unit test and bailout afterwards.
- late inclusion of the --stop-after-init: the server would simply not
   stop even if you wanted to only run test.
- no easy way to run only the unittest without YAML test (and so,
   without installing a database with demo data).
...

This way of thinking tests prevents any pre-commit tests scenario, or
short cycle test feedbacks for dev scenarios. It forces usage of a
relatively huge setup (the integration setup) to handle even minor
tests, and forces the test in post-push policy.

As a matter of fact, having an easy single point of entry to launch test
won't impede integration and can only encourage every developper to run
the tests more often. It can't be a bad thing.

I know that some types of tests requires heavy installation (a working
postgres instances with a database for instance), and these should be
separable from the quick tests that do not need anything but the openerp
code.

Ideally, I'd like to find in odoo a simple entrypoint that just run the
tests with a few arguments to select tests and types of tests. This
command should also be able to launch all available tests (Javascript
ones). And finally, the runbot would use this command.



On 30/04/14 11:58, Georges Racinet wrote:

> Hi there, I'm starting to take a look at the future v8, and have a few
> questions about testing practices.
>
> First, openerpcommand's run_test option has been removed (actually in
> saas-3) with this comment :
>
>    "remove run_tests command, use -d -u modname --test-enable or --test-file"
>
> Was it considered a useless duplication, or is there a deeper reason to
> that ?
>
> For the record, at Anybox, we're using mostly nosetests to run openerp
> modules tests, because of its superior flexibility (test selection,
> replay, coverage, profiling), but our integration is a bit crude for the
> time being.
>
> I also noticed the addition of grunt tests. Is there a reference doc or
> statement about these ?
>
> Lastly, 'oe initialize' now has a mandatory port argument. My guess is
> that it's for PhantomJS tests. If true, wouldn't it be better to skip
> the phantomjs tests if no port is supplied ? Some people may want to run
> them separately. Some other people may want to be able to init a db
> without reserving a port.
>
> Thanks all for the hard work !
>
> Regards,
>
>
>
> _______________________________________________
> Mailing list: https://launchpad.net/~openerp-expert-framework
> Post to     : [hidden email]
> Unsubscribe : https://launchpad.net/~openerp-expert-framework
> More help   : https://help.launchpad.net/ListHelp
>


--
Valentin LAB
Ingénieur Développement

tel:  +33 6 71 39 62 13
mail: [hidden email]

_______________________________________________
Mailing list: https://launchpad.net/~openerp-expert-framework
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-expert-framework
More help   : https://help.launchpad.net/ListHelp
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Questions about tests in v8 (trunk)

Antony Lesuisse
The --test-file options supports python files too.

On 07/02/2014 04:21 PM, Valentin LAB wrote:

> I recent commits, the whole "openerpcommand" disappeared also. My first
> concerns is also about tests.
>
> I believe that tests command should be as slick and easy to run than it could
> be and I'm not really keen of a runbot only solution (but this is better than
> anything). The run_tests command was a way to try to gather some tests in a
> single simple command that anyone could run in various scenarios.
>
> I'm wondering what is the general view of the openerp on tests policy.
>
> Here some points that seems to suggest that "runbot" perspective is the only
> one in town:
> - the command to run unittest is the command to run the server.
> - the server is run, HTTP server is setup (port is open), database
>    installed even to run unit test and bailout afterwards.
> - late inclusion of the --stop-after-init: the server would simply not
>    stop even if you wanted to only run test.
> - no easy way to run only the unittest without YAML test (and so,
>    without installing a database with demo data).
> ...
>
> This way of thinking tests prevents any pre-commit tests scenario, or short
> cycle test feedbacks for dev scenarios. It forces usage of a relatively huge
> setup (the integration setup) to handle even minor tests, and forces the test
> in post-push policy.
>
> As a matter of fact, having an easy single point of entry to launch test won't
> impede integration and can only encourage every developper to run the tests
> more often. It can't be a bad thing.
>
> I know that some types of tests requires heavy installation (a working
> postgres instances with a database for instance), and these should be
> separable from the quick tests that do not need anything but the openerp code.
>
> Ideally, I'd like to find in odoo a simple entrypoint that just run the tests
> with a few arguments to select tests and types of tests. This command should
> also be able to launch all available tests (Javascript ones). And finally, the
> runbot would use this command.
>
>
>
> On 30/04/14 11:58, Georges Racinet wrote:
>> Hi there, I'm starting to take a look at the future v8, and have a few
>> questions about testing practices.
>>
>> First, openerpcommand's run_test option has been removed (actually in
>> saas-3) with this comment :
>>
>>    "remove run_tests command, use -d -u modname --test-enable or --test-file"
>>
>> Was it considered a useless duplication, or is there a deeper reason to
>> that ?
>>
>> For the record, at Anybox, we're using mostly nosetests to run openerp
>> modules tests, because of its superior flexibility (test selection,
>> replay, coverage, profiling), but our integration is a bit crude for the
>> time being.
>>
>> I also noticed the addition of grunt tests. Is there a reference doc or
>> statement about these ?
>>
>> Lastly, 'oe initialize' now has a mandatory port argument. My guess is
>> that it's for PhantomJS tests. If true, wouldn't it be better to skip
>> the phantomjs tests if no port is supplied ? Some people may want to run
>> them separately. Some other people may want to be able to init a db
>> without reserving a port.
>>
>> Thanks all for the hard work !
>>
>> Regards,
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openerp-expert-framework
>> Post to     : [hidden email]
>> Unsubscribe : https://launchpad.net/~openerp-expert-framework
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>

_______________________________________________
Mailing list: https://launchpad.net/~openerp-expert-framework
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-expert-framework
More help   : https://help.launchpad.net/ListHelp
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Questions about tests in v8 (trunk)

Georges Racinet
In reply to this post by Valentin LAB
Hi Valentin, nice to hear from you !

On 07/02/2014 04:21 PM, Valentin LAB wrote:
> I recent commits, the whole "openerpcommand" disappeared also. My first
> concerns is also about tests.
>
> I believe that tests command should be as slick and easy to run than it
> could be and I'm not really keen of a runbot only solution (but this is
> better than anything). The run_tests command was a way to try to gather
> some tests in a single simple command that anyone could run in various
> scenarios.

It was already not enough for our needs.
But clearly Anthony must think it's simpler that way, and I'm not going
to argue about this. Actually, with the buildout recipe, I can use and
provide something more stable than the back-and-forth we've seen with
openerpcommand, and more powerful.

>
> I'm wondering what is the general view of the openerp on tests policy.
>
> Here some points that seems to suggest that "runbot" perspective is the
> only one in town:
> - the command to run unittest is the command to run the server.
> - the server is run, HTTP server is setup (port is open), database
>   installed even to run unit test and bailout afterwards.
> - late inclusion of the --stop-after-init: the server would simply not
>   stop even if you wanted to only run test.
> - no easy way to run only the unittest without YAML test (and so,
>   without installing a database with demo data).
> ...
>
> This way of thinking tests prevents any pre-commit tests scenario, or
> short cycle test feedbacks for dev scenarios. It forces usage of a
> relatively huge setup (the integration setup) to handle even minor
> tests, and forces the test in post-push policy.
Not much to add on this, I've came more or less to the same conclusions.

That being said, you can launch real unit tests (not needing any DB) if
you have some, the regular way with a general purpose test launcher
(such as nose) ; at worst you'll need to tweak sys.path

To gain back some flexibility in integration setup, using nose (or any
test launcher of your choice) with the buildout recipe is a reasonible
compromise.
As I have lots of reason to be wary of mocks, this is mostly what we are
doing at Anybox.

>
> As a matter of fact, having an easy single point of entry to launch test
> won't impede integration and can only encourage every developper to run
> the tests more often. It can't be a bad thing.

>
> I know that some types of tests requires heavy installation (a working
> postgres instances with a database for instance), and these should be
> separable from the quick tests that do not need anything but the openerp
> code.

But the odoo code is ultimately a machinery to make SQL statements.
If there was an alternate, reliable, implementation of the whole API to
provide a testing harness, then going full unit would be appreciable.
Does the new API come with that ?

(I'm assuming you're not speaking of internal tests of the server, and
are considering mostly module development).

For me, the development workflow (for a module) consisting of

- module install on fresh DB
- test, code, test, code, test, code
- DB upgrade (new fields etc)
- test, code, again etc

is very acceptable, provided I have a decent test launcher.
Unfortunately, those that don't do TDD ever don't realize how faster one
can develop with well-thought tests and a good launcher.

Writing tests that make my module rely on assumptions I made on how the
ORM is supposed to react is clearly not.
That doesn't preclude also separating non trivial treatments that dont
need the API and unit-testing that, granted.

>
> Ideally, I'd like to find in odoo a simple entrypoint that just run the
> tests with a few arguments to select tests and types of tests. This
> command should also be able to launch all available tests (Javascript
> ones). And finally, the runbot would use this command.
>
>
>
> On 30/04/14 11:58, Georges Racinet wrote:
>> Hi there, I'm starting to take a look at the future v8, and have a few
>> questions about testing practices.
>>
>> First, openerpcommand's run_test option has been removed (actually in
>> saas-3) with this comment :
>>
>>    "remove run_tests command, use -d -u modname --test-enable or
>> --test-file"
>>
>> Was it considered a useless duplication, or is there a deeper reason to
>> that ?
>>
>> For the record, at Anybox, we're using mostly nosetests to run openerp
>> modules tests, because of its superior flexibility (test selection,
>> replay, coverage, profiling), but our integration is a bit crude for the
>> time being.
>>
>> I also noticed the addition of grunt tests. Is there a reference doc or
>> statement about these ?
>>
>> Lastly, 'oe initialize' now has a mandatory port argument. My guess is
>> that it's for PhantomJS tests. If true, wouldn't it be better to skip
>> the phantomjs tests if no port is supplied ? Some people may want to run
>> them separately. Some other people may want to be able to init a db
>> without reserving a port.
>>
>> Thanks all for the hard work !
>>
>> Regards,
>>
>>
>>
>> _______________________________________________
>> Mailing list: https://launchpad.net/~openerp-expert-framework
>> Post to     : [hidden email]
>> Unsubscribe : https://launchpad.net/~openerp-expert-framework
>> More help   : https://help.launchpad.net/ListHelp
>>
>
>

--
Georges Racinet
Anybox SAS, http://anybox.fr
Bureau: 09 72 39 50 97 / 09 72 39 13 06
Portable: 06 51 32 07 27
GPG: 0x33AB0A35, sur serveurs publics


_______________________________________________
Mailing list: https://launchpad.net/~openerp-expert-framework
Post to     : [hidden email]
Unsubscribe : https://launchpad.net/~openerp-expert-framework
More help   : https://help.launchpad.net/ListHelp

signature.asc (915 bytes) Download Attachment
Loading...